# ZIO

> Type-safe, composable asynchronous and concurrent programming for Scala.

This file contains all documentation content in a single document following the llmstxt.org standard.

## Basic Concurrency

ZIO is a highly concurrent framework, powered by _fibers_, which are lightweight virtual threads that achieve massive scalability compared to threads, augmented with resource-safe cancellation, which powers many features in ZIO.

This powerful concurrency model lets you do more with less, achieving highly-scalable, ultra low-latency applications that are globally efficient and resource-safe.

In this section, you will learn the basics of fibers, and become acquainted with some of the powerful high-level operators that are powered by fibers.

## Fibers

All effects in ZIO are executed by _some_ fiber. If you did not create the fiber, then the fiber was created by some operation you are using (if the operation is concurrent or parallel), or by the ZIO runtime system.

Even if you only write "single-threaded" code, with no parallel or concurrent operations, there will be at least one fiber: the "main" fiber that executes your effect.

Like operating system-level threads, ZIO fibers have a well-defined lifecycle, defined by the effect they are executing.

Every fiber exits with failure or success, depending on whether the effect it is executing fails or succeeds.

Also like operating system threads, ZIO fibers have unique identities, stacks (including stack traces), local state, and a status (such as _done_, _running_, or _suspended_).

Compared to operating system threads, ZIO fibers:

 - Consume almost no memory
 - Have dynamic stacks that grow and shrink
 - Don't waste operating system threads with blocking operations
 - Can be safely interrupted at any point in time
 - Are strongly typed
 - Let you query them to discover their children 
 - Will be garbage collected automatically if they are suspended and cannot be reactivated

These make fibers a superior choice for building modern applications.

Fibers are scheduled onto operating system threads by the ZIO runtime. Because fibers cooperatively yield to each other, ZIO fibers always execute concurrently, even when running in a single-threaded environment like JavaScript (or the JVM, when ZIO is configured with one work thread).

### The Fiber Data Type

The `Fiber` data type in ZIO represents a "handle" on the execution of an effect. The `Fiber` data type is most similar to Scala's `Future` data type, which represents a "handle" on a running asynchronous operation.

The `Fiber[E, A]` data type in ZIO has two type parameters:

 - **`E` Failure Type**. The fiber may fail with a value of this type.
 - **`A` Success Type**. The fiber may succeed with a value of this type.

Fibers do not have an `R` type parameter, because fibers only execute effects that have already had their requirements provided to them.

### Forking Effects

The most fundamental way of creating a fiber is to take an existing effect and _fork_ it. Conceptually, _forking_ an effect begins executing the effect on a new fiber, giving you a reference to the newly-created fiber.

The following code creates a single fiber using `fork`, which executes `fib(100)` independently of the main fiber:

```scala
def fib(n: Long): UIO[Long] = 
  ZIO.suspendSucceed {
    if (n <= 1) ZIO.succeed(n)
    else fib(n - 1).zipWith(fib(n - 2))(_ + _)
  }

val fib100Fiber: UIO[Fiber[Nothing, Long]] = 
  for {
    fiber <- fib(100).fork
  } yield fiber
```

### Joining Fibers

One of the methods on `Fiber` is `Fiber#join`, which returns an effect. The effect returned by `Fiber#join` will succeed or fail as per the fiber:

```scala
for {
  fiber   <- ZIO.succeed("Hi!").fork
  message <- fiber.join
} yield message
```

When a parent fiber joins a child fiber, it will succeed or fail in the same way as the child fiber, and the local states of the fibers will be merged.

### Awaiting Fibers

Another method on `Fiber` is `Fiber#await`, which returns an effect containing an `Exit` value, which provides full information on how the fiber completed.

```scala
for {
  fiber <- ZIO.succeed("Hi!").fork
  exit  <- fiber.await
} yield exit
```

Awaiting the exit values of fibers is different than joining them, because awaiting will not tie the fate of the parent fiber to that of the child fiber, and nor will it attempt to merge the local states of the fibers.

### Interrupting Fibers

A fiber whose result is no longer needed may be _interrupted_, which immediately terminates the fiber, safely releasing all resources by running all finalizers.

Like `await`, `Fiber#interrupt` returns an `Exit` describing how the fiber completed.

```scala
for {
  fiber <- ZIO.succeed("Hi!").forever.fork
  exit  <- fiber.interrupt
} yield exit
```

By design, the effect returned by `Fiber#interrupt` does not resume until the fiber has completed, which helps ensure your code does not spin up new fibers until the old one has terminated.

If this behavior (often called "back-pressuring") is not desired, you can `ZIO#fork` the interruption itself into a new fiber:

```scala
for {
  fiber <- ZIO.succeed("Hi!").forever.fork
  _     <- fiber.interrupt.fork // I don't care!
} yield ()
```

There is a shorthand for background interruption, which is the method `Fiber#interruptFork`.

### Composing Fibers

ZIO lets you compose fibers with `Fiber#zip` or `Fiber#zipWith`. 

These methods combine two fibers into a single fiber that produces the results of both. If either fiber fails, then the composed fiber will fail.

```scala
for {
  fiber1 <- ZIO.succeed("Hi!").fork
  fiber2 <- ZIO.succeed("Bye!").fork
  fiber   = fiber1.zip(fiber2)
  tuple  <- fiber.join
} yield tuple
```

Another way fibers compose is with `Fiber#orElse`. If the first fiber succeeds, the composed fiber will succeed with its result; otherwise, the composed fiber will complete with the exit value of the second fiber (whether success or failure).

```scala
for {
  fiber1 <- ZIO.fail("Uh oh!").fork
  fiber2 <- ZIO.succeed("Hurray!").fork
  fiber   = fiber1.orElse(fiber2)
  message  <- fiber.join
} yield message
```

## Parallelism

ZIO provides parallel versions of many methods, which are named with a `Par` suffix that helps you identify opportunities to parallelize your code.

For example, the ordinary `ZIO#zip` method zips two effects together sequentially. But there is also a `ZIO#zipPar` method, which zips two effects together in parallel.

The following table summarizes some of the sequential operations and their corresponding parallel versions:

| **Description**                | **Sequential**    | **Parallel**         |
| -----------------------------: | :---------------: | :------------------: |
| Zips two effects into one      | `ZIO#zip`         | `ZIO#zipPar`         |
| Zips two effects into one      | `ZIO#zipWith`     | `ZIO#zipWithPar`     |
| Zips multiple effects into one | `ZIO#tupled`      | `ZIO#tupledPar`      |
| Collects from many effects     | `ZIO.collectAll`  | `ZIO.collectAllPar`  |
| Effectfully loop over values   | `ZIO.foreach`     | `ZIO.foreachPar`     |
| Reduces many values            | `ZIO.reduceAll`   | `ZIO.reduceAllPar`   |
| Merges many values             | `ZIO.mergeAll`    | `ZIO.mergeAllPar`    |

Because all these parallel operators return all the results, if any effect being parallelized fails, ZIO will automatically cancel the other running effects, as their results will not be used.

If the fail-fast behavior is not desired, potentially failing effects can be first converted into infallible effects using the `ZIO#either` or `ZIO#option` methods.

## Racing

ZIO lets you race multiple effects concurrently, returning the first successful result:

```scala
for {
  winner <- ZIO.succeed("Hello").race(ZIO.succeed("Goodbye"))
} yield winner
```

If you want the first success **or** failure, rather than the first success, then you can use `left.either.race(right.either)`, for any two effects `left` and `right`.

## Timeout

ZIO has resource-safe, compositional timeouts that work on "small" effects, such as querying a database or calling a cloud API, or even "large" effects, such as running a streaming pipeline or fully handling a web request.

ZIO lets you timeout effects using the `ZIO#timeout` method, which returns a new effect that succeeds with an `Option` value. 

A value of `None` indicates the timeout elapsed before the effect completed.

```scala
ZIO.succeed("Hello").timeout(10.seconds)
```

If an effect times out, then instead of continuing to execute in the background, it will be interrupted, for automatic efficiency.

## Next Steps

If you are comfortable with basic concurrency, the next step is to learn about [running effects](running-effects.md).

---

## Basic Operations

Like the `String` data type, as well as the collection data types in Scala (such as `List`, `Map`, and `Set`), ZIO effects are _immutable_, and cannot be changed. 

In order to transform or combine ZIO effects, you can use the methods on the ZIO data type, which return _new effects_, with the specified transformations or combinations applied to them.

There are two categories of methods on the ZIO data type:

 - **Transformations**. Transformation functions alter an effect in some well-defined way, allowing you to customize runtime behavior. For example, calling `effect.timeout(60.seconds)` on an effect returns a new effect, which, when executed, will apply a timeout to the original effect.
 - **Combinations**. Combination functions combine two or more effects together in a single effect. For example, calling `effect1.orElse(effect2)` combines two effects in such a fashion that the returned effect, when executed, will first execute the left hand side, and if that fails, it will then execute the right hand side. This lets you specify a fallback effect in case a primary effect fails.

## Mapping

If you have an effect that succeeds with some value, you can use `ZIO#map` to obtain a new effect, which will transform the value using the function you provide.

```scala

val succeeded: ZIO[Any, Nothing, Int] = ZIO.succeed(21).map(_ * 2)
```

In a similar fashion, you can transform an effect that has one error to an effect with a different error using the `ZIO#mapError` method, which requires you supply a function to do the conversion:

```scala
val failed: ZIO[Any, Exception, Unit] = 
  ZIO.fail("No no!").mapError(msg => new Exception(msg))
```

Note that mapping the error or success value of an effect does not change _whether or not_ the effect fails or succeeds. This is similar to how mapping over Scala's `Either` data type does not change whether the `Either` is `Left` or `Right`.

## Chaining

You can execute two effects sequentially with the `flatMap` method. The `flatMap` method requires that you pass a callback, which will receive the success value of the first effect, and must return a second effect, which depends on this value:

```scala
val sequenced: ZIO[Any, IOException, Unit] =
  Console.readLine.flatMap(input => Console.printLine(s"You entered: $input"))
```

If the first effect fails, the callback passed to `flatMap` will never be invoked, and the effect returned by `flatMap` will also fail.

In _any_ chain of effects created with `flatMap`, the first failure will short-circuit the whole chain, just like throwing an exception will prematurely exit a sequence of statements.

## For Comprehensions

Because the ZIO data type supports both `flatMap` and `map`, you can use Scala's _for comprehensions_ to build imperative effects:

```scala
val program: ZIO[Any, IOException, Unit] =
  for {
    _    <- Console.printLine("Hello! What is your name?")
    name <- Console.readLine
    _    <- Console.printLine(s"Hello, ${name}, welcome to ZIO!")
  } yield ()
```

_For comprehensions_ provide a procedural syntax for creating chains of effects, and are the fastest way for most programmers to get up to speed using ZIO.

## Zipping

You can combine two effects into a single effect with the `ZIO#zip` method. 

The method returns an effect that will execute the left effect first, followed by the right effect, and which will place both success values into a tuple:

```scala
val zipped: ZIO[Any, Nothing, (String, Int)] = 
  ZIO.succeed("4").zip(ZIO.succeed(2))
```

In any `zip` operation, if either the left or right-hand side fails, the composed effect will fail, because _both_ values are required to construct the tuple. If the left side fails, the right side will not be executed at all.

Sometimes, when the success value of an effect is not useful (for example, if it is `Unit`), it can be more convenient to use the `ZIO#zipLeft` or `ZIO#zipRight` functions, which first perform a `zip` and then map over the tuple to discard one side or the other:

```scala
val zipRight1: ZIO[Any, IOException, String] =
  Console.printLine("What is your name?").zipRight(Console.readLine)
```

The `zipRight` and `zipLeft` functions have symbolic aliases, known as `*>` and `<*`, respectively. Some developers find these operators easier to read:

```scala
val zipRight2: ZIO[Any, IOException, String] =
  Console.printLine("What is your name?") *>
    Console.readLine
```

## Next Steps

If you are comfortable with the basic operations on ZIO effects, the next step is to learn about [error handling](handling-errors.md).

---

## Creating Effects

This section explores some of the common ways to create ZIO effects from values, from computations, and from common Scala data types.

## From Values

Using the `ZIO.succeed` method, you can create an effect that, when executed, will succeed with the specified value:

```scala
val s1 = ZIO.succeed(42)
```

The `succeed` method takes a so-called _by-name parameter_, which ensures that if you pass the method some code to execute, that this code will be stored inside the ZIO effect so that it can be managed by ZIO, and benefit from features like retries, timeouts, and automatic error logging.

## From Failure Values

Using the `ZIO.fail` method, you can create an effect that, when executed, will fail with the specified value:

```scala
val f1 = ZIO.fail("Uh oh!")
```

For the `ZIO` data type, there is no restriction on the error type. You may use strings, exceptions, or custom data types appropriate for your application.

Many applications will model failures with classes that extend `Throwable` or `Exception`:

```scala
val f2 = ZIO.fail(new Exception("Uh oh!"))
```

## From Scala Values

Scala's standard library contains a number of data types that can be converted into ZIO effects.

### Option

An `Option` can be converted into a ZIO effect using `ZIO.fromOption`:

```scala
val zoption: IO[Option[Nothing], Int] = ZIO.fromOption(Some(2))
```

The error type of the resulting effect is `Option[Nothing]`, signifying that if such an effect fails, it will fail with the value `None` (which has type `Option[Nothing]`).

You can transform a failure into some other error value using `orElseFail`, one of many methods that ZIO provides for error management:

```scala
val zoption2: ZIO[Any, String, Int] = zoption.orElseFail("It wasn't there!")
```

ZIO has a variety of other operators designed to make interfacing with `Option` code easier. In the following advanced example, the operators `some`  and `asSomeError` are used to make it easier to interface with methods returning `Option`, similar to the `OptionT` type in some Scala libraries.

```scala
val maybeId: ZIO[Any, Option[Nothing], String] = ZIO.fromOption(Some("abc123"))
def getUser(userId: String): ZIO[Any, Throwable, Option[User]] = ???
def getTeam(teamId: String): ZIO[Any, Throwable, Team] = ???

val result: ZIO[Any, Throwable, Option[(User, Team)]] = (for {
  id   <- maybeId
  user <- getUser(id).some
  team <- getTeam(user.teamId).asSomeError 
} yield (user, team)).unsome 
```

### Either

An `Either` can be converted into a ZIO effect using `ZIO.fromEither`:

```scala
val zeither: ZIO[Any, Nothing, String] = ZIO.fromEither(Right("Success!"))
```

The error type of the resulting effect will be that of the `Left` case, while the success type will be that of the `Right` case.

### Try

A `Try` value can be converted into a ZIO effect using `ZIO.fromTry`:

```scala

val ztry = ZIO.fromTry(Try(42 / 0))
```

The error type of the resulting effect will always be `Throwable` because `Try` can only fail with values of type `Throwable`.

### Future

A Scala `Future` can be converted into a ZIO effect using `ZIO.fromFuture`:

```scala

lazy val future = Future.successful("Hello!")

val zfuture: ZIO[Any, Throwable, String] =
  ZIO.fromFuture { implicit ec =>
    future.map(_ => "Goodbye!")
  }
```

The function passed to `fromFuture` is provided an `ExecutionContext`, which allows ZIO to manage where the `Future` runs (of course, you can ignore this `ExecutionContext`).

The error type of the resulting effect will always be `Throwable`, because `Future` values can only fail with values of type `Throwable`.

## From Code

ZIO can convert any code (such as a call to some method) into an effect, whether that code is so-called _synchronous_ (directly returning a value), or _asynchronous_ (passing a value to callbacks).

If done properly, when you convert code into a ZIO effect, this code will be stored inside the effect so that it can be managed by ZIO, and benefit from features like retries, timeouts, and automatic error logging.

The conversion functions that ZIO has allow you to seamlessly use all features of ZIO with non-ZIO code written in Scala or Java, including third-party libraries.

### Synchronous Code

Synchronous code can be converted into a ZIO effect using `ZIO.attempt`:

```scala

val readLine: ZIO[Any, Throwable, String] =
  ZIO.attempt(StdIn.readLine())
```

The error type of the resulting effect will always be `Throwable`, because synchronous code may throw exceptions with any value of type `Throwable`.

If you know for a fact that some code does not throw exceptions (except perhaps runtime exceptions), you can convert the code into a ZIO effect using `ZIO.succeed`:

```scala
def printLine(line: String): UIO[Unit] =
  ZIO.succeed(println(line))
```

Sometimes, you may know that code throws a specific exception type, and you may wish to reflect this in the error parameter of your ZIO effect.

For this purpose, you can use the `ZIO#refineToOrDie` method:

```scala

val readLine2: ZIO[Any, IOException, String] =
  ZIO.attempt(StdIn.readLine()).refineToOrDie[IOException]
```

### Asynchronous Code

Asynchronous code that exposes a callback-based API can be converted into a ZIO effect using `ZIO.async`:

```scala
object legacy {
  def login(
    onSuccess: User => Unit,
    onFailure: AuthError => Unit): Unit = ???
}

val login: ZIO[Any, AuthError, User] =
  ZIO.async[Any, AuthError, User] { callback =>
    legacy.login(
      user => callback(ZIO.succeed(user)),
      err  => callback(ZIO.fail(err))
    )
  }
```

Asynchronous effects are much easier to use than callback-based APIs, and they benefit from ZIO features like interruption, resource-safety, and error management.

## Blocking Synchronous Code

Some synchronous code may engage in so-called _blocking IO_, which puts a thread into a waiting state, as it waits for some operating system call to complete. For maximum throughput, this code should not run on your application's primary thread pool, but rather, in a special thread pool that is dedicated to blocking operations.

ZIO has a blocking thread pool built into the runtime, and lets you execute effects there with `ZIO.blocking`:

```scala

def download(url: String) =
  ZIO.attempt {
    Source.fromURL(url)(Codec.UTF8).mkString
  }

def safeDownload(url: String) =
  ZIO.blocking(download(url))
```

As an alternative, if you wish to convert blocking code directly into a ZIO effect, you can use the `ZIO.attemptBlocking` method:

```scala
val sleeping =
  ZIO.attemptBlocking(Thread.sleep(Long.MaxValue))
```

The resulting effect will be executed on ZIO's blocking thread pool.

If you have some synchronous code that will respond to Java's `Thread.interrupt` (such as `Thread.sleep` or lock-based code), then you can convert this code into an interruptible ZIO effect using the `ZIO.attemptBlockingInterrupt` method.

Some synchronous code can only be cancelled by invoking some other code, which is responsible for canceling the running computation. To convert such code into a ZIO effect, you can use the `ZIO.attemptBlockingCancelable` method:

```scala

def accept(l: ServerSocket) =
  ZIO.attemptBlockingCancelable(l.accept())(ZIO.succeed(l.close()))
```
## Next Steps

If you are comfortable creating effects from values, converting from Scala types into effects, and converting synchronous and asynchronous code into effects, the next step is learning [basic operations](basic-operations.md) on effects.

---

## Handling Errors

ZIO effects may fail due to foreseen or unforeseen problems. In order to help you build robust applications, ZIO tracks foreseen errors at compile-time, letting you know which effects can fail, and how they can fail. For non-recoverable problems, ZIO gives you full insight into the cause of failures (even if unexpected or catastrophic), preserving all information and automatically logging unhandled errors.

In this section, you will learn about some of the tools ZIO gives you to build applications with robust error management.

## Either

With the `ZIO#either` method, you can transform an effect that fails into an infallible effect that places both failure and success into Scala's `Either` type. This brings the error from the error channel to the success channel, which is useful because many ZIO operators work on the success channel, not the error channel.

```scala
val zeither: ZIO[Any, Nothing, Either[String, Nothing]] = 
  ZIO.fail("Uh oh!").either
```

## Catching All Errors

If you want to catch and recover from all types of recoverable errors and effectfully attempt recovery, then you can use the `catchAll` method, which lets you specify an error handler that returns the effect to execute in the event of an error:

```scala
val z: ZIO[Any, IOException, Array[Byte]] = 
  openFile("primary.json").catchAll { error => 
    for {
      _    <- ZIO.logErrorCause("Could not open primary file", Cause.fail(error))
      file <- openFile("backup.json")
    } yield file 
  }
```

In the error handler passed to `catchAll`, you may return an effect with a _different_ error type (perhaps `Nothing`, if the error handler cannot fail), which is then reflected in the type of effect returned by `catchAll`.

## Catching Some Errors

If you want to catch and recover from only some types of recoverable errors and effectfully attempt recovery, then you can use the `catchSome` method:

```scala
val data: ZIO[Any, IOException, Array[Byte]] = 
  openFile("primary.data").catchSome {
    case _ : FileNotFoundException => 
      openFile("backup.data")
  }
```

Unlike `catchAll`, `catchSome` cannot reduce or eliminate the error type, although it can widen the error type to a broader class of errors.

## Fallback

You can try one effect or if it fails, try another effect with the `orElse` combinator:

```scala
val primaryOrBackupData: ZIO[Any, IOException, Array[Byte]] = 
  openFile("primary.data").orElse(openFile("backup.data"))
```

## Folding

In the Scala standard library, the data types `Option` and `Either` have a `fold` method, which lets you handle both failure and success cases at the same time.

In a similar fashion, `ZIO` effects also have several methods that allow you to handle both failure and success at the same time.

The first fold method, `fold`, lets you separately convert both failure and success into some common type:

```scala
lazy val DefaultData: Array[Byte] = Array(0, 0)

val primaryOrDefaultData: ZIO[Any, Nothing, Array[Byte]] = 
  openFile("primary.data").fold(
    _    => DefaultData, // Failure case
    data => data)        // Success case
```

The second fold method, `foldZIO`, lets you separately handle both failure and success by specifying effects that will be executed in each respective case:

```scala
val primaryOrSecondaryData: ZIO[Any, IOException, Array[Byte]] = 
  openFile("primary.data").foldZIO(
    _    => openFile("secondary.data"), // Error handler
    data => ZIO.succeed(data))          // Success handler
```

The `foldZIO` method is almost the most powerful error recovery method in ZIO, with only `foldCauseZIO` being more powerful. Most other operators, such as `either` or `orElse`, are implemented in terms of these powerful methods.

In the following additional example, `foldZIO` is used to handle both the failure and the success of the `readUrls` method:

```scala
val urls: ZIO[Any, Nothing, Content] =
  readUrls("urls.json").foldZIO(
    error   => ZIO.succeed(Content.NoContent(error)), 
    success => fetchContent(success)
  )
```

## Retrying

In order to deal with transient errors, which are the norm when interacting with external cloud systems, ZIO provides very powerful retry mechanisms.

One of these mechanisms is the `ZIO#retry` method, which takes a `Schedule`, and returns a new effect that will retry the original effect if it fails, according to the specified schedule:

```scala
val retriedOpenFile: ZIO[Any, IOException, Array[Byte]] = 
  openFile("primary.data")
      .retry(Schedule.recurs(5))
```

The next most powerful function is `ZIO#retryOrElse`, which allows specification of a fallback to use if the effect does not succeed with the specified policy:

```scala
val retryOpenFile: ZIO[Any, IOException, DefaultData) = 
  openFile("primary.data")
      .retryOrElse(Schedule.recurs(5), (_, _) => ZIO.succeed(DefaultData))
```

For more information on how to build schedules, see the documentation on [Schedule](../reference/schedule/index.md).

## Next Steps

If you are comfortable with basic error handling, including applying simple retry logic to effects, the next step is to learn about safe [resource handling](handling-resources.md).

---

## Handling Resources

Ensuring that your applications never leak resources is one of the keys to maximizing application throughput, minimizing latency, and maximizing per-node uptime.

Yet, achieving resource safety in the presence of asynchronous operations, concurrency, and ZIO's interruption model (which will automatically cancel running effects anytime their results will no longer be used) is challenging.

In this section, you will learn a few of the tools that ZIO provides to create safe applications that never leak resources, even in the case of failure, interruption, or defects in your application.

## Finalizing

In many languages, the `try` / `finally` construct provides a language-level way to guarantee that when the `try` code exits, either normally or abnormally, the _finalizer_ code in the `finally` block will be executed.

ZIO provides a version of this with the `ZIO#ensuring` method, whose guarantees hold across concurrent and async effects. ZIO goes one step further in automatically and losslessly aggregating errors from finalizers.

As with `try` / `finally`, the `ensuring` method guarantees if the effect it is called on begins executing and terminates (either normally or abnormally), then the finalizer will begin execution.

```scala
val finalizer: UIO[Unit] = 
  ZIO.succeed(println("Finalizing!"))
// finalizer: UIO[Unit] = Sync(
//   trace = "repl.MdocSession.MdocApp.finalizer(handling-resources.md:15)",
//   eval = <function0>
// )

val finalized: IO[String, Unit] = 
  ZIO.fail("Failed!").ensuring(finalizer)
// finalized: IO[String, Unit] = DynamicNoBox(
//   trace = "repl.MdocSession.MdocApp.finalized(handling-resources.md:19)",
//   update = 1L,
//   f = zio.ZIO$$$Lambda$19563/0x00007f02a6ecf668@4761552
// )
```

In ZIO, finalizers are not allowed to fail in any recoverable way, which means that you must handle all of the errors that your code can produce.

Like `try` / `finally`, finalizers can be nested, and the failure of any inner finalizer will not affect outer finalizers. Nested finalizers will be executed in reverse order and sequentially, with later finalizers executed only after earlier finalizers.

## Acquire Release 

A common use for `try` is safely acquiring and releasing resources, such as new socket connections or opened files:

```scala 
val handle = openFile(name)

try {
  processFile(handle)
} finally closeFile(handle)
```

ZIO encapsulates this common pattern with `ZIO.acquireReleaseWith`, which allows you to specify an _acquire_ effect, which acquires a resource; a _release function_, which returns an effect to release the resource; and a _use function_, which returns an effect that _uses_ the resource.

So long as the acquire effect succeeds, the release effect is guaranteed to be executed by the runtime system, even in the presence of errors or interruption.

```scala
val groupedFileData: IO[IOException, Unit] = 
  ZIO.acquireReleaseWith(openFile("data.json"))(closeFile(_)) { file =>
    for {
      data    <- decodeData(file)
      grouped <- groupData(data)
    } yield grouped
  }
```

Like `ensuring`, `acquireReleaseWith` has compositional semantics, so if one `acquireReleaseWith` is nested inside another `acquireReleaseWith`, and the outer resource is acquired, then the outer release will always be called, even if, for example, the inner release fails.

For resources which implement the AutoClosable interface, the convenience method `fromAutoClosable` can be used, which can be seen as the ZIO equivalent of try-with-resource.

```scala
val bytesInFile: IO[Throwable, Int] =
  ZIO.scoped {
    for {
      stream <- ZIO.fromAutoCloseable(openFileInputStream("data.json"))
      data   <- ZIO.attemptBlockingIO(stream.readAllBytes())
    } yield data.length
  }
```

## Next Steps

If you are comfortable with basic resource handling, the next step is to learn about [basic concurrency](basic-concurrency.md).

---

## Getting Started with ZIO

## Teach Your Coding Agent Latest ZIO Knowledge

The `zio-knowledge` skill teaches your coding agent to fetch live documentation from zio.dev before answering any ZIO question — so you always get accurate, up-to-date answers, not guesses from stale training data:

```bash
npx skills add zio/zio-skills --skill zio-knowledge
```

## Installation

Include ZIO in your project by adding the following to your `build.sbt` file:

```
libraryDependencies += "dev.zio" %% "zio" % "2.1.26"
```

If you want to use ZIO streams, you should also include the following dependency:

```
libraryDependencies += "dev.zio" %% "zio-streams" % "2.1.26"
```

## Main

Your application can extend `ZIOAppDefault`, which provides a complete runtime system and allows you to write your whole program using ZIO:

```scala

object MyApp extends ZIOAppDefault {

  def run = myAppLogic

  val myAppLogic =
    for {
      _    <- printLine("Hello! What is your name?")
      name <- readLine
      _    <- printLine(s"Hello, ${name}, welcome to ZIO!")
    } yield ()
}
```
The `run` method should return a ZIO value which has all its errors handled,  
which, in ZIO parlance, is an unexceptional ZIO value.  

One way to do this is to invoke `fold` over a ZIO value, to get an unexceptional ZIO value.
That requires two handler functions: `eh: E => B` (the error handler) and `ah: A => B` (the success handler).

If `myAppLogic` fails, `eh` will be used to get from `e: E` to `b: B`;
if it succeeds, `ah` will be used to get from `a: A` to `b: B`. 

`myAppLogic`, as folded above, produces an unexceptional ZIO value, with `B` being `Int`.  
If `myAppLogic` fails, there will be a 1; if it succeeds, there will be a 0.

---

If you are integrating ZIO into an existing application, using dependency injection, or do not control your main function, then you can create a runtime system in order to execute your ZIO programs:

```scala

object IntegrationExample {
  val runtime = Runtime.default

  Unsafe.unsafe { implicit unsafe =>
    runtime.unsafe.run(ZIO.attempt(println("Hello World!"))).getOrThrowFiberFailure()
  }
}
```

Ideally, your application should have a _single_ runtime, because each runtime has its own resources (including thread pool and unhandled error reporter).

## Console

ZIO provides a module for interacting with the console. You can import the functions in this module with the following code snippet:

If you need to print text to the console, you can use `print` and `printLine`:

```scala

// Print without trailing line break
Console.print("Hello World")

// Print string and include trailing line break
Console.printLine("Hello World")
```

If you need to read input from the console, you can use `readLine`:

```scala

val echo = Console.readLine.flatMap(line => Console.printLine(line))
```

---

## Performance

ZIO is a high-performance framework that is powered by non-blocking fibers (which will move to _virtual threads_ under Loom).

ZIO's core execution engine minimizes allocations and automatically cancels all unused computation. All data structures included with ZIO are high-performance and non-blocking, and to the maximum extent possible on the JVM, non-boxing.

The `benchmarks` project has a variety of benchmarks that compare the performance of ZIO with other similar projects in the Scala and Java ecosystems, demonstrating 2-100x faster performance in some cases.

Benchmarks to compare the performance of HTTP, GraphQL, RDMBS, and other ZIO integrations can be found in those respective projects.

---

## Platforms

ZIO provides a consistent interface across platforms to the maximum extent possible, allowing developers to write code once and deploy it everywhere. However, there are some unavoidable differences between platforms to be aware of.

## JVM

ZIO supports Java versions 11 and above and Scala versions 2.12, 2.13, and 3.x.

On the JVM, effects may be executed on a blocking thread pool using methods like `ZIO.blocking` and `ZIO.attemptBlocking`. See the documentation on [Creating Effects](creating-effects.md) for further discussion on blocking operations.

## Scala.js

ZIO supports Scala.js 1.0.

While ZIO is a zero dependency library, some basic capabilities of the platform are assumed. In particular, due to the absence of implementations for certain `java.time` methods in Scala.js, users must bring their own `java.time` dependency. The one used by ZIO in its own internal test suites is [scala-java-time](https://github.com/cquiroz/scala-java-time). It can be added as a dependency like so:

```scala
libraryDependencies ++= Seq(
  "io.github.cquiroz" %%% "scala-java-time" % "2.2.0",
  "io.github.cquiroz" %%% "scala-java-time-tzdb" % "2.2.0"
)
```

Because of the single threaded execution model of Javascript, blocking operations are not supported on Scala.js. In addition, several other methods are not supported or are unsafe on Scala.js:

* The `readLine` method in the `Console` service is not supported because reading a line from the console blocks until input is received and the underlying method from the Scala standard library is not implemented on Scala.js.
* The synchronous execution methods on `Runtime` are not safe. All of these methods return a value synchronously and may require blocking if the effect includes asynchronous operations, including yield points introduced by the runtime to guarantee fairness. Users should use asynchronous execution methods instead.

## Scala Native

Support for Scala Native is currently experimental. More details will be added regarding support for the Scala Native platform when they are available.

---

## Running Effects

ZIO effects are precise plans that _describe_ a computation or interaction. Ultimately, every effect must be _executed_ by the ZIO runtime.

In this section, you will learn about the several ways that ZIO provides for you to execute effects in your application.

## App

If you construct a single effect for your whole program, the most natural way to run the effect is to extend `ZIOAppDefault`. 

This class provides Scala with a JVM-compatible main function, so it can be called from IDEs and launched from the command-line. All you have to do is implement the `run` method by returning the effect to run.

```scala

object MyApp extends ZIOAppDefault {

  def run =
    for {
      _    <- printLine("Hello! What is your name?")
      name <- readLine
      _    <- printLine(s"Hello, ${name}, welcome to ZIO!")
    } yield ()
}
```

If you are using a custom environment for your application, you will have to supply your environment to the effect (using `ZIO#provideEnvironment` or, if you are using layers, `ZIO#provide`) before you return it from `run`. 

`ZIOAppDefault` does not know how to supply custom environments.

## Default Runtime

Most applications are not greenfield, meaning they must integrate with legacy code and procedural libraries and frameworks.

In these cases, a better solution for running effects is to create a `Runtime`, which can be passed around and used to run effects wherever required.

ZIO contains a default runtime called `Runtime.default`.

To access it, merely use

```scala
val runtime = Runtime.default
```

Once you have a runtime, you can use it to execute effects:

```scala
Unsafe.unsafe { implicit unsafe =>
    runtime.unsafe.run(ZIO.attempt(println("Hello World!"))).getOrThrowFiberFailure()
}
```

In addition to `run`, which is for synchronous execution, there are other methods available on `Runtime` that support asynchronous execution.

## Custom Runtime

If you are using a custom environment for your application, you may find it useful to create a `Runtime` specifically tailored for that environment.

A custom `Runtime[R]` can be created with a `ZEnvironment[R]` (which holds the context required in order to execute your effects), as well as fiber refs and runtime flags (which can generally be set to default values).

For example, the following creates a `Runtime` that can provide an `Int` to effects :

```scala
val myRuntime: Runtime[Int] = 
  Runtime(ZEnvironment[Int](42), FiberRefs.empty, RuntimeFlags.default)
```

## Error Reporting

The ZIO runtime system automatically logs all errors encountered when executing your effects, so long as those errors are not handled by your ZIO code.

You can specify a custom logger easily using _ZIO Logging_, which can intercept these logged errors and handle them as configured by your logging backend.

## Next Steps

If you are comfortable with running effects, congratulations!

You are now ready to dive into other sections on the ZIO website covering data types, use cases, and interop with other systems. 

Refer to the Scaladoc for detailed documentation on all the core ZIO types and methods.

---

## Summary

ZIO is a next-generation framework for building cloud-native applications on the JVM. With a beginner-friendly yet powerful functional core, ZIO lets developers quickly build best-practice applications that are highly scalable, testable, robust, resilient, resource-safe, efficient, and observable.

At the heart of ZIO is a powerful data type called `ZIO`, which is the fundamental building block for every ZIO application.

## ZIO 

The `ZIO` data type is called a _functional effect_, and represents a unit of computation inside a ZIO application. Similar to a blueprint or a workflow, functional effects are precise plans that _describe_ a computation or interaction. When executed by the ZIO runtime system, a functional effect will either fail with some type of error, or succeed with some type of value.

Like the `List` data type, the `ZIO` data type is a _generic_ data type, and uses type parameters for improved type-safety. The `List` data type has a single type parameter, which represents the type of element that is stored in the `List`. The `ZIO` data type has three type parameters: `ZIO[R, E, A]`.

The type parameters of the `ZIO` data type have the following meanings:

 - **`R` - Environment Type**. The environment type parameter represents the type of contextual data that is required by the effect before it can be executed. For example, some effects may require a connection to a database, while others might require an HTTP request, and still others might require a user session. If the environment type parameter is `Any`, then the effect has no requirements, meaning the effect can be executed without first providing it any specific context.
 - **`E` - Failure Type**. The failure type parameter represents the type of error that the effect can fail with when it is executed. Although `Exception` or `Throwable` are common failure types in ZIO applications, ZIO imposes no requirement on the error type, and it is sometimes useful to define custom business or domain error types for different parts of an application. If the error type parameter is `Nothing`, it means the effect cannot fail.
 - **`A` - Success Type**. The success type parameter represents the type of success that the effect can succeed with when it is executed. If the success type parameter is `Unit`, it means the effect produces no useful information (similar to a `void`-returning method), while if it is `Nothing`, it means the effect runs forever, unless it fails.

 As several examples of how to interpret the types of ZIO effects:

  - An effect of type `ZIO[Any, IOException, Byte]` has no requirements, and when executed, such an effect may fail with a value of type `IOException`, or may succeed with a value of type `Byte`.
  - An effect of type `ZIO[Connection, SQLException, ResultSet]` requires a `Connection`, and when executed, such an effect may fail with a value of type `SQLException`, or may succeed with a value of type `ResultSet`.
  - An effect of type `ZIO[HttpRequest, HttpFailure, HttpSuccess]` requires an `HttpRequest`, and when executed, such an effect may fail with a value of type `HttpFailure`, or may succeed with a value of type `HttpSuccess`.

The environment type parameter is a _composite type parameter_, because sometimes, a single effect can require _multiple_ values of _different_ types. If you see that an effect has a type of `ZIO[UserSession with HttpRequest, E, A]` (Scala 2.x) or `ZIO[UserSession & HttpRequest, E, A]` (Scala 3.x), it means that the effect requires multiple contextual values before it can be executed.

Although this analogy is not precise, a ZIO effect can be thought of as a function:

```scala
R => Either[E, A]
```

This function requires an `R` and produces a failure of type `E` or a success value of type `A`. 

ZIO effects are not actually functions, of course, because they model complex computations and interactions, which may be asynchronous, concurrent, or resourceful.

## Type Aliases

The `ZIO` data type is the only effect type in ZIO. However, there are a family of type aliases that reduce the need to type:

 - `UIO[A]` — A type alias for `ZIO[Any, Nothing, A]`, representing an effect that has no requirements, cannot fail, and can succeed with an `A`.
 - `URIO[R, A]` — A type alias for `ZIO[R, Nothing, A]`, representing an effect that requires an `R`, cannot fail, and can succeed with an `A`.
 - `Task[A]` — A type alias for `ZIO[Any, Throwable, A]`, representing an effect that has no requirements, may fail with a `Throwable` value, or succeed with an `A`.
 - `RIO[R, A]` — A type alias for `ZIO[R, Throwable, A]`, representing an effect that requires an `R`, may fail with a `Throwable` value, or succeed with an `A`.
 - `IO[E, A]` — A type alias for `ZIO[Any, E, A]`, representing an effect that has no requirements, may fail with an `E`, or succeed with an `A`.

**Tips For Getting Started With Type Aliases**

If you are new to functional effects, we recommend starting with the `Task` type, which has a single type parameter and corresponds most closely to the `Future` data types built into the Scala and Java standard libraries.

If you are using _Cats Effect_ libraries, you may find the `RIO` type useful, since it allows you to thread context through third-party libraries.

No matter what type alias you use in your application, `UIO` can be useful for describing infallible effects, including those resulting from handling all errors.

Finally, if you are an experienced functional programmer, then direct use of the `ZIO` data type is recommended, although you may find it useful to create your own family of type aliases in different parts of your application.

## Next Steps

If you are comfortable with the `ZIO` data type, and its family of type aliases, the next step is learning how to [create effects](creating-effects.md).

---

## Architectural Patterns

In this section, we are going to talk about the design elements of a ZIO application and the ZIO idiomatic way of structuring codes to write ZIO applications.

## Onion Architecture

Onion architecture is a software architecture pattern that is used to create loosely coupled, maintainable, and testable applications by layering the application into a set of concentric circles.

- The innermost layer contains the domain model. Its language has the highest level of abstraction.
- From the very center, the language domain model is surrounded successively by other layers, each of which is more technical and has a lower level of abstraction than the previous one.
- The outermost layer contains the final language is the one that is closest to the environment in which the application is running. For example, the outermost layer could be the user interface, a web API, etc.

Onion architecture is based on the _inversion of control_ principle. So each layer is dependent on the underlying layer, but not on the layers above it. This means that the innermost layer is independent of any other layers.

In ZIO by taking advantage of both functional and object-oriented programming, we can implement onion architecture in a very simple and elegant way. To implement this architecture, please refer to the [Writing ZIO Services](../service-pattern/index.md) section which empowers you to create layers (services) in the onion architecture. In order to assemble all layers and make the whole application work, please refer to the [Dependency Injection In ZIO](../di/index.md) section.

## Streaming Architecture

Many reasons make streaming architecture a good choice for building applications:

- From the technical perspective when we are dealing with files, sockets, HTTP requests, databases, etc we are working with streams of data.
- In addition from a business standpoint, the area of data processing is growing rapidly and the need for processing continuous streams of data is increasing, such as real-time analytics, fraud detection, monitoring, social media platforms, financial trading, etc.

In such cases, we may decide to use streaming architecture. ZIO Streams is a library that provides a purely functional, composable, and type-safe way to work with streams of data. We can use ZIO Streams to model both stateful and stateless streaming data processing pipelines.

ZIO Streams is on top of ZIO. So we can think of `ZStream` as a specialized functional effect that has more power than `ZIO`. It is built on top of ZIO and supports backpressure using a pull-based model. To learn more about ZIO Streams, please refer to the [ZIO Streams](../stream/index.md) section.

## Sidecar Pattern

The sidecar pattern is a microservice architecture pattern that is used to separate cross-cutting concerns from the main business logic. It is a very useful pattern when we have to deal with concerns like logging, metrics, profiling, monitoring, etc. These concerns are not part of the main service logic, but they are important for the service to work correctly.

In ZIO, we can implement the sidecar pattern by using compositional apps, or by using the `bootstrap` layer.

### Composable ZIO Applications

In the following example, as we have multiple applications (`UserApp` and `DocumentApp`), we use compositional apps to implement this pattern:

```scala

object UserApp extends ZIOAppDefault {
  def run = Server.serve(userHttpApp).provide(Server.defaultWithPort(8080))
}

object DocumentApp extends ZIOAppDefault {
  def run = Server.serve(documentHttpApp).provide(Server.defaultWithPort(8081))
}

object Metrics extends ZIOAppDefault {
  private val metricsConfig = ZLayer.succeed(MetricsConfig(5.seconds))

  def run =
    Server
      .serve(
        Routes(Method.GET / "metrics" ->
          handler(ZIO.serviceWithZIO[PrometheusPublisher](_.get.map(Response.text)))
        )
      )
      .provide(
        Server.defaultWithPort(8082),
        metricsConfig,
        prometheus.publisherLayer,
        prometheus.prometheusLayer
      )
}

object MainApp extends ZIOApp.Proxy(UserApp <> DocumentApp <> Metrics)
```

### Bootstrap Layer

If we had only one application, we could use the `bootstrap` layer to implement this pattern:

```scala

object MetricsService {
  private val metricsConfig = ZLayer.succeed(MetricsConfig(5.seconds))

  private val exporter: ZLayer[PrometheusPublisher, Nothing, Unit] =
    ZLayer.fromZIO {
      Server
        .serve(
          Routes(Method.GET / "metrics" ->
            handler(ZIO.serviceWithZIO[PrometheusPublisher](_.get.map(Response.text)))
          )
        )
        .provideSome[PrometheusPublisher](Server.defaultWithPort(8081))
        .forkDaemon
        .unit
    }

  val layer: ZLayer[Any, Nothing, Unit] =
    ZLayer.make[Unit](
      exporter,
      metricsConfig,
      prometheus.publisherLayer,
      prometheus.prometheusLayer
    )

}

object UserAoo extends ZIOAppDefault {
  override val bootstrap = MetricsService.layer

  def run = Server.serve(userHttpApp).provideSome(Server.defaultWithPort(8080))
}
```

---

## Functional Design Patterns

When designing an API, there are patterns that are commonly used. In this section, we are going to talk about some of these patterns:

  1. Functional Data Modeling
  2. Functional Domain Modeling
     1. Declarative Encoding
     2. Executable Encoding

## Functional Data Modeling

Before we start talking about functional data modeling, let's first recap the object-oriented way of modeling data.

The essence of object-oriented data modeling is inheritance. We have classes, traits, abstract classes, and subtyping. The core idea is to describe the commonalities between different types of data in a base interface or abstract class, and then extend it to describe the differences in the subclasses; so each subclass has its type-specific details while sharing the commonalities with the base type:

```scala
abstract class Event {
  def id: String
  def timestamp: Long
}

case class ClickEvent(id: String, timestamp: Long, element: String) extends Event
case class ViewEvent(id: String, timestamp: Long, page: String) extends Event
```

There is some problem with this approach, let's iterate some of them:

One of the problems with this approach is that it is not a good fit when we want to write generic operations on the `Event` type. For example, if we want to write an operation that changes the timestamp of an event, we should match the input event and then do the transformation for each case. Then finally, we should use `asInstanceOf` to cast the result back to the original type:

```scala
def updateTimestamp[E <: Event](event: E, timestamp: Long): E =
  event match {
    case e: ClickEvent => e.copy(timestamp = timestamp).asInstanceOf[E]
    case e: ViewEvent => e.copy(timestamp = timestamp).asInstanceOf[E]
  }
```

This introduces a lot of boilerplate code. It also has a type-safety issue. If we forget to add all the cases for the match expression, the compiler will not be able to detect it.

In functional data modeling, we don't use inheritance. All tools we have are sum types and product types. By using these two types, we can mathematically model the data. Let's try to model the previous example using sum and product types:

```scala
case class Event(id: String, timestamp: Long, details: EventType)

sealed trait EventType
object EventType {
  final case class ClickEvent(element: String) extends EventType
  final case class ViewEvent(page: String) extends EventType
}
```

In this approach, we describe commonalities with product types, and differences with sum types. So we push `id`, `timestamp`, and `details` to the `Event` case class, and encode the type-specific details of the event in the `EventType` which is a sum type.

The product and sum types are called "Algebraic Data Types" (ADT). They are the building blocks of modeling data in functional programming:

- **Product types** are the cartesian product of the types they contain. For example, `Event` is the product of `String`, `Long`, and `EventType`. In scala, we use `case class` to model product types.
- **Sum types** are the disjoint union of the types they represent. For example, `EventType` is the either `ClickEvent` or `ViewEvent`. In scala 2, we use `sealed trait`s and In Scala 3, we use `enum`s to model sum types.

## Functional Domain Modeling

Functional domain modeling is the process of modeling solutions to problems in a specific domain using functional programming. It is a very broad topic, and we are not going to cover all the details here. However, we are going to talk about the general patterns that are commonly used in functional domain modeling, in a nutshell.

In functional programming, we have two primary tools:

1. Nouns (Data)
2. Verbs (Operators)

So to provide a solution to every domain problem, we should follow these steps:

1. **Extracting the Core Model**— First, we need to focus on extracting the "minimum" information required to describe the "solution" to the "fundamental problem" in that domain. So we should ask ourselves, "what is the most fundamental problem we have in this domain?".

2. **Providing Operators**— Once we find out what is the core model of our domain, we should provide a set of "orthogonal operators" that are going to provide a "solution" to the "complex problems" by combining sub-problems.

3. **Packaging the Data Type**— To achieve great modularity, we package both the "core model" and "operators" in one place which is called "data type".

4. **Defining Constructors**— Also to have a better ergonomic API, we put all solutions to the "basic and simple problems" in that domain in the companion object of the "data type".

The ZIO ecosystem defines all data types in such a way, including ZIO, Fiber, Reference, Stream, etc.

There are two main encoding styles in functional domain modeling:

1. **Executable Encoding** is a functional domain modeling where we use operators and constructors to provide the "final" solution to the domain problems.

2. **Declarative Encoding** is a functional domain modeling where we use constructors and operators to provide the "description" of the solution to the domain problems. Later, we interpret the description to get the "final" solution.

A real-life example will help you understand the differences between these two approaches. Let's say we want to write an effect system from scratch. To keep things simple we will create an effect system that only supports sequencial effects. So at the end, we would like to write a program like this:

```scala
object Main extends scala.App {
  val app =
    for {
      _    <- IO.succeed(print("Enter your name: "))
      name <- IO.succeed(scala.io.StdIn.readLine())
      _    <- IO.succeed(println(s"Hello, $name"))
    } yield ()

  app.unsafeRunSync()
}
```

### Executable Encoding

The executable encoding is straightforward. After defining the core model, we just need to think about the execution steps for each operator.

In the following example, we describe the core model as `IO` case class, which has a `thunk` of code that is going to be executed when we call `unsafeRunSync`: `case class IO[+A](private val thunk: () => A)`.

We should provide a set of operators: `IO#map`, `IO#flatMap`, and `IO#unsafeRunSync`. We have also one constructor: `IO.succeed`. Implementing these operators requires us to think about how they should actually executed to produce the desired result.

For example, when we implement `map`,  should execute the `thunk` of the current `IO` and then apply the `f` function to the result. The same applies to `flatMap` and `unsafeRunSync`:

```scala
final case class IO[+A](private val thunk: () => A) {
  def map[B](f: A => B): IO[B]         = IO.succeed(f(thunk()))
  def flatMap[B](f: A => IO[B]): IO[B] = IO.succeed(f(thunk()).unsafeRunSync())
  def unsafeRunSync(): A               = thunk()
}

object IO {
  def succeed[A](value: => A): IO[A] = IO(() => value)
}
```

Now we can run the greeting program with the above encoding.

### Declarative Encoding

In contrast to the executable encoding, the declarative encoding is lazy. This means that the definition of the language is separate from how it is interpreted.

In this example, we describe the problem of sequencing effects as a data type called `FlatMap` that contains two information: the first effect to be executed and the function that should be applied to the result of the first effect. To describe the thunk of code that will be executed by the interpreter, we create another data type called `Succeed`. Similarly, we need another data type called `SucceedNow` to describe the already evaluated values.

```scala
sealed trait IO[+A]
object IO {
  final case class SucceedNow[A](value: A)                    extends IO[A]
  final case class Succeed[A](thunk: () => A)                 extends IO[A]
  final case class FlatMap[A, B](io: IO[A], cont: A => IO[B]) extends IO[B]
}
```

Assume we have written the following program:

```scala
val app: IO[Int] = 
  for {
    a <- IO.succeed(scala.io.StdIn.readLine("Enter the first number: ").toInt)
    b <- IO.succeed(scala.io.StdIn.readLine("Enter the second number: ").toInt) 
  } yield (a + b)
```

This program will be translated into the following tree data structure using the declarative encoding:

```scala
val app: IO[Int] = 
  IO.FlatMap(
    IO.Succeed(() => scala.io.StdIn.readLine("Enter the first number: ").toInt),
    (a: Int) =>
      IO.FlatMap(
        IO.Succeed(() => scala.io.StdIn.readLine("Enter the second number: ").toInt),
        (b: Int) => IO.SucceedNow(a + b)
      )
  )
```

The only operator that deals with the interpretation of the program is `unsafeRunSync`. Let's see how the whole encoding looks like:

```scala
sealed trait IO[+A] { self =>
  def map[B](f: A => B): IO[B] = flatMap(f andThen IO.succeedNow)

  def flatMap[B](f: A => IO[B]): IO[B] = IO.FlatMap(self, f)

  def unsafeRunSync(): A = {
    type Cont = Any => IO[Any]

    def run(stack: List[Cont], currentIO: IO[Any]): A = {
      def continue(value: Any) =
        stack match {
          case ::(cont, next) => run(next, cont(value))
          case Nil            => value.asInstanceOf[A]
        }

      currentIO match {
        case IO.SucceedNow(value) => continue(value)
        case IO.Succeed(thunk)    => continue(thunk())
        case IO.FlatMap(io, cont) => run(stack appended cont, io)
      }
    }

    run(stack = Nil, currentIO = self)
  }
}

object IO {
  def succeedNow[A](value: A): IO[A] = IO.SucceedNow(value)
  def succeed[A](value: => A): IO[A] = IO.Succeed(() => value)

  final case class SucceedNow[A](value: A)                    extends IO[A]
  final case class Succeed[A](thunk: () => A)                 extends IO[A]
  final case class FlatMap[A, B](io: IO[A], cont: A => IO[B]) extends IO[B]
}
```

---

## Non-functional Requirements

## Introduction

Designing and architecting a software system is a complex task. We should consider both the functional and non-functional requirements of the system.

The functional requirements are the features of the system which are directly related to the business domain and its problems. They are the core of the system and the main reason why we are designing and building the application.

Non-functional requirements are characteristics of the system that are used to qualify it in terms of "what should the system be" rather than "what should the system do," e.g.:

  1. [Correctness](#1-correctness)
  2. [Testability](#2-testability)
  3. [Maintainability](#3-maintainability)
  4. [Low Latency](#4-low-latency)
  5. [High Throughput](#5-high-throughput)
  6. [Robustness](#6-robustness)
  7. [Resiliency](#7-resiliency)
  8. [Efficiency](#8-efficiency)
  9. [Developer Productivity](#9-developer-productivity)

In this article, from the perspective of application architecture, we are going to look at some design elements that we can apply to our ZIO applications to make them more ergonomic and maintainable.

## 1. Correctness

Correctness is the ability of a system to do what it is supposed to do. ZIO provides us correctness property through local reasoning because of referential transparency and its type-safety.

When we have referential transparency, we do not need to look at the whole program to understand the behavior of a piece of code. We can reason about the application behavior locally and then make sure that all components work together correctly, from button to top.

The type system of ZIO also prevents us to introduce common bugs at runtime. Here are two examples:

  1. **Resource Management**— When we have a ZIO effect that has a type of `ZIO[Scope, IOException, FileInputStream]`, we can be sure that this effect will open a resource, and we should care about closing it. So then by using `ZIO.scoped(effect)` we can be sure that the resource will be closed after the effect is executed and the type of effect will be changed to `ZIO[Any, IOException, FileInputStream`. To learn more about `ZIO.scoped` and resource management using `Scope`, please refer to the [Scope][11] of the [resource management][12].

  2. **Error Management**— In ZIO errors are typed, so we can describe all possible errors that can happen in our effect. And from the correctness perspective, the type system helps us to be sure we have handled all errors or not. For example, if we have an effect of type `ZIO[Any, IOException, FileInputStream]`, by looking at the effect type, we can be sure the effect is exceptional, and we should handle its error. To learn more about error management in ZIO, please refer to the [error management][13] section.

## 2. Testability

ZIO has a strong focus on testability which supports:

  1. Property-based Checking
  2. Testing Effectful and Asynchronous Codes
  3. Testing Passages of Time
  4. Sharing Layers Between Specs
  5. Resource Management While Testing
  6. Dynamic Test Generation
  7. Test Aspects (AOP)
  8. Non-flaky Tests

To learn more about testing in ZIO, please refer to the [testing][14] section.

## 3. Maintainability

When we use ZIO, we take advantage of both functional and object-oriented programming paradigms to make our code maintainable:

- By using functional programming we can make sure that our code is correct, readable, testable, and reusable.

- The object-oriented programming paradigm helps us to make our code well-organized and highly cohesive by using objects, packages, and modules.

The ZIO's support for type safety is another factor that makes our code maintainable, especially when we refactor our codes we can be sure that we are not breaking anything.

## 4. Low Latency

Latency is the time it takes for a request to be processed and a response to be returned. ZIO is designed to support low latency applications by providing various concurrency and parallelism tools such as `ZIO.foreachPar`, `Fiber`, `Promise`, `Ref`, `Queue`, etc. To learn more about concurrency and parallelism in ZIO, please refer to the [concurrency][15] section.

## 5. High Throughput

ZIO fibers are lightweight threads (green threads). They are very cheap to create and destroy. So we can potentially have thousands of fibers running in parallel on a single machine, which helps us to achieve high throughput:

```scala

object MainApp extends ZIOAppDefault {
  def doWork(n: Int): ZIO[Any, Nothing, Unit] =  ??? 
  
  def run =
    ZIO
      .foreach(1 to 100000)(n => doWork(n).fork)
      .flatMap(f => Fiber.collectAll(f).join)
}
```

Other than low-level concurrency tools like `Fiber`, `Promise`, `Ref`, etc., ZIO Streams is a high-level abstraction for processing high-throughput data streams:

```scala

object MainApp extends ZIOAppDefault {
  def doWork(n: Int): ZIO[Any, Nothing, Unit] =  ??? 

  def run =
    ZStream
      .fromIterable(1 to 100000)
      .mapZIOParUnordered(Int.MaxValue)(doWork)
      .runDrain
}
```

:::note
The above examples are just for demonstration purposes. In real-world applications, depending on the nature of the problem to reach a better performance it may be better to control the level of parallelism instead of using unbounded parallelism.
:::

Another factor that helps us to achieve high throughput is the fact that we may have high workloads for some periods. In such cases, we can benefit from buffering the incoming requests instead of rejecting them and trying to process them later. We can use [`Queue`][16] for this purpose or the [`ZStream#buffer` operator][17].

To learn more about ZIO Streams, please refer to the [ZIO Streams][18] section.

## 6. Robustness

With the help of ZIO's error channel, we can write applications whose errors are fully specified and handled at the compile time. Having this feature helps us to make our applications more robust.

It also gives us the ability to lossless translation of errors from one domain to another. For example, when writing a web application, we can reliably translate errors inside the application to HTTP response codes. ZIO uses the compile to ensure that we have mapped all possible errors to HTTP response codes.

To learn more about error management in ZIO, please refer to the [error management][13] section.

## 7. Resiliency

For resiliency, we can use ZIO's retry operator along with the retry policy to make our application resilient to failures. `Schedule` is a powerful composable data type that helps us to compose multiple policies together and make a complex retry policy:

```scala

object MainApp extends ZIOAppDefault {
  sealed trait DownloadError    extends Throwable
  case object BandwidthExceeded extends DownloadError
  case object NetworkError      extends DownloadError

  // flaky api
  def download(id: String): ZIO[Any, DownloadError, Array[Byte]] = ???

  def isRecoverable(e: DownloadError): Boolean =
    e match {
      case BandwidthExceeded => false
      case NetworkError      => true
    }

  val policy =
    (Schedule.recurs(20) &&
      Schedule.exponential(100.millis))
      .whileInput(isRecoverable)

  def run = download("123").retry(policy)
}
```

To learn more about resiliency and scheduling in ZIO, please refer to the [resiliency][19] section.

## 8. Efficiency

ZIO is designed to be extraordinarily efficient. Let's take a look at some of the features that make ZIO efficient:

1. ZIO Streams are pull-based, so the source of the stream starts producing elements only when the stream is consumed. This lazy semantic helps us to avoid unnecessary work and save resources:

```scala
def downloadAsCsv(id: String): ZStream[Db, IOException, Byte] =
  jdbc
    .selectMany(sql"SELECT * FROM events WHERE userId = $id")
    .map(toCSV)
    .via(ZPipeline.utf8Encode)
    .via(ZPipeline.gzip)
```

In the above example, tries to consume a minimum amount of computation that is necessary. So if we use this workflow in a web application, when the client downloads half of the CSV file, only half of the data will be pulled from the database. So we can save resources and infrastructure costs.

2. ZIO is designed to be interruptible (unlike the `Future` in Scala). So we can cancel any running effect at any time. This feature enables us to have efficient high-level operators such as `ZIO#race` on top of the ZIO interruption model. With `race` we can run two different workflows in parallel and the loser of the workflow will be canceled:

```scala
val loaded = loadFromCache(productId).race(loadFromDb(productId))
```

Or if we do a bunch of things in parallel and one of those things fails, all the other ones which are currently running in parallel will be canceled automatically:

```scala
val aggregated =
  ZIO.foreach(account.statements) { statement =>
    downloadStatement(statement.s3Bucket) 
  }.map(aggregateStatements(_))
```

If we timeout a workflow in ZIO, once the timeout is reached, the workflow will be canceled automatically:

```scala
val timedOut = aggregated.timeout(10.seconds)
```

So in the above example, all running workflows will be simultaneously canceled once the timeout is reached and all resources will be released.

3. Another ZIO feature that helps us to have efficient workflows is its resource management. ZIO provides a great model for resource management with the help of the `Scope` data type. `Scope` is a contextual data type that whenever appears in the environment of an effect, denotes this effect will open one or more resources. Using `ZIO.scoped` we can ensure that all resources enclosed in this operator will be automatically released once the effect is completed or interrupted:

```scala

def source(name: String): ZIO[Scope, Throwable, BufferedSource] =
  ZIO.acquireRelease(ZIO.attemptBlocking(scala.io.Source.fromFile(name)))(s => ZIO.succeedBlocking(s.close()))

val fileContent: ZIO[Any, Throwable, String] =
  ZIO.scoped {
    source("file.txt").map(_.getLines()).map(_.mkString("\n"))
  }
```

In the above example, if we use the `fileContent` effect, we can be sure that the file handler will be released regardless of whether the effect is completed or interrupted.

To learn more about resource management in ZIO, please refer to the [resource management][11] section.

## 9. Developer Productivity

Developer experience and productivity are very important for choosing a technology for any large-scala and long-running project. Let's take a look at some features that make ZIO a great fit for developer productivity:

  1. Referential Transparency and Purity
  2. Composable Data Types
  3. Type-safety and Compile time Error Checking
  4. Easy to Refactor
  5. Discoverability
      1. Dot completion when developing with IDEs
      2. Consistent naming conventions
  6. Concise and Expressive API with Minimal Boilerplate
  7. Expressive Compiler Errors
  8. Empowering Meta-programming and Macros
  9. [Maintainability](#3-maintainability)
  10. Observability
      - [Logging][1]
      - [Tracing][2]
      - [Metrics][3]
  11. [Debugging Facilities][4]
  12. [Compile-time Execution Tracing][5]
  13. [Automatic Dependency Graph Generation][6]
  14. [Testability][7]
  15. [Programming Without Type Classes][8]
  16. Rich Ecosystem
      - Massive Amount of Libraries and Tools on JVM
      - [ZIO Official libraries][9]
      - [ZIO community libraries][10]

[1]: ../../zio-logging/index.md
[2]: ../../zio-telemetry/index.md
[3]: ../observability/metrics/index.md
[4]: ../../guides/migrate/migration-guide.md#debugging
[5]: ../../guides/migrate/migration-guide.md#compile-time-execution-tracing
[6]: ../di/automatic-layer-construction.md
[7]: ../test/index.md
[8]: https://www.youtube.com/watch?v=QDleESXlZJw
[9]: ../../ecosystem/officials/index.md
[10]: ../../ecosystem/community/index.md
[11]: ../resource/scope.md
[12]: ../resource/index.md
[13]: ../error-management/index.md
[14]: ../test/index.md
[15]: ../concurrency/index.md
[16]: ../concurrency/queue.md
[17]: ../stream/zstream/operations.md#buffering
[18]: ../stream/index.md
[19]: ../schedule/index.md

---

## Programming Paradigms in ZIO

It is important to realize that the programming paradigm used to write a software system has a significant impact on its design and architecture. In this section, we are going to talk the foundation of ZIO from the programming paradigm perspective:

- Functional and object-oriented programming
- Imperative and Declarative Programming
- Structured Programming
- Aspect-oriented Programming

## Functional and Object-Oriented Programming

Every computer program is written in the form of a set of operations and data structures:

- A data are nouns in the program, such as `Person`, `Address`, `Order`, etc. They represent a piece of information, configuration, or state that is used by operations.
- Operations are verbs, such as `createOrder`, `updateOrder`, `deleteOrder`, and etc. They are methods or functions that operate on data.

The way we organize these two elements in our program determines the programming paradigm we use; object-oriented programming (OOP) or functional programming (FP).

### Object-Oriented Programming

In object-oriented programming, we organize our program by bundling related data and operations into a single unit called an object. Each object has its own state and behavior. This is the fundamental construct of object-oriented programming. All other constructs like classes, interfaces, inheritances, subtyping are built around this concept.

Therefore, the object is the only option we have in object-oriented programming to organize our programs. We have only one hammer for all classes of design problems.

So, we can say that the most important result of object-oriented programming is modularity. We can package related data and operations into a single unit and reuse them in other parts of our program.

#### Classes

In statical-typed object-oriented programming, we can define a "category" of objects by using a class. A class is a blueprint that defines the structure of all objects in that category. A class is basically define the whole class of objects that all have similarities.

We can create an instance of a class by using the `new` keyword. This instance is called an object.

#### Subtyping

Another great feature of object-oriented programming is sub-typing. We can define a new class that inherits from an existing class. This new class is called a sub-class or a child class, and the existing class is called a super-class or a parent class. Using sub-typing, we can define a whole class of objects and then classify it into sub-classes.

#### Interfaces and Polymorphism

In object-oriented programming, we can also define an interface. An interface is a contract that defines the behavior that is shared by all classes that implement that interface.

Using interfaces, we can achieve polymorphism when writing services. For example, we can define a `Logger` interface that defines the `log` method. Then we can define a `ConsoleLogger`, `FileLogger`, or `JsonLogger` that all implement the same `Logger` interface. This way, we can use the same `Logger` interface to inject different implementations of the `Logger` interface into our services.

### Functional Programming

In the previous section, we discussed object-oriented programming and saw that the object is the basis of object-oriented programming. Let's talk about functional programming now and see what its basis is.

A functional program is modeled as a set of mathematical functions. By mathematical functions, we mean those that take an immutable input and produce an immutable output while having referential transparency.

Functions are the basis of FP, and the basis of functions is the lambda. Lambdas are functions that can be passed as arguments to other functions or returned as results. So we can say that the lambda is a fancy term for first-class functions (functions as values).

In contrast to object-oriented programming, functional programming separates data and operations into two different worlds. Data is immutable, and operations are pure functions. This separation of data and operations is the fundamental philosophy of functional programming.

In FP, we have only two building blocks to model our programs:

- Data (Algebraic data types)
- Operations (Functions)

We describe our data (input and output) using constructs called "algebraic data types" which for Scala programmers means _sealed traits_ (or enums) and ـcase classesـ. So there is two building blocks for describing data in FP. To describe operations, we have functions.

Separation of data and operations is the fundamental concept of functional programming. In contrast to object-oriented programming, we have no tools for abstraction, modularity, interfaces, and polymorphism.

### Embracing Both Functional and Object-Oriented Programming

ZIO is a functional programming library which also brings the power of object-oriented programming into the functional world. It tries to combine the best of both worlds.

We use FP to achieve **code maintainability** and OOP to achieve **code organization**:

1. Functional programming gives us the following tools in terms of **code maintainability**:

    - **Data Modeling** using Algebraic Data Types
    - **Functional Design** using functions to create Domain Specific Languages (DSLs)
    - **Composability** using Pure and Referentially Transparent Functions

2. Object-oriented programming gives us the following tools in terms of **code organization**:

    - **Methods** which help us to operate on a specific data attached to an object
    - **Constructors** which help us to create a new instance of a data type
    - **Modules** which allows us to bundle together related operations into a single unit

So, we leverage the power of both FP and OOP to build a better software system in ZIO.

## Imperative and Declarative Programming

Another important aspect of programming paradigms is the difference between imperative and declarative programming.

In imperative programming, we describe the steps ("How") the computer should take to solve a problem. In declarative programming, we describe the problem itself ("What") and let the computer figure out the steps to solve it.

Although ZIO has a declarative API in terms of a functional effect model —the ZIO runtime interprets the program as a set of effects and decides what steps to take to execute it—, it is imperative in comparison to other libraries like [Libretto](https://github.com/TomasMikula/libretto).

## Structured Programming

The idea of structured concurrency is based on the structured programming paradigm. So, let's talk about structured programming first.

In the early days of programming, we used to write programs in a linear manner. Program were composed of a series of instructions that executed one by one. Using goto statements, we could jump to any part of the program and change its execution flow. This was the first programming paradigm called procedural programming. Writing programs in such a linear fashion was error-prone and hard to maintain. Such a program was also complicated to read, understand, and reason about.

Structured programming paradigms were introduced to solve this problem. Structured programming uses control structures like "if-then-else" to make the program flow more logical. Without these control structures, we cannot jump to any part of the program.

In structured programming, we use control structures to organize our code into blocks. These blocks are called "structured blocks" and are the building blocks of structured programming.

A structured control flow makes nested blocks of code with clear boundaries. Each new block of code has its own scope where all objects defined in that block are only visible inside that block. As a result, objects are bound to their enclosing blocks for their lifetime. Having clear scopes and lifetimes of objects make it easier to understand the control flow of the program.

ZIO embraces the structured programming into the next level by using this paradigm in other areas of programming such as [structured concurrency](../fiber/index.md#structured-concurrency), [scope based resource management](../resource/scope.md), and also regional interruption model.

## Aspect Oriented Programming

Aspect Oriented Programming (AOP) is a programming paradigm that allows us to separate cross-cutting concerns from the main program logic. Cross-cutting concerns are those that are not directly related to the main program logic but are still important to the program. Examples of cross-cutting concerns are logging, tracing, metrics, and security.

ZIO embraces AOP by providing `ZIOAspect` in the "core" and `TestAspect` in the "test" module. Using these two data types, we can write aspects that can be applied to any ZIO effect or test.

---

## Hub

A `Hub` is an asynchronous message hub. Publishers can publish messages to the hub and subscribers can subscribe to receive messagesfrom the hub.

Unlike a `Queue`, where each value offered to the queue can be taken by _one_ taker, each value published to a hub can be received by _all_ subscribers. Whereas a `Queue` represents the optimal solution to the problem of how to _distribute_ values, a `Hub` represents the optimal solution to the problem of how to _broadcast_ them.

The fundamental operators on a `Hub` are `publish` and `subscribe`:

```scala

trait Hub[A] {
  def publish(a: A): UIO[Boolean]
  def subscribe: ZIO[Scope, Nothing, Dequeue[A]]
}
```

The `publish` operator returns a `ZIO` effect that publishes a message of type `A` to the hub and succeeds with a value describing whether the message was successfully published to the hub.

The `subscribe` operator returns a scoped `ZIO` effect that subscribes to the hub and unsubscribes from the hub when the scope is closed. Within the scope we have access to a `Dequeue`, which is a `Queue` that can only be dequeued from, that allows us to take messages published to the hub.

For example, we can use a hub to broadcast a message to multiple subscribers like this:

```scala
Hub.bounded[String](2).flatMap { hub =>
  ZIO.scoped {
    hub.subscribe.zip(hub.subscribe).flatMap { case (left, right) =>
      for {
        _ <- hub.publish("Hello from a hub!")
        _ <- left.take.flatMap(Console.printLine(_))
        _ <- right.take.flatMap(Console.printLine(_))
      } yield ()
    }
  }
}
```

A subscriber will only receive messages that are published to the hub while it is subscribed. So if we want to make sure that a particular message is received by a subscriber we must take care that the subscription has completed before publishing the message to the hub.

We can do this by publishing a message to the hub within the scope of the subscription as in the example above or by using other coordination mechanisms such as completing a `Promise`  when scope has been opened.

Of course, in many cases such as subscribing to receive real time data we may not care about this because we are happy to just pick up with the most recent messages after we have subscribed. But for testing and simple applications this can be an important point to keep in mind.

## Constructing Hubs

The most common way to create a hub is with the `bounded` constructor, which returns an effect that creates a new hub with the specified requested capacity.

```scala
def bounded[A](requestedCapacity: Int): UIO[Hub[A]] =
  ???
```

For maximum efficiency you should create hubs with capacities that are powers of two.

Just like a bounded queue, a bounded hub applies back pressure to publishers when it is at capacity, so publishers will semantically block on calls to `publish` if the hub is full.

The advantage of the back pressure strategy is that it guarantees that all subscribers will receive all messages published to the hub while they are subscribed. However, it does create the risk that a slow subscriber will slow down the rate at which messages are published and received by other subscribers.

If you do not want this you can create a hub with the `dropping` constructor.

```scala
def dropping[A](requestedCapacity: Int): UIO[Hub[A]] =
  ???
```

A dropping hub will simply drop values published to it if the hub is at capacity, returning `false` on calls to `publish` if the hub is full to signal that the value was not successfully published.

The advantage of the dropping strategy is that publishers can continue to publish new values so when there is space in the hub the newest values can be published to the hub. However, subscribers are no longer guaranteed to receive all values published to the hub and a slow subscriber can still prevent messages from being published to the hub and received by other subscribers.

You can also create a hub with the `sliding` constructor.

```scala
def sliding[A](requestedCapacity: Int): UIO[Hub[A]] =
  ???
```

A sliding hub will drop the oldest value if a new value is published to it and the hub is at capacity, so publishing will always succeed immediately.

The advantage of the sliding strategy is that a slow subscriber cannot slow down that rate at which messages are published to the hub or received by other subscribers. However, it creates the risk that slow subscribers may not receive all messages published to the hub.

Finally, you can create a hub with the `unbounded` constructor.

```scala
def unbounded[A]: UIO[Hub[A]] =
  ???
```

An unbounded hub is never at capacity so publishing to an unbounded hub always immediately succeeds.

The advantage of an unbounded hub is that it combines the guarantees that all subscribers will receive all messages published to the hub and that a slow subscriber will not slow down the rate at which messages are published and received by other subscribers. However, it does this at the cost of potentially growing without bound if messages are published to the hub more quickly than they are taken by the slowest subscriber.

In general you should prefer bounded, dropping, or sliding hubs for this reason. However, unbounded hubs can be useful in certain situations where you do not know exactly how many values will be published to the hub but are confident that it will not exceed a reasonable size or want to handle that concern at a higher level of your application.

## Operators On Hubs

In addition to `publish` and `subscribe`, many of the same operators that are available on queues are available on hubs.

We can publish multiple values to the hub using the `publishAll` operator.

```scala
trait Hub[A] {
  def publishAll(as: Iterable[A]): UIO[Boolean]
}
```

We can check the capacity of the hub as well as the number of messages currently in the hub using the `size` and `capacity` operators.

```scala
trait Hub[A] {
  def capacity: Int
  def size: UIO[Int]
}
```

Note that `capacity` returns an `Int` because the capacity is set at hub creation and never changes. In contrast, `size` returns a `ZIO` effect that determines the current size of the hub since the number of messages in the hub can change over time.

We can also shut down the hub, check whether it has been shut down, or await its shut down. Shutting down a hub will shut down all the queues associated with subscriptions to the hub, properly propagating the shut down signal.

```scala
trait Hub[A] {
  def awaitShutdown: UIO[Unit]
  def isShutdown: UIO[Boolean]
  def shutdown: UIO[Unit]
}
```

As you can see, the operators on `Hub` are identical to the ones on `Queue` with the exception of `publish` and `subscribe` replacing `offer` and `take`. So if you know how to use a `Queue` you already know how to use a `Hub`.

In fact, a `Hub` can be viewed as a `Queue` that can only be written to.

```scala
trait Hub[A] extends Enqueue[A]
```

Here the `Enqueue` type represents a queue that can only be enqueued. Enqueing to the queue publishes a value to the hub, shutting down the queue shuts down the hub, and so on.

This can be extremely useful because it allows us to use a `Hub` anywhere we are currently using a `Queue` that we only write to.

For example, say we are using the `into` operator on `ZStream` to send all elements of a stream of financial transactions to a `Queue` for processing by a downstream consumer.

```scala

trait ZStream[-R, +E, +O] {
  def into(
    queue: Enqueue[Take[E, O]]
  ): ZIO[R, E, Unit]
}
```

We would now like to have multiple downstream consumers process each of these transactions, for example to persist them and log them in addition to applying our business logic to them. With `Hub` this is easy because we can just use the `toQueue` operator to view any `Hub` as a `Queue` that can only be written to.

```scala
type Transaction = ???

val transactionStream: ZStream[Any, Nothing, Transaction] =
  ???

val hub: Hub[Take[Nothing, Transaction]] =
  ???

transactionStream.into(hub)
```

All of the elements from the transaction stream will now be published to the hub. We can now have multiple downstream consumers process elements from the financial transactions stream with the guarantee that all downstream consumers will see all transactions in the stream, changing the topology of our data flow from one-to-one to one-to-many with a single line change.

## Hubs And Streams

Hubs play extremely well with streams.

We can create a `ZStream` from a subscription to a hub using the `fromHub` operator.

```scala

object ZStream {
  def fromHub[O](hub: Hub[O]): ZStream[Any, Nothing, O] =
    ???
}
```

This will return a stream that subscribes to receive values from a hub and then emits every value published to the hub while the subscription is active. When the stream ends the subscriber will automatically be unsubscribed from the hub.

There is also a `fromHubScoped` operator that returns the stream in the context of a scoped effect.

```scala
object ZStream {
  def fromHubScoped[O](
    hub: Hub[O]
  ): ZIO[Scope, Nothing, ZStream[Any, Nothing, O]] =
    ???
}
```

The scoped effect here describes subscribing to receive messages from the hub while the stream describes taking messages from the hub. This can be useful when we need to ensure that a consumer has subscribed before a producer begins publishing values.

Here is an example of using it:

```scala
for {
  promise <- Promise.make[Nothing, Unit]
  hub     <- Hub.bounded[String](2)
  scoped  = ZStream.fromHubScoped(hub).tap(_ => promise.succeed(()))
  stream   = ZStream.unwrapScoped(scoped)
  fiber   <- stream.take(2).runCollect.fork
  _       <- promise.await
  _       <- hub.publish("Hello")
  _       <- hub.publish("World")
  _       <- fiber.join
} yield ()
```

Notice that in this case we used a `Promise` to ensure that the subscription had completed before publishing to the hub. The scoped `ZIO` in the return type of `fromHubScoped` made it easy for us to signal when the subscription had occurred by using `tap` and completing the `Promise`.

Of course in many real applications we don't need this kind of sequencing and just want to subscribe to receive new messages. In this case we can use the `fromHub` operator to return a `ZStream` that will automatically handle subscribing and unsubscribing for us.

There is also a `fromHubWithShutdown` variant that shuts down the hub itself when the stream ends. This is useful when the stream represents your main application logic and you want to shut down other subscriptions to the hub when the stream ends.

Each of these constructors also has `Chunk` variants, `fromChunkHub` and `fromChunkHubWithShutdown`, that allow you to preserve the chunked structure of data when working with hubs and streams.

In addition to being able to create streams from subscriptions to hubs, there are a variety of ways to send values emitted by streams to hubs to build more complex data flow graphs.

The simplest of these is the `toHub` operator, which constructs a new hub and publishes each element emitted by the stream to that hub.

```scala
trait ZStream[-R, +E, +O] {
  def toHub[E1 >: E, O1 >: O](
    capacity: Int
  ): ZIO[R with Scope, Nothing, Hub[Take[E1, O1]]]
}
```

The hub will be constructed with the `bounded` constructor using the specified capacity.

If you want to send values emitted by a stream to an existing hub or a hub created using one of the other hub constructors you can use the `runIntoHub` operator.

```scala
trait ZStream[-R, +E, +O] {
  def runIntoHub[E1 >: E, O1 >: O](
    hub: => Hub[Take[E1, O1]]
  ): ZIO[R, E1, Unit]
}
```

There is an `runIntoHubScoped` variant of this if you want to send values to the hub in the context of a `Scope`.

Here is the example above adapted to publish values from a stream to the hub:

```scala
for {
  promise <- Promise.make[Nothing, Unit]
  hub     <- Hub.bounded[Take[Nothing, String]](2)
  scoped  = ZStream.fromHubScoped(hub).tap(_ => promise.succeed(()))
  stream   = ZStream.unwrapScoped(scoped).flattenTake
  fiber   <- stream.take(2).runCollect.fork
  _       <- promise.await
  _       <- ZStream("Hello", "World").runIntoHub(hub)
  _       <- fiber.join
} yield ()
```

Notice that we created a `Hub` of `Take` values this time. `Take` is an algebraic data type that represents the different potential results of pulling from a stream, including the stream emitting a chunk of values, failing with an error, or being done.

Here we automatically unwrapped the `Take` values using the `flattenTake` operator on `ZStream`. In other cases where the subscriber was not a `ZStream` the `Take` value would allow the subscriber to observe whether the stream had emitted a value, failed with an error, or ended, and handle it appropriately.

You can also create a sink that sends values to a hub.

```scala
object ZSink {
  def fromHub[I](
    hub: Hub[I]
  ): ZSink[Any, Nothing, I, Nothing, Unit] =
    ???
}
```

The sink will publish each value sent to the sink to the specified hub. Again there is a `fromHubWithShutdown` variant that will shut down the hub when the stream ends.

Finally, `Hub` is used internally to provide a highly efficient implementation of the `broadcast` family of operators, including `broadcast` and `broadcastDynamic`.

```scala
trait ZStream[-R, +E, +O] {
  def broadcast(
    n: Int,
    maximumLag: Int
  ): ZIO[R with Scope, Nothing, List[ZStream[Any, E, O]]]
  def broadcastDynamic(
    maximumLag: Int
  ): ZIO[R with Scope, Nothing, ZIO[Scope, Nothing, ZStream[Any, E, O]]]
}
```

The `broadcast` operator generates the specified number of new streams and broadcasts each value from the original stream to each of the new streams. The `broadcastDynamic` operator returns a new `ZIO` value that you can use to dynamically subscribe and unsubscribe to receive values broadcast from the original stream.

You don't have to do anything with `Hub` to take advantage of these operators other than enjoy their optimized implementation in terms of `Hub`.

With `broadcast` and other `ZStream` operators that model distributing values to different streams and combining values from different streams it is straightforward to build complex data flow graphs, all while being as performant as possible.

---

## Introduction to Concurrent Programming in ZIO

## Overview

Most of the time in concurrent programming we have a single state that we need to read and update concurrently. When we have multiple fibers reading or writing to the same memory location we encounter the race condition. The main goal in every concurrent program is to have a consistent view of states among all threads.

There are two major concurrency models which try to solve this problem:

1. **Shared State** — In this model, all threads communicate with each other by sharing the same memory location.

2. **Message Passing (Distributed State)** — This model provides primitives for sending and receiving messages, and the state is distributed. Each thread of execution has its own state. 

The _Shared State_ model has two main solutions:

1. **Lock-based** — In the locking model, the general primitives for synchronization are _locks_ that control access to critical sections. When a thread wants to modify the critical section, it acquires the lock and says _I'm the only thread that is allowed to modify the state right now_, and after its work finished it unlocks the critical section and says _I'm done, any other thread can modify this memory section_.

2. **Non-blocking** — Non-blocking algorithms usually use hardware-intrinsic atomic operations like `compare-and-swap` (CAS), without using any locks. This method follows an optimistic design with a transactional memory mechanism to roll back in conflict situations.

## Implications of Locking Mechanism

There are several drawbacks with lock-based concurrency:

1. Incorrect use of locks can lead to deadlocks. We need to care about the locking orders. If we don't place the locks in the right order, we may encounter a deadlock situation.

2. Identifying the critical section of code that is vulnerable to race conditions is overwhelming. We should always care about them and remember to lock everywhere it's required.

3. It makes our software design very sophisticated to become scalable and reliable. It doesn't scale with program size and complexity.

4. To prevent missing the releasing of the acquired locks, we should always care about exceptions and error handling inside locking sections. 

5. The locking mechanism violates the encapsulation property of the pieces of our programs. So systems that are built on a locking mechanism are difficult to compose without knowing about their internals.

## Lock-free Concurrency Model

As the lock-oriented programming does not compose and has lots of drawbacks, ZIO uses a _lock-free concurrency model_ which is a variation of non-blocking algorithms. The magic behind all of ZIO concurrency primitives is that they use the CAS (_compare-and-set_) operation. 

Let's see how the `modify` function of `Ref` is implemented without any locking mechanism:

```scala
case class Ref[A](value: AtomicReference[A]) { self =>
  def modify[B](f: A => (B, A)): UIO[B] = ZIO.succeed {
    var loop = true
    var b: B = null.asInstanceOf[B]
    while (loop) {
      val current = value.get
      val tuple   = f(current)
      b = tuple._1
      loop = !value.compareAndSet(current, tuple._2)
    }
    b
  }
}
```

The idea behind the `modify` is that a variable is only updated if it still has the same value as the time we had read the value from the original memory location. If the value has changed, it retries in the while loop until it succeeds. 

## Advantage of Using ZIO Concurrency

Let's point out the key properties of the ZIO concurrency model:

1. **Composable** — Due to the use of the lock-free concurrency model, ZIO brings us composable concurrency primitives and lots of great combinators in a declarative style.

:::note
`Ref` and `Promise` and subsequently all other ZIO concurrency primitives that are on top of these two basic primitives **are not _transactionally_ composable**.

We cannot do transactional changes across two or more of such concurrency primitives. They are susceptible to race conditions and deadlocks. **So don't use them if you need to perform an atomic operation on top of a composed sequence of multiple state-changing operations. In such a case use [`STM`](../stm/index.md) instead**. 
:::

2. **Non-blocking** — All of the ZIO primitives are a hundred percent asynchronous and nonblocking.

3. **Resource Safety** — ZIO concurrency model comes with strong guarantees of resource safety. If any interruption occurs in between concurrent operations, it won't leak any resource. So it allows us to write compositional operators like timeout and racing without worrying about any leaks.

## Concurrency Primitives

Let's take a quick look at ZIO concurrency primitives, what they are and why they exist.

### Basic Operations

`Ref` and `Promise` are the two simple concurrency primitives which provide an orthogonal basis for building concurrency structures. They are assembly language of other concurrent data structures:

- **[Ref](ref.md)** — `Ref` and all its variant like [`Ref.Synchronized`](refsynchronized.md) are building blocks for writing concurrent stateful applications. Anytime we need to share information between multiple fibers, and those fibers have to update the same information, they need to communicate through something that provides the guarantee of atomicity. So all of these `Ref` primitives are atomic and thread-safe. They provide us a reliable foundation for synchronizing concurrent programs.

- **[Promise](promise.md)** — A `Promise` is a model of a variable that may be set a single time, and awaited on by many fibers. This primitive is very useful when we need some point of synchronization between two or multiple fibers.

By using these two simple primitives, we can build lots of other asynchronous concurrent data structures like `Semaphore`, `Queue` and `Hub`.

### Others

- **[Semaphore](semaphore.md)** — A `Semaphore` is an asynchronous (non-blocking) semaphore that plays well with ZIO's interruption. `Semaphore` is a generalization of a mutex. It has a certain number of permits, which can be held and released concurrently by different parties. Attempts to acquire more permits than available result in the acquiring fiber being suspended until the specified number of permits become available.

- **[Queue](queue.md)** — A `Queue` is an asynchronous queue that never blocks, which is safe for multiple concurrent producers and consumers.

- **[Hub](hub.md)** - A `Hub` is an asynchronous message hub that allows publishers to efficiently broadcast values to many subscribers.

---

## Promise

A `Promise[E, A]` is a variable of `IO[E, A]` type that can be set exactly once.

Promise is a **purely functional synchronization primitive** which represents a single value that may not yet be available. When we create a Promise, it always started with an empty value, then it can be completed exactly once at some point, and then it will never become empty or modified again.

Promise is a synchronization primitive. So, it is useful whenever we want to wait for something to happen. Whenever we need to synchronize a fiber with another fiber, we can use Promise. It allows us to have fibers waiting for other fibers to do things. Any time we want to hand over work from one fiber to another fiber or anytime we want to suspend a fiber until some other fiber does a certain amount of work, we may want to use a Promise. Also, we can use `Promise` with `Ref` to build other concurrent primitives, like Queue and Semaphore.

By calling `await` on the Promise, the current fiber blocks until that event happens. As we know, blocking fibers in ZIO don't actually block kernel threads. They are just blocking semantically, so when a fiber is blocked the underlying thread is free to run all other fibers.

Promise is the equivalent of Scala's Promise. It's almost the same, except it has two type parameters, instead of one. Also instead of calling `future`, we need to call `await` on ZIO Promise to wait for the Promise to be completed.

Promises can be failed with a value of type `E` or succeeded with a value of type `A`. So there are two ways we can complete a Promise, with failure or success, and whoever is waiting on the Promise will get back that failure or success. 

## Operations

### Creation

Promises can be created using `Promise.make[E, A]`, which returns `UIO[Promise[E, A]]`. This is a description of creating a promise, but not the actual promise. Promises cannot be created outside of IO, because creating them involves allocating mutable memory, which is an effect and must be safely captured in IO.

### Completing

You can complete a `Promise[E, A]` in different ways:
* successfully with a value of type `A` using `succeed`
* with `Exit[E, A]` using `done` - each `await` will get this exit propagated
+ with result of effect `IO[E, A]` using `complete` - the effect will be executed once and the result will be propagated to all waiting fibers
* with effect `IO[E, A]` using `completeWith` - first fiber that calls `completeWith` wins and sets the effect that **will be executed by each `await`ing fiber**, so be careful when using `p.completeWith(someEffect)` and rather use `p.complete(someEffect)` unless executing `someEffect` by each `await`ing fiber is intended
* simply fail with `E` using `fail`
* simply defect with `Throwable` using `die`
* fail or defect with `Cause[E]` using `failCause`
* interrupt with `interrupt`

Following example shows usage of all of them:
```scala

val race: IO[String, Int] = for {
    p     <- Promise.make[String, Int]
    _     <- p.succeed(1).fork
    _     <- p.complete(ZIO.succeed(2)).fork
    _     <- p.completeWith(ZIO.succeed(3)).fork
    _     <- p.done(Exit.succeed(4)).fork
    _     <- p.fail("5")
    _     <- p.failCause(Cause.die(new Error("6")))
    _     <- p.die(new Error("7"))
    _     <- p.interrupt.fork
    value <- p.await
  } yield value
```

The act of completing a Promise results in an `UIO[Boolean]`, where the `Boolean` represents whether the Promise value has been set (`true`) or whether it was already set (`false`). This is demonstrated below:

```scala
val ioPromise1: UIO[Promise[Exception, String]] = Promise.make[Exception, String]
val ioBooleanSucceeded: UIO[Boolean] = ioPromise1.flatMap(promise => promise.succeed("I'm done"))
```

Another example with `fail(...)`:

```scala
val ioPromise2: UIO[Promise[Exception, Nothing]] = Promise.make[Exception, Nothing]
val ioBooleanFailed: UIO[Boolean] = ioPromise2.flatMap(promise => promise.fail(new Exception("boom")))
```

To re-iterate, the `Boolean` tells us whether or not the operation took place successfully (`true`) i.e. the Promise
was set with the value or the error.

### Awaiting
We can get a value from a Promise using `await`. The calling fiber will suspend until Promise is completed with a value or an error.

```scala
val ioPromise3: UIO[Promise[Exception, String]] = Promise.make[Exception, String]
val ioGet: IO[Exception, String] = ioPromise3.flatMap(promise => promise.await)
```

### Polling
If we don't want to suspend the fiber, but we only want to query the state of whether or not the Promise has been completed, we can use `poll`:

```scala
val ioPromise4: UIO[Promise[Exception, String]] = Promise.make[Exception, String]
val ioIsItDone: UIO[Option[IO[Exception, String]]] = ioPromise4.flatMap(p => p.poll)
val ioIsItDone2: IO[Option[Nothing], IO[Exception, String]] = ioPromise4.flatMap(p => p.poll.some)
```

If the Promise was not completed when we called `poll` then the IO will fail with the `Unit` value. Otherwise, we obtain an `IO[E, A]`, where `E` represents if the Promise completed with an error and `A` indicates that the Promise successfully completed with an `A` value.

`isDone` returns `UIO[Boolean]` that evaluates to `true` if Promise is already completed.

## Example Usage
Here is a scenario where we use a `Promise` to hand over a value between two `Fiber`s:

```scala

val program: ZIO[Any, IOException, Unit] = 
  for {
    promise         <-  Promise.make[Nothing, String]
    sendHelloWorld  =   (ZIO.succeed("hello world") <* ZIO.sleep(1.second)).flatMap(promise.succeed)
    getAndPrint     =   promise.await.flatMap(Console.printLine(_))
    fiberA          <-  sendHelloWorld.fork
    fiberB          <-  getAndPrint.fork
    _               <-  (fiberA zip fiberB).join
  } yield ()
```

In the example above, we create a Promise and a Fiber (`fiberA`) that completes the Promise after 1 second, and a second Fiber (`fiberB`) that will call `await` on that Promise to obtain a `String` and then print it to screen. The example prints `hello world` to the screen after 1 second. Remember, this is just a description of the program and not the execution
itself.

---

## Queue

`Queue` is a lightweight in-memory queue built on ZIO with composable and transparent back-pressure. It is fully asynchronous (no locks or blocking), purely-functional and type-safe.

A `Queue[A]` contains values of type `A` and has two basic operations: `offer`, which places an `A` in the `Queue`, and `take` which removes and returns the oldest value in the `Queue`.

```scala

val res: UIO[Int] = for {
  queue <- Queue.bounded[Int](100)
  _ <- queue.offer(1)
  v1 <- queue.take
} yield v1
```

## Creating a queue

A `Queue` can be bounded (with a limited capacity) or unbounded.

There are several strategies to process new values when the queue is full:

- The default `bounded` queue is back-pressured: when full, any offering fiber will be suspended until the queue is able to add the item
- A `dropping` queue will drop new items when the queue is full
- A `sliding` queue will drop old items when the queue is full

To create a back-pressured bounded queue:
```scala
val boundedQueue: UIO[Queue[Int]] = Queue.bounded[Int](100)
```

To create a dropping queue:
```scala
val droppingQueue: UIO[Queue[Int]] = Queue.dropping[Int](100)
```

To create a sliding queue:
```scala
val slidingQueue: UIO[Queue[Int]] = Queue.sliding[Int](100)
```

To create an unbounded queue:
```scala
val unboundedQueue: UIO[Queue[Int]] = Queue.unbounded[Int]
```

## Adding items to a queue

The simplest way to add a value to the queue is `offer`:

```scala
val res1: UIO[Unit] = for {
  queue <- Queue.bounded[Int](100)
  _ <- queue.offer(1)
} yield ()
```

When using a back-pressured queue, offer might suspend if the queue is full: you can use `fork` to wait in a different fiber.

```scala
val res2: UIO[Unit] = for {
  queue <- Queue.bounded[Int](1)
  _ <- queue.offer(1)
  f <- queue.offer(1).fork // will be suspended because the queue is full
  _ <- queue.take
  _ <- f.join
} yield ()
```

It is also possible to add multiple values at once with `offerAll`:

```scala
val res3: UIO[Unit] = for {
  queue <- Queue.bounded[Int](100)
  items = Range.inclusive(1, 10).toList
  _ <- queue.offerAll(items)
} yield ()
```

## Consuming Items from a Queue

The `take` operation removes the oldest item from the queue and returns it. If the queue is empty, this will suspend, and resume only when an item has been added to the queue. As with `offer`, you can use `fork` to wait for the value in a different fiber.

```scala
val oldestItem: UIO[String] = for {
  queue <- Queue.bounded[String](100)
  f <- queue.take.fork // will be suspended because the queue is empty
  _ <- queue.offer("something")
  v <- f.join
} yield v
```

You can consume the first item with `poll`. If the queue is empty you will get `None`, otherwise the top item will be returned wrapped in `Some`.

```scala
val polled: UIO[Option[Int]] = for {
  queue <- Queue.bounded[Int](100)
  _ <- queue.offer(10)
  _ <- queue.offer(20)
  head <- queue.poll
} yield head
```

You can consume multiple items at once with `takeUpTo`. If the queue doesn't have enough items to return, it will return all the items without waiting for more offers.

```scala
val taken: UIO[Chunk[Int]] = for {
  queue <- Queue.bounded[Int](100)
  _ <- queue.offer(10)
  _ <- queue.offer(20)
  chunk  <- queue.takeUpTo(5)
} yield chunk
```

Similarly, you can get all items at once with `takeAll`. It also returns without waiting (an empty collection if the queue is empty).

```scala
val all: UIO[Chunk[Int]] = for {
  queue <- Queue.bounded[Int](100)
  _ <- queue.offer(10)
  _ <- queue.offer(20)
  chunk  <- queue.takeAll
} yield chunk
```

## Shutting Down a Queue

It is possible with `shutdown` to interrupt all the fibers that are suspended on `offer*` or `take*`. It will also empty the queue and make all future calls to `offer*` and `take*` terminate immediately.

```scala
val takeFromShutdownQueue: UIO[Unit] = for {
  queue <- Queue.bounded[Int](3)
  f <- queue.take.fork
  _ <- queue.shutdown // will interrupt f
  _ <- f.join // Will terminate
} yield ()
```

You can use `awaitShutdown` to execute an effect when the queue is shut down. This will wait until the queue is shut down. If the queue is already shut down, it will resume right away.

```scala
val awaitShutdown: UIO[Unit] = for {
  queue <- Queue.bounded[Int](3)
  p <- Promise.make[Nothing, Boolean]
  f <- queue.awaitShutdown.fork
  _ <- queue.shutdown
  _ <- f.join
} yield ()
```

## Additional Resources

- [ZIO Queue Talk by John De Goes @ ScalaWave 2018](https://www.slideshare.net/jdegoes/zio-queue)
- [ZIO Queue Talk by Wiem Zine El Abidine @ PSUG 2018](https://www.slideshare.net/wiemzin/psug-zio-queue)
- [Elevator Control System using ZIO](https://medium.com/@wiemzin/elevator-control-system-using-zio-c718ae423c58)
- [Scalaz 8 IO vs Akka (typed) actors vs Monix](https://blog.softwaremill.com/scalaz-8-io-vs-akka-typed-actors-vs-monix-part-1-5672657169e1)

---

## Ref

`Ref[A]` models a **mutable reference** to a value of type `A` in which we can store **immutable** data. The two basic operations are `set`, which fills the `Ref` with a new value, and `get`, which retrieves its current content.

`Ref` provides us a way to functionally manage in-memory state. All operations on `Ref` are atomic and thread-safe, giving us a reliable foundation for synchronizing concurrent programs.

`Ref`:
- is purely functional and referentially transparent
- is concurrent-safe and lock-free
- updates and modifies atomically

## Concurrent Stateful Application
**`Ref` is the foundation for writing concurrent stateful applications**. Anytime we need to share information between multiple fibers, and those fibers have to update the same information, they need to communicate through something that provides the guarantee of atomicity. Because `Ref` is **concurrent-safe**, we can share the same `Ref` among many fibers. All of which can update `Ref` concurrently, removing the worry of race conditions. Even if we had ten thousand fibers all updating the same `Ref`, as long as they are using atomic update and modify functions, we will have zero race conditions.

## Operations
Though `Ref` has many operations, here we will introduce the most common and important ones.

### make
`Ref` is never empty, it always contains something. We can create a `Ref` by providing the initial value to its `make` method, a constructor of the `Ref` data type. We should pass an **immutable value** of type `A` to the constructor, and it returns an `UIO[Ref[A]]` value:

```scala
def make[A](a: A): UIO[Ref[A]]
```

As we can see, the output is wrapped in`UIO`, which means creating a `Ref` is effectful. Whenever we `make`, `update`, or `modify` the `Ref`, we are performing an effectful operation.

Let's create some `Ref`s from immutable values:

```scala
val counterRef = Ref.make(0)
// counterRef: UIO[Ref[Int]] = Sync(
//   trace = "repl.MdocSession.MdocApp.counterRef(ref.md:14)",
//   eval = zio.Ref$$$Lambda$19942/0x00007f02a6fa59c8@1e2f5fbf
// )
val stringRef = Ref.make("initial") 
// stringRef: UIO[Ref[String]] = Sync(
//   trace = "repl.MdocSession.MdocApp.stringRef(ref.md:17)",
//   eval = zio.Ref$$$Lambda$19942/0x00007f02a6fa59c8@409b8d58
// )

sealed trait State
case object Active  extends State
case object Changed extends State
case object Closed  extends State

val stateRef = Ref.make(Active) 
// stateRef: UIO[Ref[Active.type]] = Sync(
//   trace = "repl.MdocSession.MdocApp.stateRef(ref.md:32)",
//   eval = zio.Ref$$$Lambda$19942/0x00007f02a6fa59c8@3808a7da
// )
```

> _**Warning**_:  
>
> A big mistake when creating a `Ref` is trying to store mutable data inside it. A`Ref` must be used with **immutable data**. Otherwise, we lose our atomic guarantees, which can lead to collisions and race conditions. 

The following snippet compiles, but it leads to race conditions due to a mutable variable being provided to `make`:

```scala
// Compiles but don't work properly
val init = collection.mutable.Seq(1,3,5)
// init: collection.mutable.Seq[Int] = ArrayBuffer(1, 3, 5)
val counterRef = Ref.make(init)
// counterRef: UIO[Ref[collection.mutable.Seq[Int]]] = Sync(
//   trace = "repl.MdocSession.MdocApp.<local MdocApp>.counterRef(ref.md:42)",
//   eval = zio.Ref$$$Lambda$19942/0x00007f02a6fa59c8@1e6d987
// )
```

To correct this, we should change the `init` to be immutable:

```scala
val init = Seq(1,3,5)
// init: Seq[Int] = List(1, 3, 5)
val counterRef = Ref.make(init)
// counterRef: UIO[Ref[Seq[Int]]] = Sync(
//   trace = "repl.MdocSession.MdocApp.<local MdocApp>.counterRef(ref.md:52)",
//   eval = zio.Ref$$$Lambda$19942/0x00007f02a6fa59c8@5b3f7b64
// )
```

### get
The `get` method returns the current value of the reference. Its return type is `IO[EB, B]` in which `B` is the value type of the effect and in the failure case, `EB` is the error type of that effect.

```scala
def get: IO[EB, B]
```

As the `make` and `get` methods of `Ref` are effectful, we can chain them together with `flatMap`. In the following example, we create a `Ref` with `initial` value, and then we acquire the current state with the `get` method:

```scala
Ref.make("initial")
   .flatMap(_.get)
   .flatMap(current => Console.printLine(s"current value of ref: $current"))
```

We can refactor this to use a for-comprehension rather than a series of `flatMap`s to increase readability:

```scala
for {
  ref   <- Ref.make("initial")
  value <- ref.get
} yield assert(value == "initial")
```

Note that, there is no way to access the shared state outside the monadic operations.

### set
The `set` method atomically writes a new value to the `Ref`.

```scala
for {
  ref   <- Ref.make("initial")
  _     <- ref.set("update")
  value <- ref.get
} yield assert(value == "update")
```

### update
With `update`, we can atomically update the state of `Ref` with a given **pure** function, that is, it needs to be deterministic and free of side effects.

```scala
def update(f: A => A): IO[E, Unit]
```

Assume we have a counter, we can increase its value with the `update` method:

```scala
val counterInitial = 0
for {
  counterRef <- Ref.make(counterInitial)
  _          <- counterRef.update(_ + 1)
  value <- counterRef.get
} yield assert(value == 1)
```

:::caution

`update` is not the composition of `get` and `set`. This composition is not concurrent-safe. Whenever we need to update our state, we should use the `update` operation which modifies its `Ref` atomically. 
:::

For example, the following snippet is not concurrent-safe:

```scala
// Unsafe State Management
object UnsafeCountRequests extends ZIOAppDefault {

  def request(counter: Ref[Int]) = for {
    current <- counter.get
    _ <- counter.set(current + 1)
  } yield ()

  private val initial = 0
  private val myApp =
    for {
      ref <- Ref.make(initial)
      _ <- request(ref) zipPar request(ref)
      rn <- ref.get
      _ <- Console.printLine(s"total requests performed: $rn")
    } yield ()

  def run = myApp
}
```

The above snippet doesn't behave deterministically. This program sometimes prints `2` and sometimes prints `1`. We can fix it by using `update`:

```scala
// Safe State Management
object CountRequests extends ZIOAppDefault {

  def request(counter: Ref[Int]): ZIO[Any, Nothing, Unit] = {
    for {
      _ <- counter.update(_ + 1)
      reqNumber <- counter.get
      _ <- Console.printLine(s"request number: $reqNumber").orDie
    } yield ()
  }

  private val initial = 0
  private val myApp =
    for {
      ref <- Ref.make(initial)
      _ <- request(ref) zipPar request(ref)
      rn <- ref.get
      _ <- Console.printLine(s"total requests performed: $rn").orDie
    } yield ()

  def run = myApp
}
```

Here is another use case of `update` to write a `repeat` combinator:

```scala
def repeat[E, A](n: Int)(io: IO[E, A]): IO[E, Unit] =
  Ref.make(0).flatMap { iRef =>
    def loop: IO[E, Unit] = iRef.get.flatMap { i =>
      if (i < n)
        io *> iRef.update(_ + 1) *> loop
      else
        ZIO.unit
    }
    loop
  }
```

### modify
`modify` is a more powerful version of `update`. It atomically modifies `Ref` by the given function, and also computes a return value. The function that we pass to `modify` needs to be a pure function; it needs to be deterministic and free of side effects.

```scala
def modify[B](f: A => (B, A)): IO[E, B]
```

Remember the `CountRequest` example. What if we want to log the number of each request inside the `request` function? Let's see what happens if we write that function with the composition of `update` and `get` methods:

```scala
// Unsafe in Concurrent Environment
def request(counter: Ref[Int]) = {
  for {
    _  <- counter.update(_ + 1)
    rn <- counter.get
    _  <- Console.printLine(s"request number received: $rn")
  } yield ()
}
```
What happens if, between running `update` and `get`, a second `update` occurs on another fiber? This would not behave deterministically in concurrent environments. So we need a way to perform a combination of **get, set, get** atomically. This is where `modify` comes in. Here we will edit `request` to use `modify`:

```scala
// Safe in Concurrent Environment
def request(counter: Ref[Int]) = {
  for {
    rn <- counter.modify(c => (c + 1, c + 1))
    _  <- Console.printLine(s"request number received: $rn")
  } yield ()
}
```

## AtomicReference in Java 
For Java programmers, we can think of `Ref` as an `AtomicReference`. Java has a `java.util.concurrent.atomic` package which contains `AtomicReference`, `AtomicLong`, `AtomicBoolean` and so forth. `Ref` has roughly the same power, guarantees, and limitations as `AtomicReference`, but is higher-level and ZIO-friendly. 
 
## Ref vs. State Monad
Basically `Ref` allows us to have all the power of State Monad inside ZIO. State Monad lacks two important features that we use in real-life application development:

1. Concurrency Support
2. Error Handling

### Concurrency
State Monad is an effect system that only includes state. It allows us to do pure stateful computations. We can only get, set, and update (and related computations) state. State Monad updates its state with series of stateful computations sequentially, but **it can't be used to do async or concurrent computations**. `Ref`, in contrast, has great support for concurrent and async programming.

### Error Handling
In most real-life,stateful applications, we will involve some database IO and API calls and/or some concurrent and sync operations which can fail in different ways along the path of execution. So besides state management, we need a way to handle errors. The State Monad doesn't have the ability to model error management. We can combine State Monad and Either Monad with StateT monad transformer, but it imposes massive performance overhead. It doesn't buy us anything that we can't do with `Ref`. So it is an anti-pattern. In the ZIO model, errors are encoded in effects and `Ref` utilizes that. So, in addition to state management, we have the ability to handle errors without additional work.

## State Transformers

Those who live on the dark side of mutation sometimes have it easy; they can add state everywhere like it's Christmas. Behold:

```scala
var idCounter = 0
def freshVar: String = {
  idCounter += 1
  s"var${idCounter}"
}
val v1 = freshVar
val v2 = freshVar
val v3 = freshVar
```

As functional programmers, we know better and have captured state mutation in the form of functions of type `S => (A, S)`. `Ref` provides such an encoding, with `S` being the type of the value, and `modify` embodying the state mutation function.

```scala
Ref.make(0).flatMap { idCounter =>
  def freshVar: UIO[String] =
    idCounter.modify(cpt => (s"var${cpt + 1}", cpt + 1))

  for {
    v1 <- freshVar
    v2 <- freshVar
    v3 <- freshVar
  } yield ()
}
```

## Building more sophisticated concurrency primitives

`Ref` is low-level enough that it can serve as the foundation for other concurrency data types.

For example, semaphores are a classic abstract data type for controlling access to shared resources. They are defined as a triplet `S = (v, P, V)` where `v` is the number of units of the resource that are currently available, and `P` and `V` are operations that decrement and increment `v`, respectively. `P` will only complete when `v` is non-negative and must wait if it isn't.

With `Ref`, it's easy to implement such a semaphore! The only difficulty is in `P`, where we must fail and retry when either `v` is negative, or its value has changed between the moment we read it and the moment we try to update it. A naive implementation could look like:

```scala
sealed trait S {
  def P: UIO[Unit]
  def V: UIO[Unit]
}

object S {
  def apply(v: Long): UIO[S] =
    Ref.make(v).map { vref =>
      new S {
        def V = vref.update(_ + 1).unit

        def P = (vref.get.flatMap { v =>
          if (v < 0)
            ZIO.fail(())
          else
            vref.modify(v0 => if (v0 == v) (true, v - 1) else (false, v)).flatMap {
              case false => ZIO.fail(())
              case true  => ZIO.unit
            }
        } <> P).unit
      }
    }
}
```

Let's rock these crocodile boots we found the other day at the market and test our semaphore at the night club, yee-haw:

```scala

val party = for {
  dancefloor <- S(10)
  dancers <- ZIO.foreachPar(1 to 100) { i =>
    dancefloor.P *> Random.nextDouble.map(d => Duration.fromNanos((d * 1000000).round)).flatMap { d =>
      printLine(s"${i} checking my boots") *> ZIO.sleep(d) *> printLine(s"${i} dancing like it's 99")
    } *> dancefloor.V
  }
} yield ()
```

It goes without saying you should take a look at ZIO's own `Semaphore`, it does all this and more without wasting all those CPU cycles while waiting.

## Atomic Modify with Continuation

As discussed above, `Ref#modify` takes a **pure function** to atomically modify the state and compute a return value. While we shouldn't run effects inside `Ref#modify`, we can introduce a continuation—an effect that the runtime executes *after* the atomic modification:

```scala
ref.modify { state =>
  val doThisNext = someZIO
  val newState = computeNewState(state)
  (doThisNext, newState)  // "doThisNext" is the continuation
}.flatten  // flatten to execute the continuation
```

The type of `ref.modify { state => ... }` is an effect that produces another effect. Running this effect performs the atomic modification and returns the continuation effect. To execute the continuation, we must `flatten` the result.

**Important:** The continuation is **not** part of the atomic modification. The state modification completes before the continuation runs, and it doesn't depend on the continuation's result. If multiple fibers execute this pattern concurrently, their continuations may be interleaved, even though their state modifications are atomic.

If you need the continuation to be part of the atomic operation—ensuring no other fiber can modify the state between the modification and the continuation—use `Ref.Synchronized` instead. See [Ref.Synchronized](refsynchronized.md) for more details.

---

## Ref.Synchronized

`Ref.Synchronized[A]` models a **mutable reference** to a value of type `A` in which we can store **immutable** data, and update it atomically **and** effectfully.

:::note

Almost all of `Ref.Synchronized` operations are the same as `Ref`. We suggest reading [`Ref`](ref.md) at first if you are not familiar with `Ref`.
:::

Let's explain how we can update a shared state effectfully with `Ref.Synchronized`. The `update` method and all other related methods get an effectful operation, and then they run these effects to change the shared state. This is the main difference between `Ref.Synchronized` and `Ref`. 

In the following example, we should pass in `updateEffect` to it which is the description of an update operation. So `Ref.Synchronized` is going to update the `ref` by running the `updateEffect`:

```scala

for {
  ref <- Ref.Synchronized.make("current")
  updateEffect = ZIO.succeed("update")
  _ <- ref.updateZIO(_ => updateEffect)
  value <- ref.get
} yield assert(value == "update")
```

In real-world applications, there are cases where we want to run an effect, e.g. query a database, and then update the shared state. This is where `Ref.Synchronized` can help us to update the shared state in a more actor model fashion. We have a shared mutable state but for every different command or message, and we want execute our effect and update the state. 

We can pass in an effectful program into every single update. All of them will be done parallel, but the result will be sequenced in such a fashion that they only touched the state at different times, and we end up with a consistent state at the end.

In the following example, we are going to send `getAge` request to usersApi for each user and updating the state respectively:

```scala
val meanAge =
  for {
    ref <- Ref.Synchronized.make(0)
    _ <- ZIO.foreachPar(users) { user =>
      ref.updateZIO(sumOfAges =>
        api.getAge(user).map(_ + sumOfAges)
      )
    }
    v <- ref.get
  } yield (v / users.length)
```

---

## Semaphore

A `Semaphore` datatype which allows synchronization between fibers with the `withPermit` operation, which safely acquires and releases a permit.
`Semaphore` is based on `Ref[A]` datatype.

## Operations

For example, a synchronization of asynchronous tasks can 
be done via acquiring and releasing a semaphore with a given number of permits it can spend.
When the acquire operation cannot be performed due to no more available `permits` in the semaphore, such task 
is semantically blocked, until the `permits` value is large enough again:

```scala

val task = for {
  _ <- printLine("start")
  _ <- ZIO.sleep(Duration(2, TimeUnit.SECONDS))
  _ <- printLine("end")
} yield ()

val semTask = (sem: Semaphore) => for {
  _ <- sem.withPermit(task)
} yield ()

val semTaskSeq = (sem: Semaphore) => (1 to 3).map(_ => semTask(sem))

val program = for {

  sem <- Semaphore.make(permits = 1)

  seq <- ZIO.succeed(semTaskSeq(sem))

  _ <- ZIO.collectAllPar(seq)

} yield ()
```

As the binary semaphore is a special case of a counting semaphore, 
we can acquire and release any number of `permits`:

```scala
val semTaskN = (sem: Semaphore) => for {
  _ <- sem.withPermits(5)(task)
} yield ()
```

The guarantee of `withPermit` (and its corresponding counting version `withPermits`) is that each acquisition will be followed by the equivalent number of releases, regardless of whether the task succeeds, fails, or is interrupted.

---

## Introduction to Configuration in ZIO

Configuration is a core concern for any cloud-native application. So ZIO ships with built-in support for configuration by providing a front-end for configuration providers as well as metrics and logging.

So, ZIO provides a unified way to configure our applications, while still enabling customizability, flexibility, and significant integrations with configuration backends via ecosystem projects, most notably [ZIO Config](https://zio.dev/zio-config).

This configuration front-end allows ecosystem libraries and applications to declaratively describe their configuration needs and delegates the heavy lifting to a ConfigProvider, which may be supplied by third-party libraries such as ZIO Config.

The ZIO Core ships with a simple default config provider, which reads configuration data from environment variables and if not found, from system properties. This can be used for development purposes or to bootstrap applications toward more sophisticated config providers.

## Getting Started

To make our application configurable, we should know about three essential elements:

1. **Config Description**— To describe configuration data of type `A`, we should create an instance of `Config[A]`. If the configuration data is simple (such as `string`, `string`, `boolean`), we can use built-in configs inside companion object of `Config` data type. By combining primitive configs, we can model custom data types such as `HostPort`.

2. **Config Front-end**— By using `ZIO.config` we can load configuration data described by `Config`. It takes a `Config[A]` instance or expect implicit `Config[A]` and loads the config using the current `ConfigProvider`.

3. **Config Backend**— `ConfigProvider` is the underlying engine that `ZIO.config` uses to load configs. ZIO has a default config provider inside its default services. The default config provider reads configuration data from environment variables and if not found, from system properties. To change the default config provider, we can use `Runtime.setConfigProvider` layer to configure the ZIO runtime to use a custom config provider.

:::note
By introducing built-in config front-end in ZIO Core, the old way of reading configuration data using `ZLayer` is deprecated, and we don't recommend using layers for configuration anymore.
:::

## Primitive Configs

ZIO provides a set of primitive configs for the most common types like `int`, `long`, `string`, `boolean`, `double`, etc. All of these configs are available inside the `Config` object.

Let's start with a simple example of how to read configuration from environment variables and system properties:

```scala

object MainApp extends ZIOAppDefault {
  def run = {
    for {
      host <- ZIO.config(Config.string("host"))
      port <- ZIO.config(Config.int("port"))
      _    <- Console.printLine(s"Application started: $host:$port")
    } yield ()
  }
}
```

:::note
Use `ZIO.config(config: A)` overload for primitive data types instead of `ZIO.config[A]` to avoid potential implicit conflicts.
:::

If we run this application we will get the following output:

```bash
timestamp=2023-02-14T09:45:27.074151Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-4" zio.Config$Error$Or: ((Missing data at host: Expected HOST to be set in the environment) or (Missing data at host: Expected host to be set in properties))
```

This is because we have not provided any configuration. Let's try to run it with the following environment variables:

```bash
HOST=localhost PORT=8080 sbt "runMain MainApp"
```

Now we get the following output:

```bash
Application started: localhost:8080
```

We can also run it by setting system properties:

```bash
sbt -Dhost=localhost -Dport=8080 "runMain MainApp"
```

## Custom Configs

Other than primitive types, we can also define a configuration for custom types. To do so, we need to use primitive configs and combine them together using `Config` operators (`++`, `||`, `map`, etc) and constructors (`listOf`, `chunkOf`, `setOf`, `vectorOf`, `table`, etc).

### Example 1

Let's say we have the `HostPort` data type, which consists of two fields: `host` and `port`:

```scala
case class HostPort(host: String, port: Int)
```

We can define implicit config for this data type by combining primitive `string` and `int` configs:

```scala

object HostPort {
  implicit val config: Config[HostPort] =
    (Config.string("host") ++ Config.int("port")).map { case (host, port) =>
      HostPort(host, port)
    }
}
```

:::note
The best practice is to put the implicit `Config` value in the companion object of the configuration data type and call it `config`.
:::

If we use this customized config in our application, it tries to read corresponding values from environment variables (`HOST` and `PORT`) and system properties (`host` and `port`):

```scala
for {
  config <- ZIO.config[HostPort]
  _      <- Console.printLine(s"Application started: $config")
} yield ()
```

### Example 2

Now let's assume we want to have multiple `HostPort` configurations. We can define a config for a list of `HostPort` like bellow using the `listOf` constructor:

```scala
case class HostPorts(hostPorts: List[HostPort])

object HostPorts {
  implicit val config: Config[HostPorts] =
    Config.listOf(HostPort.config).map(HostPorts(_))
}
```

Then we can use this config in our application:

```scala
for {
  config <- ZIO.config[HostPorts]
  _      <- Console.printLine(s"Application started with:")
  _      <- ZIO.foreachDiscard(config.hostPorts)(e => Console.printLine(s"  - http://${e.host}:${e.port}"))
} yield ()
```

With the default config provider, we can run the application with the following environment variables:

```bash
HOST=host1,host2,host3 PORT=8080,8081,8082 sbt "runMain MainApp"
```

The output will be:

```bash
Application started with:
  - http://host1:8081
  - http://host2:8082
  - http://host3:8083
```

## Top-level and Nested Configs

So far we have seen how to define configuration in a top-level manner, whether it is a primitive or a custom type. But we can also define a nested configuration.

Assume we have a `SerivceConfig` data type that consists of two fields: `hostPort` and `timeout`:

```scala
case class ServiceConfig(hostPort: HostPort, timeout: Int)
```

Let's define a config for this type in its companion object:

```scala

object ServiceConfig {
  implicit val config: Config[ServiceConfig] =
    (HostPort.config ++ Config.int("timeout")).map {
      case (a, b) => ServiceConfig(a, b)
    }
}
```

If we use this customized config in our application, it tries to read corresponding values from environment variables (`HOST`, `PORT`, and `TIMEOUT`) and, if not found from system properties (`host`, `port`, and `timeout`).

But in most circumstances, we don't want to read all the configurations from the top-level namespace. Instead, we want to nest them under a common namespace. In this case, we want to read both `HOST` and `PORT` from the `HOSTPORT` namespace, and `TIMEOUT` from the root namespace. In order to do that, we can use the `nested` combinator:

```diff
object ServiceConfig {
  implicit val config: Config[ServiceConfig] =
-    (HostPort.config ++ Config.int("timeout")).map {
+    (HostPort.config.nested("hostport") ++ Config.int("timeout")).map {
      case (a, b) => ServiceConfig(a, b)
    }
}
```

Now, if we run our application, it tries to read corresponding values from environment variables (`HOSTPORT_HOST`, `HOSTPORT_PORT` and `TIMEOUT`) and, if not found it tries to read from system properties (`hostport.host`, `hostport.port` and `timeout`).

## Built-in Config Providers

ZIO has some built-in config providers:

- `ConfigProvider.defaultProvider` - reads configuration from environment variables and if not found, from system properties
- `ConfigProvider.envProvider` - reads configuration from environment variables
- `ConfigProvider.propsProvider` - reads configuration from system properties
- `ConfigProvider.consoleProvider` - reads configuration from interactive console prompts

Other than these built-in providers, we can also use third-party providers in ZIO ecosystem libraries, such as ZIO Config which provides a rich set of backends for reading configuration from different sources such as HOCON, JSON, YAML, etc.

## Custom Config Provider

We can also define our own custom config providers.

The default config provider is used by default, but we can also override it by using `Runtime#setConfigProvider`.

In the following example, we set the default config provider to `consoleProvider` which reads configuration from the console:

```scala

object MainAppScoped extends ZIOAppDefault {
  override val bootstrap: ZLayer[Any, Nothing, Unit] =
    Runtime.setConfigProvider(ConfigProvider.consoleProvider)

  def run =
    for {
      host <- ZIO.config(Config.string("host"))
      port <- ZIO.config(Config.int("port"))
      _ <- Console.printLine(s"Application started: http://$host:$port")
    } yield ()
}
```

:::note
The console provider is stored inside a `FiberRef`, so we can override it in a scoped manner. This is useful for changing the config provider for a specific part of the application.
:::

## Testing Services

When testing services, we sometimes need to provide some configuration to them. So we should be able to mock any backend that we use for reading configuration data.

In order to do that, we can use the `ConfigProvider.fromMap` constructor, which takes a map of configuration data and returns a config provider that reads configuration from that map. Then we can pass that to the `Runtime.setConfigProvider`, which returns a `ZLayer` that we can use to override the default config provider for our test specs using `Spec#provideLayer` operator:

```scala

object MyServiceTest extends ZIOSpecDefault {

  val mockConfigProvider: ZLayer[Any, Nothing, Unit] =
    Runtime.setConfigProvider(ConfigProvider.fromMap(Map("timeout" -> "5s")))

  // This service reads configuration data (host and port) inside its implementation
  def myService: ZIO[Any, Config.Error, Double] = ???

  override def spec = {
    val expected: Double = ??? // expected value
    test("test myService") {
      for {
        result <- myService
      } yield assertTrue(result == expected)
    }
  }.provideLayer(mockConfigProvider)
  
}
```

---

## Automatic ZLayer Derivation

ZIO's `ZLayer` is a powerful tool for building modular, testable, and composable applications. With the `ZLayer.derive`
utility, you can automatically derive simple `ZLayer` instances for your services, reducing boilerplate and simplifying
your codebase.

## Basic Use Cases

```scala

class Database(connection: String)
object Database {
  val layer: ZLayer[String, Nothing, Database] = ZLayer.derive[Database]
}

class UserService(db: Database)
object UserService {
  val layer: ZLayer[Database, Nothing, UserService] = ZLayer.derive[UserService]
}
```

## Default Values

For services that might have default values or configurations, `ZLayer.derive` can use implicit
 `ZLayer.Derive.Default[A]` values:

### Pre-defined Default Values

There are some pre-defined `ZLayer.Derive.Default[A]` instances for the following types:

#### `Config[A]`

When a service `A` has a constructor parameter `B` and there's an implicit `Config[B]` instance, `ZLayer.derive`
automatically loads `B` using `ZIO.config`.

```scala

case class APIClientConfig(appKey: String, secretKey: Config.Secret)
object APIClientConfig {
  // Because we have an implicit `Config[APIClientConfig]` in scope...
  implicit val config: Config[APIClientConfig] =
    (Config.string("appKey") ++ Config.secret("secretKey")).map {
      case (uri, key) => APIClientConfig(uri, key)
    }
}

class APIClient(config: APIClientConfig) { /* ... */ }
object APIClient {

  // `APIClientConfig` is automatically loaded using `ZIO.config` by `ZLayer.derive`,
  // instead of being required as a layer input.
  val layer: ZLayer[Any, Config.Error, APIClient] = ZLayer.derive[APIClient]
}
```

Refer to [Configuration](../configuration/index.md) for more about `Config`.

#### Some Concurrency Primitives

- `Promise[E, A]`
- `Queue[A]` (using `Queue.unbounded`)
- `Hub[A]` (using `Hub.unbounded`)
- `Ref[A]` (when `A` has a default instance)

### Creating New Default Value

There are three main ways to create a `ZLayer.Derive.Default`:

1. `ZLayer.Derive.Default.succeed` for creating default values from simple values.
2. `ZLayer.Derive.Default.fromZIO` for creating default values from effects.
3. `ZLayer.Derive.Default.fromLayer` for creating default values from layers.

### Overriding Predefined Default Values

At times, you may want to override a default value in specific scenarios. To achieve this, you can define your own
implicit value in a scope with a higher implicit priority, like a closer lexical scope.

A common scenario for this is when you want to discard a pre-defined default value and instead treat it as a dependency.
Use `ZLayer.Derive.Default.service` for this purpose:

```scala

class Wheels(number: Int)
object Wheels {
  implicit val defaultWheels: Default.WithContext[Any, Nothing, Wheels] =
    Default.succeed(new Wheels(4))
}
class Car(wheels: Wheels)

val carLayer1: ZLayer[Any, Nothing, Car] = ZLayer.derive[Car] // wheels.number == 4
val carLayer2: ZLayer[Wheels, Nothing, Car] = locally {
  // The default instance is discarded
  implicit val newWheels: Default.WithContext[Wheels, Nothing, Wheels] =
     Default.service[Wheels]

  ZLayer.derive[Car]
}
```

### Caveat: Use `Default.WithContext[R, E, A]` instead of `Default[A]` for type annotation

When providing type annotations for `ZLayer.derive`, you must use `ZLayer.Derive.Default.WithContext[R, E, A]` instead
of the more general `ZLayer.Derive.Default[A]`. Using the latter will result in a compilation error due to missing type
details.

If you're uncertain about the exact type signature, a practical approach is to omit the type annotation initially. Then,
use your IDE's autocomplete feature to insert the inferred type.

## Attaching Scoped Resources

For services requiring resource management, `ZLayer.derive` offers built-in support for scoped values. When a service
`A` implements the `ZLayer.Derive.Scoped[-R, +E]` trait, `ZLayer.derive[A]` automatically recognizes it. As a result,
the `scoped` effect is executed during the layer's construction and finalization phases.

The 'resource' might be a background task, a lock file, or etc., that can be managed by [`Scope`](../resource/scope.md).

```scala

trait Connection {
  def healthCheck: ZIO[Any, Throwable, Unit]
  // ...
}

class ThirdPartyService(connection: Connection) extends ZLayer.Derive.Scoped[Any, Nothing] {

  // Repeats health check every 10 seconds in background during the layer's lifetime
  override def scoped(implicit trace: Trace): ZIO[Scope, Nothing, Any] =
    connection.healthCheck
      .ignoreLogged
      .repeat(Schedule.spaced(10.seconds))
      .forkScoped
}

object ThirdPartyService {
  // `ZLayer.Derive.Scoped` should be used with `ZLayer.derive`
  val layer: ZLayer[Connection, Nothing, ThirdPartyService] = ZLayer.derive[ThirdPartyService]
}
```

If `scoped` fails during resource acquisition, the entire `ZLayer` initialization process fails.

### Lifecycle Hooks

Additionally, there's the `ZLayer.Derive.AcquireRelease[R, E, A]` trait. This is a specialized version of
`ZLayer.Derive.Scoped` designed for added convenience, allowing users to define initialization and finalization hooks
distinctly.

```scala

def acquireLockFile(path: String): ZIO[Any, Throwable, File] = ???
def deleteFile(file: File): ZIO[Any, Throwable, Unit] = ???

class ASingletonService(lockFilePath: String) extends ZLayer.Derive.AcquireRelease[Any, Throwable, File] {

  override def acquire: ZIO[Any, Throwable, File] =
     acquireLockFile(lockFilePath)

  override def release(lockFile: File): ZIO[Any, Nothing, Any] =
     deleteFile(lockFile).ignore
}

object ASingletonService {
  // Note: it's for illustrative example. In a real-world application, you will probably want to
  //       put the `String` in a config.
  val layer: ZLayer[String, Throwable, ASingletonService] = ZLayer.derive[ASingletonService]
}
```

### Caveat: Manual layers do not respect `ZLayer.Derive.Scoped` and `ZLayer.Derive.AcquireRelease`

When manually creating `ZLayer` instances without using `ZLayer.derive`, the lifecycle hooks won't be automatically
invoked. Refer to [Resource Management in ZIO](../resource/index.md) for more details about general resource management
in ZIO.

---

## Introduction to the ZIO's Contextual Data Types

ZIO provides a contextual abstraction that encodes the environment of the running effect. This means, every effect can work within a specific context, called an environment.

So when we have a `ZIO[R, E, A]` effect, we can say "given `R` as the environment of the effect, the effect may fail with an error type of `E`, or may succeed with a value of type `A`".

For example, when we have an effect of type `ZIO[DatabaseConnection, IOException, String]`, we can say that our effect works within the context of `DatabaseConnection`. In other words, we can say that our effect requires the `DatabaseConnection` service as a context to run.

We will see how layers can be used to eliminate the environment of an effect:

```scala

trait DatabaseConnection

// An effect which requires DatabaseConnection to run
val effect: ZIO[DatabaseConnection, IOException, String] = ???

// A layer that produces DatabaseConnection service
val dbConnection: ZLayer[Any, IOException, DatabaseConnection] = ???

// After applying dbConnection to our environmental effect the reurned
// effect has no dependency on the DatabaseConnection
val eliminated: ZIO[Any, IOException, String] = 
  dbConnection { // Provides DatabaseConnection context
    effect       // An effect running within `DatabaseConnection` context
  }
```

ZIO provides this facility through the following concepts and data types:
1. [ZIO Environment](#1-zio-environment) — The `R` type parameter of `ZIO[R, E, A]` data type.
2. [ZEnvironment](#2-zenvironment) — Built-in type-level map for maintaining the environment of a `ZIO` data type. 
3. [ZLayer](#3-zlayer) — Describes how to build one or more services in our application.

Next, we will discuss _ZIO Environment_ and _ZLayer_ and finally how to write ZIO services using the _Service Pattern_.

## 1. ZIO Environment

The `ZIO[-R, +E, +A]` data type describes an effect that requires an input of type `R`, as an environment, may fail with an error of type `E`, or succeed with a value of type `A`.

The input type is also known as _environment type_. This type-parameter indicates that to run an effect we need one or some services as an environment of that effect. In other word, `R` represents the _requirement_ for the effect to run, meaning we need to fulfill the requirement in order to make the effect _runnable_.

So we can think of `ZIO[R, E, A]` as a mental model of a function from a value of type `R` to the `Either[E, A]`:

```scala
type ZIO[R, E, A] = R => Either[E, A]
```

`R` represents dependencies; whatever services, config, or wiring a part of a ZIO program depends upon to work. We will explore what we can do with `R`, as it plays a crucial role in `ZIO`.

### Motivation

One might ask "What is the motivation behind encoding the dependency in the type parameter of `ZIO` data type"? What is the benefit of doing so?

Let's see how writing an application which requires reading from or writing to the terminal. As part of making the application _modular_ and _testable_ we define a separate service called `Terminal` which is responsible for reading from and writing to the terminal. We do that simply by writing an interface:

```scala

trait Terminal {
  def print(line: Any): Task[Unit]

  def printLine(line: Any): Task[Unit]

  def readLine: Task[String]
}
```

Now we can write our application that accepts the `Terminal` interface as a parameter:

```scala

def myApp(c: Terminal): Task[Unit] =
  for {
    _    <- c.print("Please enter your name: ")
    name <- c.readLine
    _    <- c.printLine(s"Hello, $name!")
  } yield ()
```

Similar to the object-oriented paradigm we code to interface not implementation. In order to run the application, we need to implement a production version of the `Terminal`:

```scala

object TerminalLive extends Terminal {
  override def print(line: Any): Task[Unit] =
    ZIO.attemptBlocking(scala.Predef.print(line))

  override def printLine(line: Any): Task[Unit] =
    ZIO.attemptBlocking(scala.Predef.println(line))

  override def readLine: Task[String] =
    ZIO.attemptBlocking(scala.io.StdIn.readLine())
}
```

Finally, we can provide the `TerminalLive` to our application and run the whole:

```scala

object MainApp extends ZIOAppDefault {
  def myApp(c: Terminal): Task[Unit] =
    for {
      _    <- c.print("Please enter your name: ")
      name <- c.readLine
      _    <- c.printLine(s"Hello, $name!")
    } yield ()

  def run = myApp(TerminalLive)
}
```

In the above example, we discard the fact that we could use the ZIO environment and utilize the `R` parameter of the `ZIO` data type. So instead we tried to write the application with the `Task` data type, which ignores the ZIO environment. To create our application testable, we gathered all terminal functionalities into the same interface called `Terminal`, and implemented that in another object called `TerminalLive`. Finally, at the end of the day, we provide the implementation of the `Terminal` service, i.e. `TerminalLive`, to our application.

**While this technique works for small programs, it doesn't scale.** Assume we have multiple services, and we use them in our application logic like below:

```scala
def foo(
   s1: Service1,
   s2: Service2,
   s3: Service3
)(arg1: String, arg2: String, arg3: Int): Task[Int] = ???

def bar(
  s1: Service1,
  s12: Service12,
  s18: Service18, 
  sn: ServiceN
)(arg1: Int, arg2: String, arg3: Double, arg4: Int): Task[Unit]

def myApp(s1: Service1, s2: Service2, ..., sn: ServiceN): Task[Unit] = 
  for {
    a <- foo(s1, s2, s3)("arg1", "arg2", 4) 
    _ <- bar(s1, s12, s18, sn)(7, "arg2", 1.2, a)
      ...
  } yield ()
```

Writing real applications using this technique is tedious and cumbersome because all dependencies have to be passed across all methods. We can simplify the process of writing our application by using the ZIO environment and [Service Pattern](../service-pattern/service-pattern.md).

```scala
def foo(arg1: String, arg2: String, arg3: Int): ZIO[Service1 & Service2 & Service3, Throwable, Int] = 
  for {
    s1 <- ZIO.service[Service1]
    s2 <- ZIO.service[Service2] 
      ...
  } yield ()

def bar(arg1: Int, arg2: String, arg3: Double, arg4: Int): ZIO[Service1 & Service12 & Service18 & ServiceN, Throwable, Unit] =
  for {
    s1  <- ZIO.service[Service1] 
    s12 <- ZIO.service[Service12]
      ...
  } yield ()
```

### Advantage of Using ZIO Environment

ZIO environment facility enables us to:

1. **Code to Interface** — Like object-oriented paradigm, in ZIO we are encouraged to code to interface and defer the implementation. It is the best practice, but ZIO does not enforce us to do that.

2. **Write a Testable Code** — By coding to an interface, whenever we want to test our effects, we can easily mock any external services, by providing a _test_ version of those instead of the _live_ version.

3. **Compose Services with Strong Type Inference Facility** — We can compose multiple effects that require various services, so the final effect requires the intersection of all those services:

```scala

trait ServiceA
trait ServiceB
trait ServiceC

// Requires ServiceA and produces a value of type Int
def foo: ZIO[ServiceA, Nothing, Int] = ???

// Requires ServiceB and ServiceC and produces a value of type String
def bar: ZIO[ServiceB & ServiceC, Throwable, String] = ???

// Requires ServicB and produces a value of type Double
def baz(a: Int, b: String): ZIO[ServiceB, Nothing, Double] = ???

// Requires ServiceA and ServiceB and ServiceC and produces a value of type Double
val myApp: ZIO[ServiceA & ServiceB & ServiceC, Throwable, Double] =
  for {
    a <- foo
    b <- bar
    c <- baz(a, b)
  } yield c
```

Another important note about the ZIO environment is that the type inference works well on effect composition. After we composed all the application logic together, the compiler and also IDE can infer the proper type for the environment of the final effect.

In the example above, the compiler can infer the environment type of the `myApp` effect which is `ServiceA & ServiceB & ServiceC`.

### Accessing ZIO Environment

We have two types of accessors for the ZIO environment:
1. **Service Accessor (`ZIO.service`)** is used to access a specific service from the environment.
2. **Service Member Accessors (`ZIO.serviceWith` and `ZIO.serviceWithZIO`)** are used to access capabilities of a specific service from the environment.

:::note

To access the entire ZIO environment we can use `ZIO.environment*`, but we do not use these methods regularly to access ZIO services. Instead, we use service accessors and service member accessors.
:::

#### Service Accessor

To access a service from the ZIO environment, we can use the `ZIO.service` constructor. For example, in the following program we are going to access the `AppConfig` from the environment:

```scala

case class AppConfig(host: String, port: Int)

val myApp: ZIO[AppConfig, Nothing, Unit] =
  for {
    config <- ZIO.service[AppConfig]
    _      <- ZIO.logInfo(s"Application started with config: $config")
  } yield ()
```

To run the `myApp` effect, we should provide the `AppConfig` layer (we will talk about `ZLayer` on the next section):

```scala
object MainApp extends ZIOAppDefault {
  def run = myApp.provide(ZLayer.succeed(AppConfig("localhost", 8080)))
}
```

To access multiple services from the ZIO environment, we can do the same:

```scala

trait Foo
trait Bar
trait Baz

for {
  foo <- ZIO.service[Foo]  
  bar <- ZIO.service[Bar]
  bax <- ZIO.service[Baz]
} yield ()
```

When creating ZIO layers that have multiple dependencies, this can be helpful. We will discuss this pattern in the [Service Pattern](../service-pattern/service-pattern.md) section.

#### Service Member Accessors

Sometimes instead of accessing a service, we need to access the capabilities (members) of a service. Based on the return type of each capability, we can use one of these accessors:
- **ZIO.serviceWith**
- **ZIO.serviceWithZIO**

In [Service Pattern](../service-pattern/service-pattern.md), we use these accessors to write "accessor methods" for ZIO services.

Let's look at each one in more detail:

1. **ZIO.serviceWith** — When we are accessing service members whose return type is an ordinary value, we should use the `ZIO.serviceWith`.

In the following example, we need to use the `ZIO.serviceWith` to write accessor methods for all of the `AppConfig` members:

<!-- accessor methods are now deprecated, TODO: rewrite the following so it no longer needs accessor methods -->

```scala

case class AppConfig(host: String, port: Int, poolSize: Int)

object AppConfig {
  // Accessor Methods
  def host: ZIO[AppConfig, Nothing, String]  = ZIO.serviceWith(_.host) 
  def port: ZIO[AppConfig, Nothing, Int]     = ZIO.serviceWith(_.port)
  def poolSize: ZIO[AppConfig, Nothing, Int] = ZIO.serviceWith(_.poolSize)
}

val myApp: ZIO[AppConfig, Nothing, Unit] =
  for {
    host     <- AppConfig.host
    port     <- AppConfig.port
    _        <- ZIO.logInfo(s"The service will be service at $host:$port")
    poolSize <- AppConfig.poolSize
    _        <- ZIO.logInfo(s"Application started with $poolSize pool size")
  } yield ()
```

2. **ZIO.serviceWithZIO** — When we are accessing service members whose return type is a ZIO effect, we should use the `ZIO.serviceWithZIO`.

For example, in order to write the accessor method for the `foo` member of the `Foo` service, we need to use the `ZIO.serviceWithZIO` function:

<!-- accessor methods are now deprecated, TODO: rewrite the following so it no longer needs accessor methods
perhaps we can simply remove this?
-->

```scala

trait Foo {
  def foo(input: String): Task[Unit]
}

object Foo {
  // Accessor Method
  def foo(input: String): ZIO[Foo, Throwable, Unit] =
    ZIO.serviceWithZIO(_.foo(input))
}
```

## 2. ZEnvironment

`ZEnvironment` is a built-in type-level map for maintaining the environment of a `ZIO` data type. We don't typically use this data type directly. It's okay to skip learning it at the moment. We have a [separate article](zenvironment.md) about this data type.

## 3. ZLayer

`ZLayer[-RIn, +E, +ROut]` is a recipe to build an environment of type `ROut`, starting from a value `RIn`, and possibly producing an error `E` during creation.

---

## Layer

`Layer[+E, +ROut]` is a type alias for `ZLayer[Any, E, ROut]`, which represents a layer that doesn't require any services, it may fail with an error type of `E`, and returns `ROut` as its output.

```scala
type Layer[+E, +ROut] = ZLayer[Any, E, ROut]
```

---

## RLayer

`RLayer[-RIn, +ROut]` is a type alias for `ZLayer[RIn, Throwable, ROut]`, which represents a layer that requires `RIn` as its input, it may fail with `Throwable` value, or returns `ROut` as its output.

```scala
type RLayer[-RIn, +ROut]  = ZLayer[RIn, Throwable, ROut]
```

---

## TaskLayer

`TaskLayer[+ROut]` is a type alias for `ZLayer[Any, Throwable, ROut]`, which represents a layer that doesn't require any services as its input, it may fail with `Throwable` value, and returns `ROut` as its output.

```scala
type TaskLayer[+ROut] = ZLayer[Any, Throwable, ROut]
```

---

## ULayer

`ULayer[+ROut]` is a type alias for `ZLayer[Any, Nothing, ROut]`, which represents a layer that doesn't require any services as its input, it can't fail, and returns `ROut` as its output.

```scala
type ULayer[+ROut] = ZLayer[Any, Nothing, ROut]
```

---

## URLayer

`URLayer[-RIn, +ROut]` is a type alias for `ZLayer[RIn, Nothing, ROut]`, which represents a layer that requires `RIn` as its input, it can't fail, and returns `ROut` as its output.

```scala
type URLayer[-RIn, +ROut] = ZLayer[RIn, Nothing, ROut]
```

---

## ZEnvironment

A `ZEnvironment[R]` is a built-in type-level map for the `ZIO` data type which is responsible for maintaining the environment of a `ZIO` effect. The `ZIO` data type uses this map to maintain all the environmental services and their implementations.

For example, assume we have written a `ZEnvironment` containing all built-in services as below:

```scala

val environment: ZEnvironment[Console & Clock & Random & System] =
  ZEnvironment[Console, Clock, Random, System](
    Console.ConsoleLive,
    Clock.ClockLive,
    Random.RandomLive,
    System.SystemLive
  )
```

This map contains all built-in services and their corresponding implementations. If we evaluate the `ZEnvironment#toString` method, we can see the underlying type-level map something like this.

```scala
ZEnvironment(
  Map(
    Console -> (zio.Console$ConsoleLive$@76a3e297, 0),
    Clock   -> (zio.Clock$ClockLive$@4d3167f4, 1), 
    Random  -> (RandomScala(scala.util.Random$@4eb7f003), 2), 
    System  -> (zio.System$SystemLive$@eafc191, 3)
  )
)
```

From a ZIO environment point of view, we can think of `ZIO` as the following function:

```scala
type ZIO[R, E, A] = ZEnvironment[R] => Either[E, A]
or 
type ZIO[R, E, A] = ZEnvironment[R] => IO[E, A]
```

For example, the `ZIO[Foo & Bar, Throwable, String]` can be thought of as a function from `ZEnvironment[Foo & Bar]` to `Either[Throwable, String]`:

:::note

The `ZEnvironment` is useful for manually constructing and combining the ZIO environment. So, in most cases, we do not require working directly with this data type. So you can skip reading this page if you are not an advanced user.
:::

We can eliminate the environment of `ZIO[R, E, A]` by providing `ZEnvironment[R]` to that effect. 

Also, we can access the **whole** environment using `ZIO.environment`:

```scala

case class AppConfig(poolSize: Int)

val myApp: ZIO[AppConfig, IOException, Unit] =
  ZIO.environment[AppConfig].flatMap { env =>
    val config  = env.get[AppConfig]
    Console.printLine(s"Application started with config: $config")
  }

val eliminated: IO[IOException, Unit] =
  myApp.provideEnvironment(
    ZEnvironment(AppConfig(poolSize = 10))
  )
```

:::note

In most cases, we do not require using `ZIO.environment` to access the whole environment or the `ZIO#provideEnvironment` to provide effect dependencies. Therefore, most of the time, we use `ZIO.service*` and other `ZIO#provide*` methods to access a specific service from the environment or provide services to a ZIO effect.
:::

## Creation

To create an empty ZIO environment:

```scala

val empty: ZEnvironment[Any] = ZEnvironment.empty
```

To create a ZIO environment from a simple value:

```scala

case class AppConfig(host: String, port: Int)
val config: ZEnvironment[AppConfig] = ZEnvironment(AppConfig("localhost", 8080))
```

## Operations

To **combine** two or multiple environment we can use `union` or `++` operator:

```scala

case class AppConfig(host: String, port: Int)

val app: ZEnvironment[AppConfig] =
  ZEnvironment.empty ++ ZEnvironment(AppConfig("localhost", 8080))
```

To **add** a service to an environment:

```scala

case class AppConfig(host: String, port: Int)

val app: ZEnvironment[AppConfig] =
  ZEnvironment.empty.add(AppConfig("localhost", 8080))
```

To retrieve a service from the environment, we use `get` method:

```scala

case class AppConfig(host: String, port: Int)

val app: ZEnvironment[AppConfig] =
  ZEnvironment.empty.add(AppConfig("localhost", 8080))

val appConfig: AppConfig = app.get[AppConfig]
```

## Providing Multiple Instance of the Same Interface

We can express an effect's dependency on multiple services of the type `A` which are keyed by type `K` with `Map[K, A]`. For example, the `ZIO[Map[String, Database], Throwable, Unit]` is an effect that depends on multiple `Database` versions.

To access the specified service corresponding to a specific key, we can use the `ZIO.serviceAt[Service](key)` constructor. For example, to access a `Database` service which is specified by the "inmemory" key, we can write:

```scala
val database: URIO[Map[String, Database], Option[Database]] =
  ZIO.serviceAt[Database]("inmemory")
```

A service can be updated at the specified key using the `ZIO#updateServiceAt` operator.

### Multiple Config Example

Let's see how we can create a layer comprising multiple instances of `AppConfig`:

```scala

case class AppConfig(host: String, port: Int)

object AppConfig {
  val layer: ULayer[Map[String, AppConfig]] =
    ZLayer.succeedEnvironment(
      ZEnvironment(
        Map(
          "prod" -> AppConfig("production.myapp", 80),
          "dev" -> AppConfig("development.myapp", 8080)
        )
      )
    )
}
```

And here is the application which uses different `AppConfig` from the ZIO environment based on the value of the `APP_ENV` environment variable:

```scala

object MultipleConfigExample extends ZIOAppDefault {

  val myApp: ZIO[Map[String, AppConfig], String, Unit] = for {
    env <- System.env("APP_ENV")
      .flatMap(x => ZIO.fromOption(x))
      .orElseFail("The environment variable APP_ENV cannot be found.")
    config <- ZIO.serviceAt[AppConfig](env)
      .flatMap(x => ZIO.fromOption(x))
      .orElseFail(s"The $env config cannot be found in the ZIO environment")
    _ <- ZIO.logInfo(s"Application started with: $config")
  } yield ()

  def run =
    myApp.provide(AppConfig.layer)

}
```

### Multiple Database Example

Here is an example of providing multiple instances of the `Database` service to the ZIO environment:

```scala

trait Database {
  def add(key: String, value: Array[Byte]): ZIO[Any, Throwable, Unit]
}

object Database {
  val layer: ULayer[Map[String, Database]] = {
    ZLayer.succeedEnvironment(
      ZEnvironment(
        Map(
          "persistent" -> PersistentDatabase.apply(),
          "inmemory" -> InmemoryDatabase.apply()
        )
      )
    )
  }
}

case class InmemoryDatabase() extends Database {
  override def add(key: String, value: Array[Byte]): ZIO[Any, Throwable, Unit] =
    ZIO.unit <* ZIO.logInfo(s"new $key added to the inmemory database")
}

case class PersistentDatabase() extends Database {
  override def add(key: String, value: Array[Byte]): ZIO[Any, Throwable, Unit] =
    ZIO.unit <* ZIO.logInfo(s"new $key added to the persistent database")
}

object MultipleDatabaseExample extends ZIOAppDefault {
  val myApp = for {
    inmemory <- ZIO.serviceAt[Database]("inmemory")
      .flatMap(x => ZIO.fromOption[Database](x))
      .orElseFail("failed to find an in-memory database in the ZIO environment")
    persistent <- ZIO.serviceAt[Database]("persistent")
      .flatMap(x => ZIO.fromOption[Database](x))
      .orElseFail("failed to find an persistent database in the ZIO environment")
    _ <- inmemory.add("key1", "value1".getBytes(StandardCharsets.UTF_8))
    _ <- persistent.add("key2", "value2".getBytes(StandardCharsets.UTF_8))
  } yield ()

  def run = myApp.provideLayer(Database.layer)
}
```

---

## ZIO Environment Use-cases

ZIO Environment allows us to describe workflows which carry some context that is used in the course of executing the workflow. This context can be dived into two categories:

1. Local Capabilities, e.g. scopes and transactions
2. Business Logic, e.g. services and repositories

Let's discuss each of these in turn.

## Local Capabilities

The most idiomatic use of the ZIO environment is for:

- Describing as values workflows that use capabilities that are "local" to a particular context
- Where tracking the use of this context at the type level is helpful for reasoning about programs

Let's look at [`Scope`](../resource/scope.md), which I would say is the most idiomatic usage of the environment in ZIO itself, to see what each of these mean:

### 1. Local Contexts

The first criteria is that a capability be "local" to a particular context and not shared throughout the entire application. For example, in the case of `Scope` we typically don't just have one scope for our entire application but many smaller scopes, such as the scope for using a particular file. Usages of local context also have an operator that allows locally eliminating this capability, such as the `ZIO.scoped` operator which transforms a `ZIO[R with Scope, E, A]` to a `ZIO[R, E, A]`.

Other potential examples of this would be the [`ZState`](../state-management/zstate.md) data type in ZIO, which describes some local use of state and is eliminated by the `ZIO.stateful` operator, a `Transaction` representing the usage of a database transaction, or the context of a particular HTTP request (`RequestContext`).

Notice that using the environment type allows workflows that require these capabilities to be first class values that **compose naturally** and can have their own operators for working with them, which would not be possible if we defined them as methods that required us to explicitly pass around these capabilities.

### 2. Type-level Reasoning

The second criteria is that tracking the usage of this context be helpful for reasoning about programs. We can also use a [`FiberRef`](../state-management/fiberref.md) value to maintain some local context. However, when we do so, our usage of that context is not reflected at the type level.

For example, when we log something it will use a variety of contextual information including the current log level, the current log span, and the current log annotations but none of this is reflected at the type level:

```scala
object ZIO {
  def log(message: => String)(implicit trace: Trace): ZIO[Any, Nothing, Unit]
}
```

Not reflecting this usage of contextual information at the type level can be both an advantage and a disadvantage:

- The advantage is that it can create a simpler API because we do not clutter up the environment with additional dependencies.
- The disadvantage is that we can't track at the type level whether we are using contextual information or whether we have provided it.

In the case of logging this is clearly the right trade-off. Logging is a low level concern that we don't want to require us to update our type signatures, and there is essentially no harm in running a ZIO workflow that does logging without providing this log context since we can just log at some default log level without any log spans or annotations.

In contrast, in the case of `Scope` there is tremendous value in reflecting the use of `Scope` at the type level so we know whether a workflow is resourceful and can have operators that reflect at the type level that we have provided a `Scope` to part of our application. Similarly if we have a **database transaction** reflecting at the type level that some workflow needs to be done as part of a transaction and when we are "executing" a transaction is extremely valuable.

## Business Logic

The other potential use of the ZIO environment is describing the dependencies of our business logic itself. Normally, we implement higher level services in terms of lower level services using [constructor based dependency injection with ZLayer](../di/index.md).

```scala

trait HighLevelService {
  def doSomething: ZIO[Any, Nothing, Unit]
}

object HighLevelService {
  val live: ZLayer[LowLevelService, Nothing, HighLevelService] =
    ZLayer.fromFunction(HighLevelServiceLive(_))

  final case class HighLevelServiceLive(lowLevelService: LowLevelService) extends HighLevelService {
    def doSomething: ZIO[Any, Nothing, Unit] =
      ??? // implemented in terms of `LowLevelService`
  }
}

trait LowLevelService {
  def doSomethingElse: ZIO[Any, Nothing, Unit]
}

object LowLevelService {
  val live: ZLayer[Any, Nothing, LowLevelService] = ???
}
```

This allows us to avoid using `LowLevelService` in the environment and to not "leak" implementation details, since the dependency on `LowLevelService` is an implementation detail of `HighLevelService` that might not even exist if `HighLevelService` is refactored.

However, the question arises then of how we should work with `HighLevelService` in the core of our business logic or the center of the "onion" in the [onion architecture](../architecture/architectural-patterns.md#onion-architecture)?

There are two approaches to this.

### Everything as a Service

The first approach is just that everything is a service:

```scala
sealed trait ApplicationService {
  def run: ZIO[Any, Nothing, Unit]
}

object ApplicationService {
  val live: ZLayer[Any, Nothing, LowLevelService] = ???
  
  final case class ApplicationServiceLive(highLevelService: HighLevelService) extends ApplicationService {
    val run: ZIO[Any, Nothing, Unit] = ??? // business logic implemented in terms of high level services
  }
}

object Main extends ZIOAppDefault {
  val run =
    ZIO
      .serviceWithZIO[ApplicationService](_.run)
      .provide(
        ApplicationService.live,
        HighLevelService.live,
        LowLevelService.live
      )
}
```

This style avoids any usage of the ZIO environment that is not a local capability except for possibly a single time within `ZIOAppDefault`.

### Using ZIO Environment

There can be a feeling that defining this final `ApplicationLevelService` is unnecessary, and we would like to be able to write our business logic in terms of high level services directly without making it another service:

```scala

object Main extends ZIOAppDefault {

  val myProgramLogic: ZIO[HighLevelService, Nothing, Unit] =
    for {
      _ <- ZIO.serviceWithZIO[HighLevelService](_.doSomething)
      - <- otherLogicHere
    } yield ()

  val run =
    myProgramLogic.provide(
      HighLevelSevice.live,
      LowLevelService.live
    )
}
```

There has been some movement towards the "everything is a service" approach since it avoids the need to implement [service accessors](../service-pattern/accessor-methods.md) but it can be a matter of team style which of these approaches to use. Either way our program is the same except for whether in our business logic we call methods on services directly or use the environment for that.

To learn more about this approach please see [how we can use dependency injection with the service pattern](../di/dependency-injection-in-zio.md#dependency-injection-and-service-pattern).

---

## ZLayer

A `ZLayer[-RIn, +E, +ROut]` describes a layer of an application: every layer in an application requires some services as input `RIn` and produces some services as the output `ROut`.

We can think of a layer as mental model of an asynchronous function from `RIn` to the `Either[E, ROut]`:

```scala
type ZLayer[-RIn, +E, +ROut] = RIn => async Either[E, ROut]
```

For example, a `ZLayer[Socket & Persistence, Throwable, Database]` can be thought of as a function that map `Socket` and `Persistence` services into `Database` service:

```scala
(Socket, Persistence) => Database
```

So we can say that the `Database` service has two dependencies: `Socket` and `Persistence` services.

In some cases, a `ZLayer` may not have any dependencies or requirements from the environment. In this case, we can specify `Any` for the `RIn` type parameter. The [`Layer`](layer.md) type alias provided by ZIO is a convenient way to define a layer without requirements.

ZLayers are:

1. **Recipes for Creating Services** — They describe how to create services from given dependencies. For example, the `ZLayer[Socket & Database, Throwable, UserRepo]` is a recipe for building a service that requires `Socket` and `Database` service, and it produces a `UserRepo` service.

2. **An Alternative to Constructors** — We can think of `ZLayer` as a more powerful version of a constructor, it is an alternative way to represent a constructor. Like a constructor, it allows us to build the `ROut` service in terms of its dependencies (`RIn`).

3. **Composable** — Because of their excellent **composition properties**, layers are the idiomatic way in ZIO to create services that depend on other services. We can define layers that are relying on each other.

4. **Effectful and Resourceful** — The construction of ZIO layers can be effectful and resourceful. They can be acquired effectfully and safely released when the services are done being utilized or even in case of failure, interruption, or defects in the application. 
  
  For example, to create a recipe for a `Database` service, we should describe how the `Database` will be initialized using an acquisition action. In addition, it may contain information about how the `Database` releases its connection pools.

6. **Asynchronous** — Unlike class constructors which are blocking, `ZLayer` is fully asynchronous and non-blocking. Note that in non-blocking applications we typically want to avoid creating something that is blocking inside its constructor.

  For example, when we are constructing some sort of Kafka streaming service, we might want to connect to the Kafka cluster in the constructor of our service, which takes some time. So it wouldn't be a good idea to block inside the constructor. There are some workarounds for fixing this issue, but they are not as perfect as the ZIO solution which allows for asynchronous, non-blocking constructors.

6. **Parallelism** — ZIO layers can be acquired in parallel, unlike class constructors, which do not support parallelism. When we compose multiple layers and then acquire them, the construction of each layer will occur in parallel. This will reduce the initialization time of ZIO applications with a large number of dependencies.

  With ZIO ZLayer, our constructor could be asynchronous, but it could also block. We can acquire resources asynchronously or in a blocking fashion, and spend some time doing that, and we don't need to worry about it. That is not an anti-pattern. This is the best practice with ZIO. And that is because `ZLayer` has the full power of the `ZIO` data type, and as a result, we have strictly more power on our constructors with `ZLayer`.

7. **Resilient** — Layer construction can be resilient. So if the acquiring phase fails, we can have a schedule to retry the acquiring stage. This helps us write apps that are error-proof and respond appropriately to failures.

Let's see how we can create a layer:

## Creation

There are four main ways to create a ZLayer:
1. `ZLayer.succeed` for creating layers from simple values.
2. `ZLayer.scoped` for creating layers with _for comprehension_ style from resourceful effects.
3. `ZLayer.apply`/`ZLayer.fromZIO` for creating layers with _for comprehension_ style from effectual but not resourceful effects.
4. `ZLayer.fromFunction` for creating layers that are neither effectual nor resourceful.

Now let's look at each of these methods.

### From a Simple Value or an Existing Service

With `ZLayer.succeed` we can construct a `ZLayer` from a value. It returns a `ULayer[A]` value, which represents a layer of an application that has a service of type `A`:

```scala
def succeed[A: Tag](a: A): ULayer[A]
```

Using `ZLayer.succeed` we can create a layer containing _simple value_ or a _service_:

1. To create a layer from a _simple value_:

```scala

case class AppConfig(host: String, port: Int)

val configLayer: ULayer[AppConfig] = ZLayer.succeed(AppConfig("localhost", 8080))
```

In the example above, we created a `configLayer` that provides us an instance of `AppConfig`.

2. To create a layer from an _existing service_:

```scala

trait EmailService {
  def send(email: String, content: String): UIO[Unit]
}

object EmailService {
  val layer: ZLayer[Any, Nothing, EmailService] = 
    ZLayer.succeed( 
      new EmailService {
        override def send(email: String, content: String): UIO[Unit] = ???
      }
    )
}
```

### From Non-resourceful Effects

This is the for-comprehension way of creating a ZIO service using `ZLayer.apply`:

```scala

trait A
trait B
trait C
case class CLive(a: A, b: B) extends C

object CLive {
  val layer: ZLayer[A & B, Nothing, C] =
    ZLayer {
      for {
        a <- ZIO.service[A]
        b <- ZIO.service[B]
      } yield CLive(a, b)
    }
}
```

### From Functions

A `ZLayer[R, E, A]` can be thought of as a function from `R` to `A`. So we can convert functions to the `ZLayer` using the `ZLayer.fromFunction` constructor.

In the following example, the `CLive` implementation requires two `A` and `B` services, and we can easily convert that case class to a `ZLayer`:

```scala

trait A
trait B
trait C
case class CLive(a: A, b: B) extends C

object CLive {
  val layer: ZLayer[A & B, Nothing, C] = 
    ZLayer.fromFunction(CLive.apply _)
}
```

Below is a complete working example:

```scala

case class DatabaseConfig()

object DatabaseConfig {
  val live = ZLayer.succeed(DatabaseConfig())
}

case class Database(databaseConfig: DatabaseConfig)

object Database {
  val live: ZLayer[DatabaseConfig, Nothing, Database] =
    ZLayer.fromFunction(Database.apply _)
}

case class Analytics()

object Analytics {
  val live: ULayer[Analytics] = ZLayer.succeed(Analytics())
}

case class Users(database: Database, analytics: Analytics)

object Users {
  val live = ZLayer.fromFunction(Users.apply _)
}

case class App(users: Users, analytics: Analytics) {
  def execute: UIO[Unit] =
    ZIO.debug(s"This app is made from ${users} and ${analytics}")
}

object App {
  val live = ZLayer.fromFunction(App.apply _)
}

object MainApp extends ZIOAppDefault {

  def run =
    ZIO
      .serviceWithZIO[App](_.execute)
      .provide(
        (((DatabaseConfig.live >>> Database.live) ++ Analytics.live >>> Users.live) ++ Analytics.live) >>> App.live
      )
}
```

### Automatic Derivation

Simple layers can be derived using `ZLayer.derive`. See [Automatic ZLayer Derivation](./automatic-zlayer-derivation.md).

## Converting a Layer to a Scoped Value

Every `ZLayer` can be converted to a scoped `ZIO` by using `ZLayer.build`:

```scala

trait Database {
  def close: UIO[Unit]
}

object Database {
  def connect: ZIO[Any, Throwable, Database] = ???
}

val database: ZLayer[Any, Throwable, Database] =
  ZLayer.scoped {
    ZIO.acquireRelease {
      Database.connect.debug("connecting to the database")
    } { database =>
      database.close
    }
  }

val scopedDatabase: ZIO[Scope, Throwable, ZEnvironment[Database]] =
  database.build
```

## Falling Back to an Alternate Layer

If a layer fails, we can provide an alternative layer by using `ZLayer#orElse` so it will fall back to the second layer:

```scala

trait Database

val postgresDatabaseLayer: ZLayer[Any, Throwable, Database] = ???
val inmemoryDatabaseLayer: ZLayer[Any, Throwable, Database] = ???

val databaseLayer: ZLayer[Any, Throwable, Database] =
  postgresDatabaseLayer.orElse(inmemoryDatabaseLayer)
```

## Converting a Layer to a ZIO Application

Sometimes our entire application is a ZIO Layer, e.g. an HTTP Server, so by calling the `ZLayer#launch` we can convert that to a ZIO application. This will build the layer and use it until it is interrupted.

```scala
object MainApp extends ZIOAppDefault {

  val httpServer: ZLayer[Any, Nothing, HttpServer] =
    ZLayer.make[HttpServer](
      JsonParserLive.layer,
      TemplateEngineLive.layer 
    )

  def run = httpServer.launch

}
``` 

## Retrying

We can retry constructing a layer in case of failure:

```scala

val databaseLayer: ZLayer[Any, Throwable, DatabaseConnection]   = ???

val retriedLayer : ZLayer[Clock, Throwable, DatabaseConnection] = databaseLayer.retry(Schedule.fibonacci(1.second))
```

## Layer Projection

We can project out a part of `ZLayer` by providing a projection function to the `ZLayer#project` method:

```scala

case class Connection(host: String, port: Int) 
case class Login(user: String, password: String)

case class DBConfig(
  connection: Connection, 
  login: Login
)

val connection: ZLayer[DBConfig, Nothing, Connection] = 
  ZLayer.service[DBConfig].project(_.connection)
```

## Tapping

We can perform a specified effect based on the success or failure result of the layer using `ZLayer#tap`/`ZLayer#tapError`. This would not change the layer's signature:

```scala

case class AppConfig(host: String, port: Int)

val config: ZLayer[Any, Throwable, AppConfig] =
  ZLayer.fromZIO(
    ZIO.attempt(???) // reading config from a file
  )

val res: ZLayer[Any, Throwable, AppConfig] =
  config
    .tap(cnf => ZIO.debug(s"layer acquisition succeeded with $cnf"))
    .tapError(err => ZIO.debug(s"error occurred during reading the config $err"))
```

---

## Introduction to ZIO's Control Flow Operators

Although we have access to built-in Scala control flow structures, ZIO has several control flow combinators. In this section, we are going to introduce different ways of controlling flows in ZIO applications.

## `if` Expression

When working with ZIO values, we can use built-in Scala if-then-else expressions:

```scala

def validateWeightOption(weight: Double): ZIO[Any, Nothing, Option[Double]] =
  if (weight >= 0)
    ZIO.some(weight)
  else
    ZIO.none
```

Also, we can encode invalid inputs using the error channel:

```scala

def validateWeightOrFail(weight: Double): ZIO[Any, String, Double] =
  if (weight >= 0)
    ZIO.succeed(weight)
  else
    ZIO.fail(s"negative input: $weight")
```

Even if the input has side effects, we can use `ZIO#flatMap` to access the raw value and write the if-then-else expression:

```scala

def validateWeightOrFailZIO[R](weight: ZIO[R, Nothing, Double]): ZIO[R, String, Double] =
  weight.flatMap { w =>
    if (w >= 0)
      ZIO.succeed(w)
    else
      ZIO.fail(s"negative input: $w")
  }
```

## Conditional Operators

### `when`

We can also use ZIO's combinators that are the moral equivalent to these expressions:

Instead of `if (p) expression` we can use the `ZIO.when` or `ZIO#when` operator:

```scala

def validateWeightOption(weight: Double): ZIO[Any, Nothing, Option[Double]] =
  ZIO.when(weight > 0)(ZIO.succeed(weight))
```

If the predicate is effectful, we can use `ZIO.whenZIO` or `ZIO#whenZIO` operators.

For example, the following function creates a random option of int value:

```scala

def randomIntOption: ZIO[Any, Nothing, Option[Int]] =
  Random.nextInt.whenZIO(Random.nextBoolean)
```

Another nice variant of the `when` operator is `ZIO.whenCase` and also the `ZIO.whenCaseZIO`. Using these operators, we can run an effect when our provided effectful `PartialFunction` matches the given raw or effectful input. The important note regarding this operator is that it is safe, so it will do nothing if the value does not match.

Let's try to write a game, which asks users to choose which game to play:

```scala

def minesweeper(level: String)     = ZIO.attempt(???)
def ticTacToe                      = ZIO.attempt(???)
def snake(rows: Int, columns: Int) = ZIO.attempt(???)

def myApp =
  ZIO.whenCaseZIO {
    (Console.print(
      "Please choose one game (minesweeper, snake, tictactoe)? "
    ) *> Console.readLine).orDie
  } {
    case "minesweeper" =>
      Console.print(
        "Please enter the level of the game (easy/hard/medium)?"
      ) *> Console.readLine.flatMap(minesweeper)
    case "snake" =>
      Console.printLine(
        "Please enter the size of the game: "
      ) *> Console.readLine.mapAttempt(_.toInt).flatMap(n => snake(n, n))
    case "tictactoe" => ticTacToe
  }
```

### `unless`

The `ZIO.unless` and `ZIO#unless` operators are like `when` operators, but they are moral equivalent for the `if (!p) expression` construct.

### `ifZIO`

This operator takes an _effectful predicate_, if that predicate is evaluated to true, it will run the `onTrue` effect, otherwise it will run the `onFalse` effect.

Let's try to write a simple virtual flip function:

```scala

def flipTheCoin: ZIO[Any, IOException, Unit] =
  ZIO.ifZIO(Random.nextBoolean)(
    onTrue = Console.printLine("Head"),
    onFalse = Console.printLine("Tail")
  )
```

## Loop Operators

In imperative Scala code bases, sometimes we may use `while(condition) { statement }` or `do { statement } while (condition)` constructs to perform loops:

```scala
object MainApp extends scala.App {
  def printNumbers(from: Int, to: Int): Unit = {
    var i = from
    while (i <= to) {
      println(s"$i")
      i = i + 1
    }
  }

  printNumbers(1, 3)
}
```

But in functional Scala, we tend to avoid mutable variables. So to have a loop, we would like to use recursion. Let's rewrite the previous example using recursion:

```scala

object MainApp extends scala.App {
  @tailrec
  def printNumbers(from: Int, to: Int): Unit = {
    if (from <= to) {
      println(s"$from")
      printNumbers(from + 1, to)
    } else ()
  }

  printNumbers(1, 3)
}
// 1
// 2
// 3
```

In this example, we wrote a recursive function that prints numbers from 1 to 3. While the last effort doesn't use a mutable variable, it's not a pure solution. We have a `println` statement inside our solution, calling this function is not pure so the whole solution is not pure. We know that we can model effectful functions using the ZIO effect system. So let's try rewrite that using ZIO:

```scala

object MainApp extends ZIOAppDefault {
  def printNumbers(from: Int, to: Int): ZIO[Any, IOException, Unit] = {
    if (from <= to)
      Console.printLine(s"$from") *>
        printNumbers(from + 1, to)
    else ZIO.unit
  }

  def run = printNumbers(1, 5)
}
```

ZIO provides some loop combinators that help us avoid the need to write explicit recursions. This means that we can do almost anything we want to do without using explicit recursions. Let's rewrite the last solution using `ZIO.loopDiscard`:

```scala

object MainApp extends ZIOAppDefault {
  def printNumbers(from: Int, to: Int): ZIO[Any, IOException, Unit] = {
    ZIO.loopDiscard(from)(_ <= to, _ + 1)(i => Console.printLine(i))
  }

  def run = printNumbers(1, 3)
}
```

After this short introduction to writing loops in functional Scala, now let us go further into ZIO-specific combinators for writing loops:

### `loop`

The `ZIO.loop` operator takes an initial state, then repeatedly changes the state based on the given `inc` function, until the given `cont` function evaluates to true:

```scala
object ZIO {
  def loop[R, E, A, S](
    initial: => S
  )(cont: S => Boolean, inc: S => S)(body: S => ZIO[R, E, A]): ZIO[R, E, List[A]]

  def loopDiscard[R, E, S](
    initial: => S
  )(cont: S => Boolean, inc: S => S)(body: S => ZIO[R, E, Any]): ZIO[R, E, Unit]
```

`ZIO.loop` collects all intermediate states in a list and returns it finally, while the `ZIO.loopDiscard` discards all results.

We can think of `ZIO.loop` as a moral equivalent of the following while loop:

```scala
var s  = initial
var as = List.empty[A]

while (cont(s)) {
  as = body(s) :: as
  s  = inc(s)
}

as.reverse
```

Let's try some examples:

```scala

val r1: ZIO[Any, Nothing, List[Int]] =
  ZIO.loop(1)(_ <= 5, _ + 1)(n => ZIO.succeed(n)).debug
// List(1, 2, 3, 4, 5)

val r2: ZIO[Any, Nothing, List[Int]] =
  ZIO.loop(1)(_ <= 5, _ + 1)(n => ZIO.succeed(n * 2)).debug
// List(2, 4, 6, 8, 10)

val r3: ZIO[Any, IOException, List[Unit]] =
  ZIO.loop(1)(_ <= 5, _ + 1) { index =>
    Console.printLine(s"Currently at index $index")
  }.debug
// Currently at index 1
// Currently at index 2
// Currently at index 3
// Currently at index 4
// Currently at index 5
// List((), (), (), (), ())

val r4: ZIO[Any, IOException, Unit] =
  ZIO.loopDiscard(1)(_ <= 5, _ + 1) { index =>
    Console.printLine(s"Currently at index $index")
  }.debug
// Currently at index 1
// Currently at index 2
// Currently at index 3
// Currently at index 4
// Currently at index 5
// ()

val r5: ZIO[Any, IOException, List[String]] =
  Console.printLine("Please enter three names: ") *>
    ZIO.loop(1)(_ <= 3, _ + 1) { n =>
      Console.print(s"$n. ") *> Console.readLine
    }.debug
// Please enter three names:
// 1. John
// 2. Jane
// 3. Joe
// List(John, Jane, Joe)
```

### `iterate`

To iterate with the given effectful operation we can use the `ZIO.iterate` combinator. During each iteration, it uses an effectful `body` operation to change the state, and it will continue the iteration while the `cont` function evaluates to true:

```scala
object ZIO {
  def iterate[R, E, S](
    initial: => S
  )(cont: S => Boolean)(body: S => ZIO[R, E, S]): ZIO[R, E, S]
}
```

This operator is a moral equivalent of the following while loop:

```scala
var s = initial
while (cont(s)) {
  s = body(s)
}
s
```

Let's try some examples:

```scala

val r1 = ZIO.iterate(1)(_ <= 5)(s => ZIO.succeed(s + 1)).debug
// 6

val r2 = ZIO.iterate(1)(_ <= 5)(s => ZIO.succeed(s * 2).debug).debug("result")
// 2
// 4
// 8
// result: 8
```

Here's another example. Assume we want to take many names from the user using the terminal. We don't know how many names the user is going to enter. We can ask the user to write "exit" when all inputs are finished. To write such an application, we can use recursion like below:

```scala

def getNames: ZIO[Any, IOException, List[String]] =
  Console.print("Please enter all names") *>
    Console.printLine(" (enter \"exit\" to indicate end of the list):") *> {
      def loop(
          names: List[String]
      ): ZIO[Any, IOException, List[String]] = {
        Console.print(s"${names.length + 1}. ") *> Console.readLine
          .flatMap {
            case "exit" => ZIO.succeed(names)
            case name   => loop(names.appended(name))
          }
      }
      loop(List.empty[String])
    }
// Please enter all names (enter "exit" to indicate end of the list):
// 1. John
// 2. Jane
// 3. Joe
// 4. exit
// List(John, Jane, Joe)
```

Instead of manually writing recursions, we can rely on well-tested ZIO combinators. So let's rewrite this application using the `ZIO.iterate` operator:

```scala

def getNames: ZIO[Any, IOException, List[String]] =
  Console.print("Please enter all names") *>
    Console.printLine(" (enter \"exit\" to indicate end of the list):") *>
    ZIO.iterate((List.empty[String], true))(_._2) { case (names, _) =>
      Console.print(s"${names.length + 1}. ") *>
        Console.readLine.map {
          case "exit" => (names, false)
          case name   => (names.appended(name), true)
        }
    }
    .map(_._1)
    .debug
// Please enter all names (enter "exit" to indicate end of the list):
// 1. John
// 2. Jane
// 3. Joe
// 4. exit
// List(John, Jane, Joe)
```

### `foreach`

Note that, in several cases, we can avoid these low-level operators and instead use high-level ones. For example, let's try to rewrite the `r5` with `ZIO.foreach`:

```scala

Console.printLine("Please enter three names:") *>
  ZIO.foreach(1 to 3) { index =>
    Console.print(s"$index. ") *> Console.readLine
  }.debug
// Please enter three names:
// 1. John
// 2. Jane
// 3. Joe
// Vector(John, Jane, Joe)
```

## try/catch/finally

When working with resources, just like Scala's `try`/`catch`/`finally` construct, in ZIO we have a similar operator called `acquireRelease` and also `ensuring`. We discussed them in more detail in the [resource management section](../resource/index.md). But, for now, we want to focus on their control flow behaviors.

Let's learn about the `ZIO.acquireReleaseWith` operator. This operator takes three effects:

1. **`acquire`**, an effect that describes the resource acquisition
2. **`release`**, an effect that describes the release of the resource
3. **`use`**, an effect that describes resource usage

```scala

ZIO.acquireReleaseWith(acquire = ???)(release = ???)(use = ???)
```

This operator guarantees us that if the _resource acquisition (acquire)_ succeeds, the _release_ effect will be executed whether the _use_ effect succeeded or not:

```scala

def wordCount(fileName: String): ZIO[Any, Throwable, Int] = {
  def openFile(name: => String): ZIO[Any, IOException, Source] =
    ZIO.attemptBlockingIO(Source.fromFile(name))

  def closeFile(source: => Source): ZIO[Any, Nothing, Unit] =
    ZIO.succeedBlocking(source.close())

  def wordCount(source: => Source): ZIO[Any, Throwable, Int] =
    ZIO.attemptBlocking(source.getLines().length)

  ZIO.acquireReleaseWith(openFile(fileName))(closeFile(_))(wordCount(_))
}
```

Let's try a simple `acquireRelease` workflow to see how its control flow works:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    ZIO.acquireReleaseWith {
      ZIO.succeed("resource").tap(r => ZIO.debug(s"$r acquired"))
    } { i =>
      ZIO.debug(s"$i released")
    } { i =>
      ZIO.debug(s"start using $i")
    }
}
// Output:
// resource acquired
// start using resource
// resource released
```

---

## Cause

The `ZIO[R, E, A]` effect is polymorphic in values of type `E` and we can work with any error type that we want, but there is a lot of information that is not inside an arbitrary `E` value. So as a result ZIO needs somewhere to store things like **unexpected errors or defects**, **stack and execution traces**, **cause of fiber interruptions**, and so forth.

ZIO is very strict about preserving the full information related to a failure. It captures all type of errors into the `Cause` data type. ZIO uses `Cause[E]` to store the full story of failure, so its error model is **lossless**. It doesn't throw away information related to the failure result. So we can figure out exactly what happened during the operation of our effects.

It is important to note that `Cause` is the underlying data type for the ZIO data type, and we don't usually deal with it directly. Even though we do not deal with it very often, anytime we want, we can access the `Cause` data structure, which gives us total access to all parallel and sequential errors in our codebase.

## Cause Internals

ZIO uses a data structure from functional programming called a _semiring_ for the `Cause` data type. **It allows us to take a base type `E` that represents the error type and then capture the sequential and parallel composition of errors in a fully lossless fashion**.

The following snippet shows how `Cause` is designed as a semiring data structure:

```scala
sealed abstract class Cause[+E] extends Product with Serializable { self =>

  def trace: Trace = ???

  final def ++[E1 >: E](that: Cause[E1]): Cause[E1] = Then(self, that)
  final def &&[E1 >: E](that: Cause[E1]): Cause[E1] = Both(self, that)
}

object Cause extends Serializable {
  case object Empty extends Cause[Nothing]
  final case class Fail[+E](value: E, override val trace: Trace) extends Cause[E]
  final case class Die(value: Throwable, override val trace: Trace) extends Cause[Nothing]
  final case class Interrupt(fiberId: FiberId, override val trace: Trace) extends Cause[Nothing]
  final case class Stackless[+E](cause: Cause[E], stackless: Boolean) extends Cause[E]
  final case class Then[+E](left: Cause[E], right: Cause[E]) extends Cause[E]
  final case class Both[+E](left: Cause[E], right: Cause[E]) extends Cause[E]
}
```

Using the `Cause` data structure described above, ZIO can capture all errors inside the application.

## Cause Variations

There are several causes for various errors. In this section, we will describe each of these causes. We will see how they can be created manually or how they will be automatically generated as the underlying error management data type of a ZIO application.

### Empty

The `Empty` cause indicates the lack of errors. We use `Cause.empty` constructor to create an `Empty` cause. Using `ZIO.failCause` we can create a ZIO effect that has an empty cause:

```scala

ZIO.failCause(Cause.empty).cause.debug
// Empty
```

Also, we can use `ZIO#cause` to uncover the underlying cause of an effect. For example, we know that `ZIO.succeed(5)` has no errors. So, let's check that:

```
ZIO.succeed(5).cause.debug
// Empty

ZIO.attempt(5).cause.debug
// Empty
```

### Fail

The `Fail` cause indicates the cause of an _expected error_ of type `E`. We can create one using the `Cause.fail` constructor:

```scala

ZIO.failCause(Cause.fail("Oh uh!")).cause.debug
// Fail(Oh uh!,Trace(Runtime(2,1646395282),Chunk(<empty>.MainApp.run(MainApp.scala:4))))
```

Let's uncover the cause of some ZIO effects especially when we combine them:

```scala

ZIO.fail("Oh uh!").cause.debug
// Fail(Oh uh!,Trace(Runtime(2,1646395627),Chunk(<empty>.MainApp.run(MainApp.scala:3))))

(ZIO.fail("Oh uh!") *> ZIO.dieMessage("Boom!") *> ZIO.interrupt).cause.debug
// Fail(Oh uh!,Trace(Runtime(2,1646396370),Chunk(<empty>.MainApp.run(MainApp.scala:6))))

(ZIO.fail("Oh uh!") <*> ZIO.fail("Oh Error!")).cause.debug
// Fail(Oh uh!,Trace(Runtime(2,1646396419),Chunk(<empty>.MainApp.run(MainApp.scala:9))))

val myApp: ZIO[Any, String, Int] =
  for {
    i <- ZIO.succeed(5)
    _ <- ZIO.fail("Oh uh!")
    _ <- ZIO.dieMessage("Boom!")
    _ <- ZIO.interrupt
  } yield i
myApp.cause.debug
// Fail(Oh uh!,Trace(Runtime(2,1646397126),Chunk(<empty>.MainApp.myApp(MainApp.scala:13),<empty>.MainApp.run(MainApp.scala:17))))
```

### Die

The `Die` cause indicates a defect, an _unexpected failure_ of type `Throwable`. It contains the stack trace of the defect that occurred. We can use `Cause.die` to create one:

```scala

ZIO.failCause(Cause.die(new Throwable("Boom!"))).cause.debug
// Die(java.lang.Throwable: Boom!,Trace(Runtime(2,1646479908),Chunk(<empty>.MainApp.run(MainApp.scala:3))))
```

If we have a bug in our code and something throws an unexpected exception, that information would be described inside a `Die`. Let's try to investigate some ZIO code that will die:

```scala

ZIO.succeed(5 / 0).cause.debug
// Die(java.lang.ArithmeticException: / by zero,Trace(Runtime(2,1646480112),Chunk(zio.internal.FiberContext.runUntil(FiberContext.scala:538),<empty>.MainApp.run(MainApp.scala:3))))

ZIO.dieMessage("Boom!").cause.debug
// Stackless(Die(java.lang.RuntimeException: Boom!,Trace(Runtime(2,1646398246),Chunk(<empty>.MainApp.run(MainApp.scala:7)))),true)
```

It is worth noting that the latest example is wrapped by the `Stackless` cause in the previous example. We will discuss `Stackless` further, but for now, it is enough to know that `Stackless` includes fewer stack traces than the `Die` cause.

### Interrupt

The `Interrupt` cause indicates a fiber interruption which contains information of the _fiber id_ of the interrupted fiber, and also the corresponding stack trace. Let's try an example of:

```scala

ZIO.interrupt.cause.debug
// Interrupt(Runtime(2,1646471715),Trace(Runtime(2,1646471715),Chunk(<empty>.MainApp.run(MainApp.scala:3))))

ZIO.never.fork
  .flatMap(f => f.interrupt *> f.join)
  .cause
  .debug
// Interrupt(Runtime(2,1646472025),Trace(Runtime(13,1646472025),Chunk(<empty>.MainApp.run(MainApp.scala:7))))
```

### Stackless

The `Stackless` cause stores stack traces and execution traces. It has a boolean `stackless` flag which denotes whether the ZIO runtime should print the full stack trace of the inner cause or just print a few lines of it.

For example, `ZIO.dieMessage` uses `Stackless`:

```scala

ZIO.dieMessage("Boom!").cause.debug
// Stackless(Die(java.lang.RuntimeException: Boom!,Trace(Runtime(2,1646477970),Chunk(<empty>.MainApp.run(MainApp.scala:3)))),true)
```

So when we run it the following stack traces will be printed:

```scala
timestamp=2022-03-05T11:08:19.530710679Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.RuntimeException: Boom!
at <empty>.MainApp.run(MainApp.scala:3)"
```

While `ZIO.die` doesn't use `Stackless` cause:

```scala

ZIO.die(new Throwable("Boom!")).cause.debug
// Die(java.lang.Exception: Boom!,Trace(Runtime(2,1646479093),Chunk(<empty>.MainApp.run(MainApp.scala:3))))
```

So it prints the full stack trace:

```scala
timestamp=2022-03-05T11:19:12.666418357Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.Exception: Boom!
	at MainApp$.$anonfun$run$1(MainApp.scala:4)
	at zio.ZIO$.$anonfun$die$1(ZIO.scala:3384)
	at zio.internal.FiberContext.runUntil(FiberContext.scala:255)
	at zio.internal.FiberContext.run(FiberContext.scala:115)
	at zio.internal.ZScheduler$$anon$1.run(ZScheduler.scala:151)
	at <empty>.MainApp.run(MainApp.scala:4)"
```

### Both

When we are doing parallel computation, the effect can fail for more than one reason. If we are doing two things at once and both of them fail then we actually have two errors. So, the `Both` cause stores the composition of two parallel causes.

For example, if we run two parallel fibers with `zipPar` and all of them fail, their causes will be encoded with `Both`:

```scala

val myApp: ZIO[Any, String, Unit] =
  for {
    f1 <- ZIO.fail("Oh uh!").fork
    f2 <- ZIO.dieMessage("Boom!").fork
    _ <- (f1 <*> f2).join
  } yield ()
myApp.cause.debug
// Both(Fail(Oh uh!,Trace(Runtime(13,1646481219),Chunk(<empty>.MainApp.myApp(MainApp.scala:5)))),Stackless(Die(java.lang.RuntimeException: Boom!,Trace(Runtime(14,1646481219),Chunk(<empty>.MainApp.myApp(MainApp.scala:6)))),true))
```

If we run the `myApp` effect, in the stack trace we can see two exception traces occurred on two separate fibers:

```scala
timestamp=2022-03-05T12:37:46.831096692Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-13" java.lang.String: Oh uh!
at <empty>.MainApp.myApp(MainApp.scala:5)
  Exception in thread "zio-fiber-14" java.lang.RuntimeException: Boom!
  at <empty>.MainApp.myApp(MainApp.scala:6)"
```

Other parallel operators are also the same, for example, ZIO encodes the underlying cause of `(ZIO.fail("Oh uh!") <&> ZIO.dieMessage("Boom!"))` with the `Both` cause.

### Then

ZIO uses `Then` cause to encode sequential errors. For example, if we perform ZIO's analog of `try-finally` (e.g. `ZIO#ensuring`), and both `try` and `finally` blocks fail, their causes are encoded with `Then`:

```scala

val myApp =
  ZIO.fail("first")
    .ensuring(ZIO.die(throw new Exception("second")))

myApp.cause.debug
// Then(Fail(first,Trace(Runtime(2,1646486975),Chunk(<empty>.MainApp.myApp(MainApp.scala:4),<empty>.MainApp.myApp(MainApp.scala:5),<empty>.MainApp.run(MainApp.scala:7)))),Die(java.lang.Exception: second,Trace(Runtime(2,1646486975),Chunk(zio.internal.FiberContext.runUntil(FiberContext.scala:538),<empty>.MainApp.myApp(MainApp.scala:5),<empty>.MainApp.run(MainApp.scala:7)))))
```

If we run the `myApp` effect, we can see the following stack trace:

```scala
timestamp=2022-03-05T13:30:17.335173071Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: first
	at <empty>.MainApp.myApp(MainApp.scala:4)
	at <empty>.MainApp.myApp(MainApp.scala:5)
	Suppressed: java.lang.Exception: second
		at MainApp$.$anonfun$myApp$3(MainApp.scala:5)
		at zio.ZIO$.$anonfun$die$1(ZIO.scala:3384)
		at zio.internal.FiberContext.runUntil(FiberContext.scala:255)
		at zio.internal.FiberContext.run(FiberContext.scala:115)
		at zio.internal.ZScheduler$$anon$1.run(ZScheduler.scala:151)
		at zio.internal.FiberContext.runUntil(FiberContext.scala:538)
		at <empty>.MainApp.myApp(MainApp.scala:5)"
```

As we can see in the above stack trace, the _first_ failure was suppressed by the _second_ defect.

---

## Exit

An `Exit[E, A]` value describes how fibers end life. It has two possible values:
- `Exit.Success` contain a success value of type `A`. 
- `Exit.Failure` contains a failure [Cause](cause.md) of type `E`.

This is how the `Exit` data type is defined:

```scala
sealed abstract class Exit[+E, +A] extends Product with Serializable { self =>
  // Exit operators
}
object Exit {
  final case class Success[+A](value: A)        extends Exit[Nothing, A]
  final case class Failure[+E](cause: Cause[E]) extends Exit[E, Nothing]
}
```

We can call `ZIO#exit` on our effect to determine the Success or Failure of our fiber:

```scala

val result: ZIO[Any, IOException, Unit] = 
  for {
    successExit <- ZIO.succeed(1).exit
    _ <- successExit match {
      case Exit.Success(value) =>
        printLine(s"exited with success value: ${value}")
      case Exit.Failure(cause) =>
        printLine(s"exited with failure state: $cause")
    }
  } yield ()
```

## Pre-constructed Exit Values

ZIO provides several pre-constructed `Exit` values for common cases:

- `Exit.unit` — A success exit with a `Unit` value
- `Exit.none` — A success exit with a `None` value (type: `Exit[Nothing, Option[Nothing]]`)

These values are useful when you need to return a pre-made exit without constructing it manually:

```scala

// Using Exit.unit for effects that only care about success or failure
val unitExit: Exit[String, Unit] = Exit.unit

// Using Exit.none for optional values
val noneExit: Exit[String, Option[Nothing]] = Exit.none
```

---

## Core Data Types

In this section we are going to talk about the basic data types that are required to build a ZIO application:

- **[ZIO](zio/zio.md)** — `ZIO` is a value that models an effectful program, which might fail or succeed.
    + **[UIO](zio/uio.md)** — `UIO[A]` is a type alias for `ZIO[Any, Nothing, A]`.
    + **[URIO](zio/urio.md)** — `URIO[R, A]` is a type alias for `ZIO[R, Nothing, A]`.
    + **[Task](zio/task.md)** — `Task[A]` is a type alias for `ZIO[Any, Throwable, A]`.
    + **[RIO](zio/rio.md)** — `RIO[R, A]` is a type alias for `ZIO[R, Throwable, A]`.
    + **[IO](zio/io.md)** — `IO[E, A]` is a type alias for `ZIO[Any, E, A]`.
- **[ZIOApp](zioapp.md)** — `ZIOApp` and the `ZIOAppDefault` are entry points for ZIO applications.
- **[Runtime](runtime.md)** — `Runtime[R]` is capable of executing tasks within an environment `R`.
- **[Exit](exit.md)** — `Exit[E, A]` describes the result of executing an `IO` value.
- **[Cause](cause.md)** — `Cause[E]` is a description of a full story of a fiber failure.

---

## Runtime

A `Runtime[R]` is capable of executing tasks within an environment `R`.

To run an effect, we need a `Runtime`, which is capable of executing effects. Runtimes bundle a thread pool together with the environment that effects need.

## What is a Runtime System?

Whenever we write a ZIO program, we create a ZIO effect from ZIO constructors plus using its combinators. We are building a blueprint. A ZIO effect is just a data structure that describes the execution of a concurrent program. So we end up with a tree data structure that contains lots of different data structures combined together to describe what the ZIO effect should do. This data structure doesn't do anything, it is just a description of a concurrent program.

So the most important thing we should keep in mind when we are working with a functional effect system like ZIO is that when we are writing code, printing a string onto the console, reading a file, querying a database, and so forth, we are just writing a workflow or blueprint of an application. We are just building a data structure.

So how can ZIO run these workflows? This is where the ZIO Runtime System comes into play. Whenever we run an `unsafe.run` function, the Runtime System is responsible for stepping through all the instructions described by the ZIO effect and executing them.

To simplify everything, we can think of a Runtime System like a black box that takes both the ZIO effect (`ZIO[R, E, A]`) and its environment (`R`). It will run this effect and return its result as an `Either[E, A]` value.

![ZIO Runtime System](/img/zio-runtime-system.svg)

## Responsibilities of the Runtime System

Runtime Systems have a lot of responsibilities:

1. **Execute every step of the blueprint** — They have to execute every step of the blueprint in a while loop until it's done.

2. **Handle unexpected errors** — They have to handle unexpected errors, not just the expected ones but also the unexpected ones. 

3. **Spawn concurrent fibers** — They are actually responsible for the concurrency that effect systems have. They have to spawn a new fiber every time we call `fork` on an effect.

4. **Cooperatively yield to other fibers** — They have to cooperatively yield to other fibers so that fibers that are sort of hogging the spotlight, don't get to monopolize all the CPU resources. They have to make sure that the fibers split the CPU cores among all the fibers that are working.

5. **Capture execution and stack traces** — They have to keep track of where we are in the progress of our own user-land code, so detailed execution traces can be captured. 

6. **Ensure finalizers are run appropriately** — They have to ensure finalizers are run appropriately at the right point in all circumstances to make sure that resources are closed and clean-up logic is executed. This is the feature that powers `Scope` and all the other resource-safe constructs in ZIO.

7. **Handle asynchronous callbacks** — They have to handle this messy job of dealing with asynchronous callbacks. So we don't have to deal with async code. When we are using ZIO, everything is just async out of the box. 

## Running a ZIO Effect

There are two common ways to run a ZIO effect. Most of the time, we use the [`ZIOAppDefault`](zioapp.md) trait. There are, however, some advanced use cases for which we need to directly feed a ZIO effect into the runtime system's `unsafe.run` method:

```scala

object RunZIOEffectUsingUnsafeRun extends scala.App {
  val myAppLogic = for {
    _ <- Console.printLine("Hello! What is your name?")
    n <- Console.readLine
    _ <- Console.printLine("Hello, " + n + ", good to meet you!")
  } yield ()

  Unsafe.unsafe { implicit unsafe =>
      zio.Runtime.default.unsafe.run(
        myAppLogic
      ).getOrThrowFiberFailure()
  }
}
```

We don't usually use this method to run our effects. One of the use cases of this method is when we are integrating legacy (non-effectful) code with the ZIO effect. It helps us to refactor a large legacy code base into a ZIO effect gradually: assume we have decided to refactor a component in the middle of an application and rewrite that with ZIO. We can start rewriting that component with the ZIO effect and then integrate that component with the existing code base using the `unsafe.run` function.

## Default Runtime

ZIO contains a default runtime called `Runtime.default` designed to work well for mainstream usage. It is implemented as below:

```scala
object Runtime {
  val default: Runtime[Any] =
    Runtime(ZEnvironment.empty, FiberRefs.empty, RuntimeFlags.default)
}
```

The default runtime provides the minimum capabilities to bootstrap execution of ZIO tasks.

We can easily access the default `Runtime` to run an effect:

```scala
object MainApp extends scala.App {
  val myAppLogic = ZIO.succeed(???)

  val runtime = Runtime.default

  Unsafe.unsafe { implicit unsafe =>
    runtime.unsafe.run(myAppLogic).getOrThrowFiberFailure()
  }
}
```

## Top-level and Locally Scoped Runtimes

In ZIO, we have two types of runtimes:

- **Top-level runtime** is the one that is used to run the entire ZIO application from the very beginning. There is only one top-level runtime when running a ZIO application. Here are some use-cases:
  - Creating a top level runtime in a mixed application. For example, if we are using an HTTP library that does not have direct support for ZIO we may need to use `Runtime.unsafe.run` in the implementations of each of our routes.
  - Another use-case is when we want to install a custom monitoring or supervisor from the very beginning of the application.

- **Locally scoped runtimes** are used during the execution of the ZIO application. They are local to a specific region of the code. Suppose we want to change the runtime configurations in the middle of a ZIO application. In such cases, we use locally scoped runtimes, for example:
  - When we want to import an effectful or side-effecting application with a specific runtime.
  - In some performance-critical regions, we want to disable logging temporarily.
  - When we want to have a customized executor for running a portion of our code.

ZLayer provides a consistent way to customize and configure runtimes. Using layers to customize the runtime enables us to use ZIO workflows. So a configuration workflow can be pure, effectful, or resourceful. Let's say we want to customize the runtime based on configuration information from a file or database.

In most cases, it is sufficient to customize application runtime using the [`bootstrap` layer](#configuring-runtime-using-bootstrap-layer) or [providing a custom configuration](#configuring-runtime-by-providing-configuration-layers) directly to our application. If none of these solutions fit to our problem, we can use [top-level runtime configurations](#top-level-runtime-configuration).

Let's talk about each solution in detail.

## Locally Scoped Runtime Configuration

In ZIO all runtime configurations are inherited from their parent workflows. So whenever we access a runtime configuration, or obtain a runtime inside a workflow, we are accessing the runtime of the parent workflow. We can override the runtime configuration of the parent workflow by providing a new configuration to a region of the code. This is called locally scoped runtime configuration. When the execution of that region is finished, the runtime configuration will be restored to its original value.

We mainly use `ZIO#provideXYZ` operators to provide a new runtime configuration to a specific region of the code:

### Configuring Runtime by Providing Configuration Layers

By providing (`ZIO#provideXYZ`) runtime configuration layers to a ZIO workflow, we can change the runtime configs easily:

```scala

object MainApp extends ZIOAppDefault {
  val addSimpleLogger: ZLayer[Any, Nothing, Unit] =
    Runtime.addLogger((_, _, _, message: () => Any, _, _, _, _) => println(message()))

  def run = {
    for {
      _ <- ZIO.log("Application started!")
      _ <- ZIO.log("Application is about to exit!")
    } yield ()
  }.provide(Runtime.removeDefaultLoggers ++ addSimpleLogger)
}
```

The output:

```scala
Application started!
Application is about to exit!
```

To provide runtime configuration to a specific region of a ZIO application, we should provide the configuration layer only to that specific region:

```scala

object MainApp extends ZIOAppDefault {
  val addSimpleLogger: ZLayer[Any, Nothing, Unit] =
    Runtime.addLogger((_, _, _, message: () => Any, _, _, _, _) => println(message()))

  def run =
    for {
      _ <- ZIO.log("Application started!")
      _ <- {
        for {
          _ <- ZIO.log("I'm not going to be logged!")
          _ <- ZIO.log("I will be logged by the simple logger.").provide(addSimpleLogger)
          _ <- ZIO.log("Reset back to the previous configuration, so I won't be logged.")
        } yield ()
      }.provide(Runtime.removeDefaultLoggers)
      _ <- ZIO.log("Application is about to exit!")
    } yield ()
}
```

The output:

```scala
timestamp=2022-08-31T14:28:34.711461Z level=INFO thread=#zio-fiber-6 message="Application started!" location=<empty>.MainApp.run file=ZIOApp.scala line=9
I will be logged by the simple logger.
timestamp=2022-08-31T14:28:34.832035Z level=INFO thread=#zio-fiber-6 message="Application is about to exit!" location=<empty>.MainApp.run file=ZIOApp.scala line=17
```

### Configuring Runtime Using `bootstrap` Layer

The `bootstrap` layer is a special layer that is mainly used to acquire and release services that are necessary for the application to run. However, it can also be applied to runtime customization as well. This solution requires us to override the `bootstrap` layer from the `ZIOApp` trait.

By using this technique, after initialization of the top-level runtime, it will provide the `bootstrap` layer to the ZIO application given through the `run` method.

```scala

object MainApp extends ZIOAppDefault {
  val addSimpleLogger: ZLayer[Any, Nothing, Unit] =
    Runtime.addLogger((_, _, _, message: () => Any, _, _, _, _) => println(message()))

  override val bootstrap: ZLayer[Any, Nothing, Unit] =
    Runtime.removeDefaultLoggers ++ addSimpleLogger

  def run =
    for {
      _ <- ZIO.log("Application started!")
      _ <- ZIO.log("Application is about to exit!")
    } yield ()
}
```

The output:

```scala
Application started!
Application is about to exit!
```

Although using this method will apply the configuration layer to the whole ZIO application, it is categorized as local runtime configuration because the `bootstrap` layer is evaluated and applied after the top-level runtime is initialized. So it will only be applied to the ZIO application given through the `run` method.

To elaborate more on this, let's look at the following example:

```scala

object MainApp extends ZIOAppDefault {
  val addSimpleLogger: ZLayer[Any, Nothing, Unit] =
    Runtime.addLogger((_, _, _, message: () => Any, _, _, _, _) => println(message()))
  
  val effectfulConfiguration: ZLayer[Any, Nothing, Unit] =
    ZLayer.fromZIO(ZIO.log("Started effectful workflow to customize runtime configuration"))

  override val bootstrap: ZLayer[Any, Nothing, Unit] =
    Runtime.removeDefaultLoggers ++ addSimpleLogger ++ effectfulConfiguration

  def run =
    for {
      _ <- ZIO.log("Application started!")
      _ <- ZIO.log("Application is about to exit!")
    } yield ()
}
```

What do we expect to see as the output? We have `Runtime.removeDefaultLoggers` which removes the default logger from the runtime. So we expect to see log messages only from the simple logger. But that is not the case. We have an effectful configuration layer that is evaluated after the top-level runtime is initialized. So we can see the log message related to the initialization of `effectfulConfiguration` layer from the default logger:

```scala
timestamp=2022-09-01T08:07:47.870219Z level=INFO thread=#zio-fiber-6 message="Started effectful workflow to customize runtime configuration" location=<empty>.MainApp.effectfulConfiguration file=ZIOApp.scala line=8
Application started!
Application is about to exit!
```

### Enabling Virtual Threads

It is possible to configure the Runtime to use Java virtual threads if you are using JDK 21 and later.
ZIO offers two ways to utilize virtual threads:

1. For the main executor (handles non-blocking ZIO operations):

```scala

object MainApp extends ZIOAppDefault {

  override val bootstrap = 
    Runtime.enableLoomBasedExecutor

  override def run = ZIO.attempt {
    println(s"Task running on a virtual-thread: ${Thread.currentThread().getName()}")
  }
}
```

2. For the blocking executor (handles blocking operations):

```scala

object MainApp extends ZIOAppDefault {

  override val bootstrap = 
    Runtime.enableLoomBasedBlockingExecutor

  override def run = ZIO.attemptBlocking {
    println(s"Blocking task running on a virtual-thread: ${Thread.currentThread().getName()}")
  }
}
```

## Top-level Runtime Configuration

When we write a ZIO application using the `ZIOAppDefault` trait, a default top-level runtime is created and used to run the application automatically under the hood. Further, we can customize the rest of the ZIO application by providing locally scoped configuration layers using [`provideXYZ` operations](#configuring-runtime-by-providing-configuration-layers) or [`bootstrap` layer](#configuring-runtime-using-bootstrap-layer).

This is usually sufficient for lots of ZIO applications, but it is not always the case. There are cases where we want to customize the runtime of the entire ZIO application from the top level.

In such cases, we need to create a top-level runtime by unsafely running the configuration layer to convert that configuration to the `Runtime` by using the `Runtime.unsafe.fromLayer` operator:

```scala
val runtime: Runtime[Any] =
  Unsafe.unsafe { implicit unsafe =>
    Runtime.unsafe.fromLayer(layer)
  }
```

Let's try a fully working example:

```scala

object MainApp extends ZIOAppDefault {

  // In a real-world application we might need to implement a `sl4jlogger` layer
  val addSimpleLogger: ZLayer[Any, Nothing, Unit] =
    Runtime.addLogger((_, _, _, message: () => Any, _, _, _, _) => println(message()))

  val layer: ZLayer[Any, Nothing, Unit] =
    Runtime.removeDefaultLoggers ++ addSimpleLogger

  override val runtime: Runtime[Any] =
    Unsafe.unsafe { implicit unsafe =>
      Runtime.unsafe.fromLayer(layer)
    }

  def run = ZIO.log("Application started!")
}
```

:::caution
Keep in mind that only the "bootstrap" layer of applications will be combined when we compose two ZIO applications. Therefore, when we compose two ZIO programs, top-level runtime configurations won't be integrated.
:::

Another use-case of top-level runtimes is when we want to integrate our ZIO application inside a legacy application:

```scala

object MainApp {
  val sl4jlogger: ZLogger[String, Any] = ???

  def legacyApplication(input: Int): Unit = ???

  val zioWorkflow: ZIO[Any, Nothing, Int] = ???

  val runtime: Runtime[Unit] =
    Unsafe.unsafe { implicit unsafe =>
      Runtime.unsafe
        .fromLayer(
          Runtime.removeDefaultLoggers ++ Runtime.addLogger(sl4jlogger)
        )
    }

  def zioApplication(): Int =
    Unsafe.unsafe { implicit unsafe =>
      runtime.unsafe
        .run(zioWorkflow)
        .getOrThrowFiberFailure()
    }

  def main(args: Array[String]): Unit = {
    val result = zioApplication()
    legacyApplication(result)
  }

}
```

## Providing Environment to Runtime System

The custom runtime can be used to run many different effects that all require the same environment, so we don't have to call `ZIO#provide` on all of them before we run them.

For example, assume we want to create a `Runtime` for services that are for testing purposes, and they don't interact with real external APIs. So we can create a runtime especially for testing.

Let's say we have defined two `LoggingService` and `EmailService` services:

```scala
trait LoggingService {
  def log(line: String): UIO[Unit]
}

object LoggingService {
  def log(line: String): URIO[LoggingService, Unit] =
    ZIO.serviceWithZIO[LoggingService](_.log(line))
}

trait EmailService {
  def send(user: String, content: String): Task[Unit]
}

object EmailService {
  def send(user: String, content: String): ZIO[EmailService, Throwable, Unit] =
    ZIO.serviceWithZIO[EmailService](_.send(user, content))
}
```

We are going to implement a live version of `LoggingService` and also a fake version of `EmailService` for testing:

```scala
case class LoggingServiceLive() extends LoggingService {
  override def log(line: String): UIO[Unit] =
    ZIO.succeed(print(line))
}

case class EmailServiceFake() extends EmailService {
  override def send(user: String, content: String): Task[Unit] =
    ZIO.attempt(println(s"sending email to $user"))
}
```

Let's create a custom runtime that contains these two service implementations in its environment:

```scala
val testableRuntime = Runtime(
  ZEnvironment[LoggingService, EmailService](LoggingServiceLive(), EmailServiceFake()),
  FiberRefs.empty,
  RuntimeFlags.default
)
```

Also, we can replace the environment of the default runtime with our own custom environment, which allows us to add new services to the ZIO environment:

```scala
val testableRuntime: Runtime[LoggingService with EmailService] =
  Runtime.default.withEnvironment {
    ZEnvironment[LoggingService, EmailService](LoggingServiceLive(), EmailServiceFake())
  }
```

Now we can run our effects using this custom `Runtime`:

```scala
Unsafe.unsafe { implicit unsafe =>
    testableRuntime.unsafe.run(
      for {
        _ <- LoggingService.log("sending newsletter")
        _ <- EmailService.send("David", "Hi! Here is today's newsletter.")
      } yield ()
    ).getOrThrowFiberFailure()
}
```

---

## IO

`IO[E, A]` is a type alias for `ZIO[Any, E, A]`, which represents an effect that has no requirements, and may fail with an `E`, or succeed with an `A`.

:::note
In Scala, the _type alias_ is a way to give a name to another type, to avoid having to repeat the original type again and again. It doesn't affect the type-checking process. It just helps us to have an expressive API design.
:::

Let's see how the `IO` type alias is defined:

```scala
type IO[+E, +A] = ZIO[Any, E, A]
```

So `IO` is equal to a `ZIO` that doesn't need any requirement.

`ZIO` values of type `IO[E, Nothing]` (where the value type is `Nothing`) are considered _unproductive_, because the `Nothing` type is _uninhabitable_, i.e. there can be no actual values of type `Nothing`. Values of this type may fail with an `E`, but will never produce a value.

:::note Principle of Least Power

The `ZIO` data type is the most powerful effect in the ZIO library. It helps us to model various types of workflows. On the other hand, the type aliases are a way of specializing the `ZIO` type for less powerful workflows. 

Often, we don't need such a piece of powerful machinery. So as a rule of thumb, whenever we require a less powerful effect, it's better to use the appropriate specialized type alias.

So there is no need to convert type aliases to the `ZIO` data type, and whenever the `ZIO` data type is required, we can use the most precise type alias to fit our workflow requirement.
:::

---

## RIO

`RIO[R, A]` is a type alias for `ZIO[R, Throwable, A]`, which represents an effect that requires an `R`, and may fail with a `Throwable` value, or succeed with an `A`.

:::note

In Scala, the _type alias_ is a way to give a name to another type, to avoid having to repeat the original type again and again. It doesn't affect the type-checking process. It just helps us to have an expressive API design.
:::

Let's see how `RIO` is defined:

```scala
type RIO[-R, +A]  = ZIO[R, Throwable, A]
```

So `RIO` is equal to a `ZIO` that requires `R`, and whose error channel is `Throwable`. It succeeds with `A`.

:::note _Principle of Least Power_

The `ZIO` data type is the most powerful effect in the ZIO library. It helps us to model various types of workflows. On the other hand, the type aliases are a way of specializing the `ZIO` type for less powerful workflows. 

Often, we don't need such a piece of powerful machinery. So as a rule of thumb, whenever we require a less powerful effect, it's better to use the appropriate specialized type alias.

So there is no need to convert type aliases to the `ZIO` data type, and whenever the `ZIO` data type is required, we can use the most precise type alias to fit our workflow requirement.
:::

---

## Task

`Task[A]` is a type alias for `ZIO[Any, Throwable, A]`, which represents an effect that has no requirements, and may fail with a `Throwable` value, or succeed with an `A`.

:::note

In Scala, a _type alias_ is a way to give a name to another type, to avoid having to repeat the original type again and again. It doesn't affect results of the type-checking process. It just helps us to have an expressive API design.
:::

Let's see how the `Task` type alias is defined:

```scala
type Task[+A] = ZIO[Any, Throwable, A]
```

So a `Task` is equivalent to a `ZIO` that doesn't need any requirement, and may fail with a `Throwable`, or succeed with an `A` value.

Sometimes we know that our effect may fail, but we don't care about the type of that exception. This is where we can use `Task`. The signature of this type alias is similar to `Future[T]` and Cats `IO`.

If we want to be less precise and eliminate the need to think about requirements and error types, we can use `Task`. This type alias is a good starting point for anyone who wants to refactor an existing code base which is written with Cats `IO` or Monix `Task`. 

:::note Principle of Least Power

The `ZIO` data type is the most powerful effect in the ZIO library. It helps us to model various types of workflows. On the other hand, the type aliases are a way of specializing the `ZIO` type for less powerful workflows. 

Often, we don't need such a piece of powerful machinery. So as a rule of thumb, whenever we require a less powerful effect, it's better to use the appropriate specialized type alias.

So there is no need to convert type aliases to the `ZIO` data type, and whenever the `ZIO` data type is required, we can use the most precise type alias to fit our workflow requirement.
:::

---

## UIO

`UIO[A]` is a type alias for `ZIO[Any, Nothing, A]`, which represents an **Unexceptional** effect that doesn't require any specific environment, and cannot fail, but can succeed with an `A`.

:::note

In Scala, the _type alias_ is a way to give a name to another type, to avoid having to repeat the original type again and again. It doesn't affect the type-checking process. It just helps us to have an expressive API design.
:::

Let's see how the `UIO` type alias is defined:

```scala
type UIO[+A] = ZIO[Any, Nothing, A]
```

So `UIO` is equal to a `ZIO` that doesn't need any requirement (because it accepts `Any` environment) and that cannot fail (because in Scala the `Nothing` type is _uninhabitable_, i.e. there can be no actual value of type `Nothing`). It succeeds with `A`.

`ZIO` values of type `UIO[A]` are considered _infallible_. Values of this type may produce an `A`, but will never fail.

Let's write a Fibonacci function. In the following example, the `fib` function is an unexceptional effect, since it has no requirements, we don't expect any failure, and it succeeds with a value of type `Int`:

```scala

def fib(n: Int): UIO[Int] =
  if (n <= 1) {
    ZIO.succeed(1)
  } else {
    for {
      fiber1 <- fib(n - 2).fork
      fiber2 <- fib(n - 1).fork
      v2     <- fiber2.join
      v1     <- fiber1.join
    } yield v1 + v2
  }
```

:::note Principle of Least Power

The `ZIO` data type is the most powerful effect in the ZIO library. It helps us to model various types of workflows. On the other hand, the type aliases are a way of specializing the `ZIO` type for less powerful workflows. 

Often, we don't need such a piece of powerful machinery. So as a rule of thumb, whenever we require a less powerful effect, it's better to use the appropriate specialized type alias.

So there is no need to convert type aliases to the `ZIO` data type, and whenever the `ZIO` data type is required, we can use the most precise type alias to fit our workflow requirement.
:::

---

## URIO

`URIO[R, A]` is a type alias for `ZIO[R, Nothing, A]`, which represents an effect that requires an `R`, and cannot fail, but can succeed with an `A`.

:::note

In Scala, the _type alias_ is a way to give a name to another type, to avoid having to repeat the original type again and again. It doesn't affect the type-checking process. It just helps us to have an expressive API design.
:::

Let's see how the `URIO` type alias is defined:

```scala
type URIO[-R, +A] = ZIO[R, Nothing, A]
```

So `URIO` is equal to a `ZIO` that requires `R` and cannot fail (because in Scala the `Nothing` type has no inhabitant, so we can't create an instance of type `Nothing`). It succeeds with `A`.

:::note Principle of Least Power

The `ZIO` data type is the most powerful effect in the ZIO library. It helps us to model various types of workflows. On the other hand, the type aliases are a way of specializing the `ZIO` type for less powerful workflows. 

Often, we don't need such a piece of powerful machinery. So as a rule of thumb, whenever we require a less powerful effect, it's better to use the appropriate specialized type alias.

So there is no need to convert type aliases to the `ZIO` data type, and whenever the `ZIO` data type is required, we can use the most precise type alias to fit our workflow requirement.
:::

---

## ZIO

A `ZIO[R, E, A]` value is an immutable value that lazily describes a workflow or job. The workflow requires some environment `R`, and may fail with an error of type `E`, or succeed with a value of type `A`.

A value of type `ZIO[R, E, A]` is like an effectful version of the following function type:

```scala
R => Either[E, A]
```

This function, which requires an `R`, might produce either an `E`, representing failure, or an `A`, representing success. ZIO effects are not actually functions, of course, they can model synchronous, asynchronous, concurrent, parallel, and resourceful computations.

ZIO effects use a fiber-based concurrency model, with built-in support for
scheduling, fine-grained interruption, structured concurrency, and high scalability.

The `ZIO[R, E, A]` data type has three type parameters:

- **`R` - Environment Type**. The effect requires an environment of type `R`. If this type parameter is `Any`, it means the effect has no requirements, because we can run the effect with any value (for example, the unit value `()`).
- **`E` - Failure Type**. The effect may fail with a value of type `E`. Some applications will use `Throwable`. If this type parameter is `Nothing`, it means the effect cannot fail, because there are no values of type `Nothing`.
- **`A` - Success Type**. The effect may succeed with a value of type `A`. If this type parameter is `Unit`, it means the effect produces no useful information, while if it is `Nothing`, it means the effect runs forever (or until failure).

In the following example, the `readLine` function does not require any services, it may fail with value of type `IOException`, or may succeed with a value of type `String`:

```scala

val readLine: ZIO[Any, IOException, String] =
  Console.readLine
```

`ZIO` values are immutable, and all `ZIO` functions produce new `ZIO` values, enabling `ZIO` to be reasoned about and used like any ordinary Scala immutable data structure.

`ZIO` values do not actually _do_ anything; they are just values that _model_ or _describe_ effectful interactions.

`ZIO` can be _interpreted_ by the ZIO runtime system into effectful interactions with the external world. Ideally, this occurs at a single time, in our application's `main` function. The `App` class provides this functionality automatically.

## Creation

In this section we explore some of the common ways to create ZIO effects from values, from common Scala types, and from both synchronous and asynchronous side-effects. Here is the summary list of them:

### Success Values

Using the `ZIO.succeed` method, we can create an effect that succeeds with the specified value:

```scala

val s1 = ZIO.succeed(42)
```

### Failure Values

Using the `ZIO.fail` method, we can create an effect that models failure:

```scala

val f1 = ZIO.fail("Uh oh!")
```

For the `ZIO` data type, there is no restriction on the error type. We may use strings, exceptions, or custom data types appropriate for our application.

Many applications will model failures with classes that extend `Throwable` or `Exception`:

```scala

val f2 = ZIO.fail(new Exception("Uh oh!"))
```

### From Values

ZIO contains several constructors which help us to convert various data types into `ZIO` effects.

#### Option

1. **`ZIO.fromOption`**— An `Option` can be converted into a ZIO effect using `ZIO.fromOption`:

```scala

val zoption: IO[Option[Nothing], Int] = ZIO.fromOption(Some(2))
```

The error type of the resulting effect is `Option[Nothing]`, which provides no information on why the value is not there. We can change the `Option[Nothing]` into a more specific error type using `ZIO#mapError`:

```scala

val zoption2: IO[String, Int] = zoption.mapError(_ => "It wasn't there!")
```

We can also readily compose it with other operators while preserving the optional nature of the result (similar to an `OptionT`):

```scala

val maybeId: IO[Option[Nothing], String] = ZIO.fromOption(Some("abc123"))
def getUser(userId: String): IO[Throwable, Option[User]] = ???
def getTeam(teamId: String): IO[Throwable, Team] = ???

val result: IO[Throwable, Option[(User, Team)]] = (for {
  id   <- maybeId
  user <- getUser(id).some
  team <- getTeam(user.teamId).asSomeError
} yield (user, team)).unsome
```

2. **`ZIO.some`**/**`ZIO.none`**— These constructors can be used to directly create ZIO of optional values:

```scala

val someInt: ZIO[Any, Nothing, Option[Int]]     = ZIO.some(3)
val noneInt: ZIO[Any, Nothing, Option[Nothing]] = ZIO.none
```

3. **`ZIO.getOrFail`**— We can lift an `Option` into a `ZIO` and if the option is not defined we can fail the ZIO with the proper error type:

- `ZIO.getOrFail` fails with `Throwable` error type.
- `ZIO.getOrFailUnit` fails with `Unit` error type.
- `ZIO.getOrFailWith` fails with custom error type.

```scala

def parseInt(input: String): Option[Int] = input.toIntOption

// If the optional value is not defined it fails with Throwable error type:
val r1: ZIO[Any, Throwable, Int] =
  ZIO.getOrFail(parseInt("1.2"))

// If the optional value is not defined it fails with Unit error type:
val r2: ZIO[Any, Unit, Int] =
  ZIO.getOrFailUnit(parseInt("1.2"))

// If the optional value is not defined it fail with given error type:
val r3: ZIO[Any, NumberFormatException, Int] =
  ZIO.getOrFailWith(new NumberFormatException("invalid input"))(parseInt("1.2"))
```

4. **`ZIO.noneOrFail`**— It lifts an option into a ZIO value. If the option is empty it succeeds with `Unit` and if the option is defined it fails with a proper error type:

- `ZIO.noneOrFail` fails with the content of the optional value.
- `ZIO.noneOrFailUnit` fails with the `Unit` error type.
- `ZIO.noneOrFailWith` fails with custom error type.

```scala

val optionalValue: Option[String] = ???

// If the optional value is empty it succeeds with Unit
// If the optional value is defined it will fail with the content of the optional value
val r1: ZIO[Any, String, Unit] =
  ZIO.noneOrFail(optionalValue)

// If the optional value is empty it succeeds with Unit
// If the optional value is defined, it will fail by applying the error function to it:
val r2: ZIO[Any, NumberFormatException, Unit] =
  ZIO.noneOrFailWith(optionalValue)(e => new NumberFormatException(e))
```

#### Either

| Function     | Input Type     | Output Type               |
|--------------|----------------|---------------------------|
| `fromEither` | `Either[E, A]` | `IO[E, A]`                |
| `left`       | `A`            | `UIO[Either[A, Nothing]]` |
| `right`      | `A`            | `UIO[Either[Nothing, A]]` |

An `Either` can be converted into a ZIO effect using `ZIO.fromEither`:

```scala

val zeither = ZIO.fromEither(Right("Success!"))
```

The error type of the resulting effect will be whatever type the `Left` case has, while the success type will be whatever type the `Right` case has.

#### Try

| Function  | Input Type          | Output Type |
|-----------|---------------------|-------------|
| `fromTry` | `scala.util.Try[A]` | `Task[A]`   |

A `Try` value can be converted into a ZIO effect using `ZIO.fromTry`:

```scala

val ztry = ZIO.fromTry(Try(42 / 0))
```

The error type of the resulting effect will always be `Throwable`, because `Try` can only fail with values of type `Throwable`.

#### Future

| Function              | Input Type                                       | Output Type        |
|-----------------------|--------------------------------------------------|--------------------|
| `fromFuture`          | `ExecutionContext => scala.concurrent.Future[A]` | `Task[A]`          |
| `fromFutureJava`      | `java.util.concurrent.Future[A]`                 | `RIO[Blocking, A]` |
| `fromFunctionFuture`  | `R => scala.concurrent.Future[A]`                | `RIO[R, A]`        |
| `fromFutureInterrupt` | `ExecutionContext => scala.concurrent.Future[A]` | `Task[A]`          |

A `Future` can be converted into a ZIO effect using `ZIO.fromFuture`:

```scala

lazy val future = Future.successful("Hello!")

val zfuture: Task[String] =
  ZIO.fromFuture { implicit ec =>
    future.map(_ => "Goodbye!")
  }
```

The function passed to `fromFuture` is passed an `ExecutionContext`, which allows ZIO to manage where the `Future` runs (of course, we can ignore this `ExecutionContext`).

The error type of the resulting effect will always be `Throwable`, because `Future` can only fail with values of type `Throwable`.

#### Promise

| Function           | Input Type                    | Output Type |
|--------------------|-------------------------------|-------------|
| `fromPromiseScala` | `scala.concurrent.Promise[A]` | `Task[A]`   |

A `Promise` can be converted into a ZIO effect using `ZIO.fromPromiseScala`:

```scala

val func: String => String = s => s.toUpperCase
for {
  promise <- ZIO.succeed(scala.concurrent.Promise[String]())
  _ <- ZIO.attempt {
    Try(func("hello world from future")) match {
      case Success(value) => promise.success(value)
      case Failure(exception) => promise.failure(exception)
    }
  }.fork
  value <- ZIO.fromPromiseScala(promise)
  _ <- Console.printLine(s"Hello World in UpperCase: $value")
} yield ()
```

#### Fiber

| Function       | Input Type           | Output Type |
|----------------|----------------------|-------------|
| `fromFiber`    | `Fiber[E, A]`        | `IO[E, A]`  |
| `fromFiberZIO` | `IO[E, Fiber[E, A]]` | `IO[E, A]`  |

A `Fiber` can be converted into a ZIO effect using `ZIO.fromFiber`:

```scala

val io: IO[Nothing, String] = ZIO.fromFiber(Fiber.succeed("Hello from Fiber!"))
```

### From Side-Effects

ZIO can convert both synchronous and asynchronous side-effects into ZIO effects (pure values).

These functions can be used to wrap procedural code, allowing us to seamlessly use all features of ZIO with legacy Scala and Java code, as well as third-party libraries.

#### Synchronous

| Function  | Input Type | Output Type | Note                                        |
|-----------|------------|-------------|---------------------------------------------|
| `succeed` | `A`        | `UIO[A]`    | Imports a total synchronous effect          |
| `attempt` | `A`        | Task[A]     | Imports a (partial) synchronous side-effect |

A synchronous side-effect can be converted into a ZIO effect using `ZIO.attempt`:

```scala

val getLine: Task[String] =
  ZIO.attempt(StdIn.readLine())
```

The error type of the resulting effect will always be `Throwable`, because side-effects may throw exceptions with any value of type `Throwable`.

If a given side-effect is known to not throw any exceptions, then the side-effect can be converted into a ZIO effect using `ZIO.succeed`:

```scala

def printLine(line: String): UIO[Unit] =
  ZIO.succeed(println(line))

val succeedTask: UIO[Long] =
  ZIO.succeed(java.lang.System.nanoTime())
```

We should be careful when using `ZIO.succeed`—when in doubt about whether or not a side-effect is total, prefer `ZIO.attempt` to convert the effect.

If this is too broad, the `refineOrDie` method of `ZIO` may be used to retain only certain types of exceptions, and to die on any other types of exceptions:

```scala

val printLine2: IO[IOException, String] =
  ZIO.attempt(scala.io.StdIn.readLine()).refineToOrDie[IOException]
```

##### Blocking Synchronous Side-Effects

| Function                    | Input Type                          | Output Type                     |
|-----------------------------|-------------------------------------|---------------------------------|
| `blocking`                  | `ZIO[R, E, A]`                      | `ZIO[R, E, A]`                  |
| `attemptBlocking`           | `A`                                 | `RIO[Blocking, A]`              |
| `attemptBlockingCancelable` | `effect: => A`, `cancel: UIO[Unit]` | `RIO[Blocking, A]`              |
| `attemptBlockingInterrupt`  | `A`                                 | `RIO[Blocking, A]`              |
| `attemptBlockingIO`         | `A`                                 | `ZIO[Blocking, IOException, A]` |

By default, ZIO is asynchronous and all effects will be executed on a default primary thread pool which is optimized for asynchronous operations. As ZIO uses a fiber-based concurrency model, if we run **Blocking I/O** or **CPU Work** workloads on a primary thread pool, they are going to monopolize all threads of **primary thread pool**.

ZIO has a separate **blocking thread pool** specially designed for **Blocking I/O** and, also **CPU Work** workloads. We should run blocking workloads on this thread pool by using `ZIO.blocking` or `ZIO.attemptBlocking*` constructors to prevent interfering with the primary thread pool.

:::note
ZIO has an auto-blocking mechanism that detects blocking operations and runs them on a separate blocking thread pool. However, if you know that some code is blocking you can use the `ZIO.blocking` constructor to give a "hint" of this to the ZIO runtime.
:::

The `blocking` operator takes a ZIO effect and returns another effect that is going to run on a blocking thread pool:

```scala
val program = ZIO.foreachPar((1 to 100).toArray)(t => ZIO.blocking(blockingTask(t)))
```

A blocking side-effect can be converted directly into a ZIO effect using the `attemptBlocking` method:

```scala

def blockingTask(n: Int) = ZIO.attemptBlocking {
  do {
    println(s"Running blocking task number $n on dedicated blocking thread pool")
    Thread.sleep(3000)
  } while (true)
}
```

The resulting effect will be executed on a separate thread pool designed specifically for blocking effects.

Blocking side-effects can be interrupted by invoking `Thread.interrupt` using the `attemptBlockingInterrupt` method.

Some blocking side-effects can only be interrupted by invoking a cancellation effect. We can convert these side-effects using the `attemptBlockingCancelable` method:

```scala

def accept(l: ServerSocket) =
  ZIO.attemptBlockingCancelable(l.accept())(ZIO.succeed(l.close()))
```

If a side-effect has already been converted into a ZIO effect, then instead of `attemptBlocking`, the `blocking` method can be used to ensure the effect will be executed on the blocking thread pool:

```scala

def download(url: String) =
  ZIO.attempt {
    Source.fromURL(url)(Codec.UTF8).mkString
  }

def safeDownload(url: String) =
  ZIO.blocking(download(url))
```

#### Asynchronous

| Function         | Input Type                                                     | Output Type    |
|------------------|----------------------------------------------------------------|----------------|
| `async`          | `(ZIO[R, E, A] => Unit) => Any`                                | `ZIO[R, E, A]` |
| `asyncZIO`       | `(ZIO[R, E, A] => Unit) => ZIO[R, E, Any]`                     | `ZIO[R, E, A]` |
| `asyncMaybe`     | `(ZIO[R, E, A] => Unit) => Option[ZIO[R, E, A]]`               | `ZIO[R, E, A]` |
| `asyncInterrupt` | `(ZIO[R, E, A] => Unit) => Either[URIO[R, Any], ZIO[R, E, A]]` | `ZIO[R, E, A]` |

An asynchronous side-effect with a callback-based API can be converted into a ZIO effect using `ZIO.async`:

```scala

object legacy {
  def login(
    onSuccess: User => Unit,
    onFailure: AuthError => Unit): Unit = ???
}

val login: IO[AuthError, User] =
  ZIO.async[Any, AuthError, User] { callback =>
    legacy.login(
      user => callback(ZIO.succeed(user)),
      err  => callback(ZIO.fail(err))
    )
  }
```

Asynchronous ZIO effects are much easier to use than callback-based APIs, and they benefit from ZIO features like interruption, resource-safety, and superior error handling.

### Creating Suspended Effects

| Function         | Input Type     | Output Type    |
|------------------|----------------|----------------|
| `suspend`        | `RIO[R, A]`    | `RIO[R, A]`    |
| `suspendSucceed` | `ZIO[R, E, A]` | `ZIO[R, E, A]` |

A `RIO[R, A]` effect can be suspended using `suspend` function:

```scala

val suspendedEffect: RIO[Any, ZIO[Any, IOException, Unit]] =
  ZIO.suspend(ZIO.attempt(Console.printLine("Suspended Hello World!")))
```

## Mapping

### map

We can change an `IO[E, A]` to an `IO[E, B]` by calling the `map` method with a function `A => B`. This lets us transform values produced by actions into other values.

```scala

val mappedValue: UIO[Int] = ZIO.succeed(21).map(_ * 2)
```

## Tapping

Using `ZIO.tap` we can peek into a success value and perform any effectful operation, without changing the returning value of the original effect:

```scala
trait ZIO[-R, +E, +A] {
  def tap[R1 <: R, E1 >: E](f: A => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
  def tapSome[R1 <: R, E1 >: E](f: PartialFunction[A, ZIO[R1, E1, Any]]): ZIO[R1, E1, A]
}
```

```scala

object MainApp extends ZIOAppDefault {
  def isPrime(n: Int): Boolean =
    if (n <= 1) false else (2 until n).forall(i => n % i != 0)

  val myApp: ZIO[Any, IOException, Unit] =
    for {
      ref <- Ref.make(List.empty[Int])
      prime <-
        Random
          .nextIntBetween(0, Int.MaxValue)
          .tap(random => ref.update(_ :+ random))
          .repeatUntil(isPrime)
      _ <- Console.printLine(s"found a prime number: $prime")
      tested <- ref.get
      _ <- Console.printLine(
        s"list of tested numbers: ${tested.mkString(", ")}"
      )
    } yield ()

  def run = myApp
}
```

## Chaining

We can execute two actions in sequence with the `flatMap` method. The second action may depend on the value produced by the first action.

```scala

val chainedActionsValue: UIO[List[Int]] = ZIO.succeed(List(1, 2, 3)).flatMap { list =>
  ZIO.succeed(list.map(_ + 1))
}
```

If the first effect fails, the callback passed to `flatMap` will never be invoked, and the composed effect returned by `flatMap` will also fail.

In _any_ chain of effects, the first failure will short-circuit the whole chain, just like throwing an exception will prematurely exit a sequence of statements.

Because the `ZIO` data type supports both `flatMap` and `map`, we can use Scala's _for comprehensions_ to build sequential effects:

```scala

val program =
  for {
    _    <- Console.printLine("Hello! What is your name?")
    name <- Console.readLine
    _    <- Console.printLine(s"Hello, ${name}, welcome to ZIO!")
  } yield ()
```

_For comprehensions_ provide a more procedural syntax for composing chains of effects.

## Zipping

We can combine two effects into a single effect with the `zip` method. The resulting effect succeeds with a tuple that contains the success values of both effects:

```scala

val zipped: UIO[(String, Int)] =
  ZIO.succeed("4").zip(ZIO.succeed(2))
```

Note that `zip` operates sequentially: the effect on the left side is executed before the effect on the right side.

In any `zip` operation, if either the left or right-hand sides fail, then the composed effect will fail, because _both_ values are required to construct the tuple.

### zipLeft and zipRight

Sometimes, when the success value of an effect is not useful (for example, it is `Unit`), it can be more convenient to use the `zipLeft` or `zipRight` functions, which first perform a `zip`, and then map over the tuple to discard one side or the other:

```scala

val zipRight1 =
  Console.printLine("What is your name?").zipRight(Console.readLine)
```

The `zipRight` and `zipLeft` functions have symbolic aliases, known as `*>` and `<*`, respectively. Some developers find these operators easier to read:

```scala

val zipRight2 =
  Console.printLine("What is your name?") *>
  Console.readLine
```

## Parallelism

ZIO provides many operations for performing effects in parallel. These methods are all named with a `Par` suffix that helps us identify opportunities to parallelize our code.

For example, the ordinary `ZIO#zip` method zips two effects together, sequentially. But there is also a `ZIO#zipPar` method, which zips two effects together in parallel.

The following table summarizes some of the sequential operations and their corresponding parallel versions:

|              **Description** |  **Sequential**  |    **Parallel**     |
| ---------------------------: | :--------------: | :-----------------: |
|     Zip two effects into one |    `ZIO#zip`     |    `ZIO#zipPar`     |
|     Zip two effects into one |  `ZIO#zipWith`   |  `ZIO#zipWithPar`   |
|    Collect from many effects | `ZIO.collectAll` | `ZIO.collectAllPar` |
| Effectfully loop over values |  `ZIO.foreach`   |  `ZIO.foreachPar`   |
|           Reduce many values | `ZIO.reduceAll`  | `ZIO.reduceAllPar`  |
|            Merge many values |  `ZIO.mergeAll`  |  `ZIO.mergeAllPar`  |

For all the parallel operations, if one effect fails, then others will be interrupted, to minimize unnecessary computation.

If the fail-fast behavior is not desired, potentially failing effects can be first converted into infallible effects using the `ZIO#either` or `ZIO#option` methods.

### Racing

ZIO lets us race multiple effects in parallel, returning the first successful result:

```scala

for {
  winner <- ZIO.succeed("Hello").race(ZIO.succeed("Goodbye"))
} yield winner
```

If we want the first success or failure, rather than the first success, then we can use `left.either race right.either`, for any effects `left` and `right`.

## Timeout

ZIO lets us timeout any effect using the `ZIO#timeout` method, which returns a new effect that succeeds with an `Option`. A value of `None` indicates the timeout elapsed before the effect completed.

```scala

ZIO.succeed("Hello").timeout(10.seconds)
```

If an effect times out, then instead of continuing to execute in the background, it will be interrupted so no resources will be wasted.

## Error Management

### Either

| Function      | Input Type                | Output Type             |
| ------------- | ------------------------- | ----------------------- |
| `ZIO#either`  | `ZIO[R, E, A]`            | `URIO[R, Either[E, A]]` |
| `ZIO.absolve` | `ZIO[R, E, Either[E, A]]` | `ZIO[R, E, A]`          |

We can surface failures with `ZIO#either`, which takes a `ZIO[R, E, A]` and produces a `ZIO[R, Nothing, Either[E, A]]`.

```scala
val zeither: UIO[Either[String, Int]] =
  ZIO.fail("Uh oh!").either
```

We can submerge failures with `ZIO.absolve`, which is the opposite of `either` and turns a `ZIO[R, Nothing, Either[E, A]]` into a `ZIO[R, E, A]`:

```scala
def sqrt(io: UIO[Double]): IO[String, Double] =
  ZIO.absolve(
    io.map(value =>
      if (value < 0.0) Left("Value must be >= 0.0")
      else Right(Math.sqrt(value))
    )
  )
```

### Catching

| Function              | Input Type                                                  | Output Type       |
| --------------------- | ----------------------------------------------------------- | ----------------- |
| `ZIO#catchAll`        | `E => ZIO[R1, E2, A1]`                                      | `ZIO[R1, E2, A1]` |
| `ZIO#catchAllCause`   | `Cause[E] => ZIO[R1, E2, A1]`                               | `ZIO[R1, E2, A1]` |
| `ZIO#catchAllDefect`  | `Throwable => ZIO[R1, E1, A1]`                              | `ZIO[R1, E1, A1]` |
| `ZIO#catchAllTrace`   | `((E, Option[StackTrace])) => ZIO[R1, E2, A1]`              | `ZIO[R1, E2, A1]` |
| `ZIO#catchSome`       | `PartialFunction[E, ZIO[R1, E1, A1]]`                       | `ZIO[R1, E1, A1]` |
| `ZIO#catchSomeCause`  | `PartialFunction[Cause[E], ZIO[R1, E1, A1]]`                | `ZIO[R1, E1, A1]` |
| `ZIO#catchSomeDefect` | `PartialFunction[Throwable, ZIO[R1, E1, A1]]`               | `ZIO[R1, E1, A1]` |
| `ZIO#catchSomeTrace`  | `PartialFunction[(E, Option[StackTrace]), ZIO[R1, E1, A1]]` | `ZIO[R1, E1, A1]` |

#### Catching All Errors

If we want to catch and recover from all types of errors and effectfully attempt recovery, we can use the `catchAll` method:

```scala
val z: IO[IOException, Array[Byte]] =
  readFile("primary.json").catchAll(_ =>
    readFile("backup.json"))
```

In the callback passed to `catchAll`, we may return an effect with a different error type (or perhaps `Nothing`), which will be reflected in the type of effect returned by `catchAll`.

#### Catching Some Errors

If we want to catch and recover from only some types of exceptions and effectfully attempt recovery, we can use the `catchSome` method:

```scala
val data: IO[IOException, Array[Byte]] =
  readFile("primary.data").catchSome {
    case _ : FileNotFoundException =>
      readFile("backup.data")
  }
```

Unlike `catchAll`, `catchSome` cannot reduce or eliminate the error type, although it can widen the error type to a broader class of errors.

### Fallback

| Function         | Input Type                | Output Type                 |
| ---------------- | ------------------------- | --------------------------- |
| `orElse`         | `ZIO[R1, E2, A1]`         | `ZIO[R1, E2, A1]`           |
| `orElseEither`   | `ZIO[R1, E2, B]`          | `ZIO[R1, E2, Either[A, B]]` |
| `orElseFail`     | `E1`                      | `ZIO[R, E1, A]`             |
| `orElseOptional` | `ZIO[R1, Option[E1], A1]` | `ZIO[R1, Option[E1], A1]`   |
| `orElseSucceed`  | `A1`                      | `URIO[R, A1]`               |

We can try one effect, or, if it fails, try another effect, with the `orElse` combinator:

```scala
val primaryOrBackupData: IO[IOException, Array[Byte]] =
  readFile("primary.data").orElse(readFile("backup.data"))
```

### Folding

| Function       | Input Type                                                                           | Output Type      |
| -------------- | ------------------------------------------------------------------------------------ | ---------------- |
| `fold`         | `failure: E => B, success: A => B`                                                   | `URIO[R, B]`     |
| `foldCause`    | `failure: Cause[E] => B, success: A => B`                                            | `URIO[R, B]`     |
| `foldZIO`      | `failure: E => ZIO[R1, E2, B], success: A => ZIO[R1, E2, B]`                         | `ZIO[R1, E2, B]` |
| `foldCauseZIO` | `failure: Cause[E] => ZIO[R1, E2, B], success: A => ZIO[R1, E2, B]`                  | `ZIO[R1, E2, B]` |
| `foldTraceZIO` | `failure: ((E, Option[StackTrace])) => ZIO[R1, E2, B], success: A => ZIO[R1, E2, B]` | `ZIO[R1, E2, B]` |

Scala's `Option` and `Either` data types have `fold`, which lets us handle both failure and success at the same time. In a similar fashion, `ZIO` effects also have several methods that allow us to handle both failure and success.

The first fold method, `fold`, lets us non-effectfully handle both failure and success by supplying a non-effectful handler for each case:

```scala
lazy val DefaultData: Array[Byte] = Array(0, 0)

val primaryOrDefaultData: UIO[Array[Byte]] =
  readFile("primary.data").fold(
    _    => DefaultData,
    data => data)
```

The second fold method, `foldZIO`, lets us effectfully handle both failure and success by supplying an effectful (but still pure) handler for each case:

```scala
val primaryOrSecondaryData: IO[IOException, Array[Byte]] =
  readFile("primary.data").foldZIO(
    _    => readFile("secondary.data"),
    data => ZIO.succeed(data))
```

Nearly all error handling methods are defined in terms of `foldZIO`, because it is both powerful and fast.

In the following example, `foldZIO` is used to handle both failure and success of the `readUrls` method:

```scala
val urls: UIO[Content] =
  readUrls("urls.json").foldZIO(
    error   => ZIO.succeed(NoContent(error)),
    success => fetchContent(success)
  )
```

### Retrying

| Function            | Input Type                                                           | Output Type                 |
| ------------------- | -------------------------------------------------------------------- | --------------------------- |
| `retry`             | `Schedule[R1, E, S]`                                                 | `ZIO[R1, E, A]`             |
| `retryN`            | `n: Int`                                                             | `ZIO[R, E, A]`              |
| `retryOrElse`       | `policy: Schedule[R1, E, S], orElse: (E, S) => ZIO[R1, E1, A1]`      | `ZIO[R1, E1, A1]`           |
| `retryOrElseEither` | `schedule: Schedule[R1, E, Out], orElse: (E, Out) => ZIO[R1, E1, B]` | `ZIO[R1, E1, Either[B, A]]` |
| `retryUntil`        | `E => Boolean`                                                       | `ZIO[R, E, A]`              |
| `retryUntilEquals`  | `E1`                                                                 | `ZIO[R, E1, A]`             |
| `retryUntilZIO`     | `E => URIO[R1, Boolean]`                                             | `ZIO[R1, E, A]`             |
| `retryWhile`        | `E => Boolean`                                                       | `ZIO[R, E, A]`              |
| `retryWhileEquals`  | `E1`                                                                 | `ZIO[R, E1, A]`             |
| `retryWhileZIO`     | `E => URIO[R1, Boolean]`                                             | `ZIO[R1, E, A]`             |

When we are building applications we want to be resilient in the face of a transient failure. This is where we need to retry to overcome these failures.

There are a number of useful methods on the ZIO data type for retrying failed effects.

The most basic of these is `ZIO#retry`, which takes a `Schedule` and returns a new effect that will retry the first effect if it fails according to the specified policy:

```scala
val retriedOpenFile: ZIO[Any, IOException, Array[Byte]] =
  readFile("primary.data").retry(Schedule.recurs(5))
```

The next most powerful function is `ZIO#retryOrElse`, which allows specification of a fallback to use if the effect does not succeed with the specified policy:

```scala
readFile("primary.data").retryOrElse(
  Schedule.recurs(5),
  (_, _:Long) => ZIO.succeed(DefaultData)
)
```

The final method, `ZIO#retryOrElseEither`, allows returning a different type for the fallback.

## Resource Management

ZIO's resource management features work across synchronous, asynchronous, concurrent, and other effect types, and provide strong guarantees even in the presence of failure, interruption, or defects in the application.

### Finalizing

Scala has a `try` / `finally` construct which helps us to make sure we don't leak resources because no matter what happens in the `try`, the `finally` block will be executed. So we can open files in the `try` block, and then we can close them in the `finally` block, and that gives us the guarantee that we will not leak resources.

#### Asynchronous Try / Finally

The problem with the `try` / `finally` construct is that it only applies to synchronous code, meaning it doesn't work for asynchronous code. ZIO gives us a method called `ensuring` that works with either synchronous or asynchronous actions. So we have a functional `try` / `finally` even for asynchronous regions of our code.

Like `try` / `finally`, the `ensuring` operation guarantees that if an effect begins executing and then terminates (for whatever reason), then the finalizer will begin executing:

```scala

val finalizer =
  ZIO.succeed(println("Finalizing!"))

val finalized: IO[String, Unit] =
  ZIO.fail("Failed!").ensuring(finalizer)
```

The finalizer is not allowed to fail, which means that it must handle any errors internally.

Like `try` / `finally`, finalizers can be nested, and the failure of any inner finalizer will not affect outer finalizers. Nested finalizers will be executed in reverse order, and linearly (not in parallel).

Unlike `try` / `finally`, `ensuring` works across all types of effects, including asynchronous and concurrent effects.

Here is another example of ensuring that our clean-up action is called before our effect is done:

```scala

var i: Int = 0
val action: Task[String] =
  ZIO.succeed(i += 1) *>
    ZIO.fail(new Throwable("Boom!"))
val cleanupAction: UIO[Unit] = ZIO.succeed(i -= 1)
val composite = action.ensuring(cleanupAction)
```

:::caution
Finalizers offer very powerful guarantees, but they are low-level, and should generally not be used for releasing resources. For higher-level logic built on `ensuring`, see `ZIO#acquireReleaseWith` in the acquire release section.
:::

#### Unstoppable Finalizers

In Scala when we nest `try` / `finally` finalizers, they cannot be stopped. If we have nested finalizers and one of them fails for some sort of catastrophic reason the ones on the outside will still be run and in the correct order.

```scala
try {
  try {
    try {
      ...
    } finally f1
  } finally f2
} finally f3
```

Also in ZIO like `try` / `finally`, the finalizers are unstoppable. This means if we have a buggy finalizer that is going to leak some resources, we will leak the minimum amount of resources because all other finalizers will still be run in the correct order.

```scala
val io = ???
io.ensuring(f1)
 .ensuring(f2)
 .ensuring(f3)
```

### Acquire Release

In Scala `try` / `finally` is often used to manage resources. A common use for `try` / `finally` is safely acquiring and releasing resources, such as new socket connections or opened files:

```scala
val handle = openFile(name)

try {
  processFile(handle)
} finally closeFile(handle)
```

ZIO encapsulates this common pattern with `ZIO#acquireRelease`, which allows us to specify an _acquire_ effect, which acquires a resource; a _release_ effect, which releases it; and a _use_ effect, which uses the resource. Acquire release lets us open a file and close the file and no matter what happens when we are using that resource.

The release action is guaranteed to be executed by the runtime system, even if the utilize action throws an exception or the executing fiber is interrupted.

Acquire release is a built-in primitive that let us safely acquire and release resources. It is used for a similar purpose as `try` / `catch` / `finally`, only acquire release work with synchronous and asynchronous actions, work seamlessly with fiber interruption, and is built on a different error model that ensures no errors are ever swallowed.

Acquire release consist of an _acquire_ action, a _utilize_ action (which uses the acquired resource), and a _release_ action.

```scala

val groupedFileData: IO[IOException, Unit] = ZIO.acquireReleaseWith(openFile("data.json"))(closeFile(_)) { file =>
  for {
    data    <- decodeData(file)
    grouped <- groupData(data)
  } yield grouped
}
```

Acquire releases have compositional semantics, so if an acquire release is nested inside another acquire release, and the outer resource is acquired, then the outer release will always be called, even if, for example, the inner release fails.

Let's look at a full working example on using acquire release:

```scala

object Main extends ZIOAppDefault {

  // run my acquire release
  def run = myAcquireRelease

  def closeStream(is: FileInputStream) =
    ZIO.succeed(is.close())

  def convertBytes(is: FileInputStream, len: Long) =
    ZIO.attempt {
      val buffer = new Array[Byte](len.toInt)
      is.read(buffer)
      println(new String(buffer, StandardCharsets.UTF_8))
    }

  // myAcquireRelease is just a value. Won't execute anything here until interpreted
  val myAcquireRelease: Task[Unit] = for {
    file   <- ZIO.attempt(new File("/tmp/hello"))
    len    = file.length
    string <- ZIO.acquireReleaseWith(ZIO.attempt(new FileInputStream(file)))(closeStream)(convertBytes(_, len))
  } yield string
}
```

## Caching and Memoization

Memoization caches the result of an effect or function computation, preventing redundant calculations when the same input is requested multiple times. This section covers indefinite memoization with `ZIO#memoize` and `ZIO.memoize`. For time-limited caching, see the "Time-Limited Caching" subsection below.

### Memoizing Effects

To memoize an effect and cache its result—useful when the same expensive computation may be executed multiple times—call `memoize` on it:

```scala

val expensiveComputation: ZIO[Any, Nothing, Int] = ZIO.succeed(42)

val memoized: ZIO[Any, Nothing, ZIO[Any, Nothing, Int]] =
  expensiveComputation.memoize
```

The memoized effect produces a cached effect that runs the original computation only once. Subsequent calls return the cached result:

```scala

def computeValue: ZIO[Any, Nothing, Int] = {
  ZIO.succeed {
    println("Computing...")
    42
  }
}

object Example extends ZIOAppDefault {
  def run =
    for {
      memoized <- computeValue.memoize
      _        <- memoized  // prints "Computing..."
      _        <- memoized  // returns cached result, no print
      _        <- memoized  // returns cached result, no print
    } yield ()
}
```

:::info
When a fiber computing a memoized value is interrupted, the result is discarded and awaiting fibers transparently retry the computation. This ensures that interruption of one fiber does not propagate to others waiting for the same memoized result.
:::

### Memoizing Functions

To create a memoized version of a function that returns a `ZIO` effect, use the `ZIO.memoize` constructor, which caches results based on input arguments:

```scala

val expensiveLookup: String => ZIO[Any, Nothing, Int] = key => ZIO.succeed(key.length)

for {
  memoized <- ZIO.memoize(expensiveLookup)
  result1  <- memoized("hello")   // computes and caches
  result2  <- memoized("hello")   // returns cached result
  result3  <- memoized("world")   // different input, computes anew
} yield (result1, result2, result3)
```

### Time-Limited Caching

Use `ZIO#cached` to cache the result of an effect with an automatic expiration time. This is useful when results have a limited lifetime and should be refreshed periodically. The cache is thread-safe and supports concurrent access from multiple fibers.

Note: `IO[E, A]` used in this section is a type alias for `ZIO[Any, E, A]`, representing an effect that has no environment requirements.

#### Basic Caching with `cached`

Call `cached` with a time-to-live duration to create a cached version of an effect. The `cached` method returns an effect that produces an `IO` (which is a type alias for `ZIO[Any, E, A]`). When you execute the returned `IO`, it will run the original effect once and cache the result for the specified duration:

```scala

val expensiveData: ZIO[Any, Nothing, String] = ZIO.succeed("data")

for {
  // cached is of type IO[Nothing, String] (equivalent to ZIO[Any, Nothing, String])
  cachedIO <- expensiveData.cached(5.minutes)
  result1  <- cachedIO  // runs computation and caches result
  result2  <- cachedIO  // returns cached result (within 5 minutes)
} yield (result1, result2)
```

The return type is `ZIO[Any, Nothing, IO[Nothing, String]]`, which means `cached` returns an effect that, when executed, produces a cached `IO` effect that you can reuse multiple times.

When the time-to-live duration expires, the cache is invalidated and the effect runs again on the next call:

```scala

def fetchUserData: ZIO[Any, Nothing, String] = ZIO.succeed("user-data")

for {
  cached <- fetchUserData.cached(5.minutes)
  _      <- cached                      // runs and caches
  _      <- ZIO.sleep(6.minutes)
  result <- cached                      // TTL expired, recomputes
} yield result
```

**Comparison with `memoize`**: Unlike `ZIO.memoize` which caches results indefinitely (based on function arguments), `cached` provides time-limited caching with automatic expiration. Use `cached` when you need periodic refresh of results, and `memoize` when you want permanent caching of expensive computations.

#### Caching with Manual Invalidation

Call `cachedInvalidate` to obtain both the cached effect and a separate effect for manually invalidating the cache before its TTL expires (useful when you need to cache-bust based on external events), returning a tuple of the cached effect and an invalidation function:

```scala

def freshData: ZIO[Any, Nothing, String] = ZIO.succeed("data")

for {
  pair              <- freshData.cachedInvalidate(1.hour)
  (cached, invalidate) = pair
  result1 <- cached      // runs and caches
  result2 <- cached      // returns cached result
  _       <- invalidate  // manually clear cache before TTL expires
  result3 <- cached      // recomputes since cache was invalidated
} yield (result1, result2, result3)
```

#### Concurrent Access

Multiple fibers can safely await the same cached result. The first fiber triggers computation while others wait for the result. This ensures the underlying effect runs only once even with concurrent access:

```scala

def expensiveComputation: ZIO[Any, Nothing, Int] = ZIO.succeed {
  println("Computing...")
  42
}

for {
  cached <- expensiveComputation.cached(5.minutes)
  fiber1 <- cached.fork
  fiber2 <- cached.fork
  fiber3 <- cached.fork
  result1 <- fiber1.join  // one executes the computation
  result2 <- fiber2.join  // others wait for the same result
  result3 <- fiber3.join  // all get 42, but computed only once
} yield (result1, result2, result3)
```

:::note
The cache uses a `Ref.Synchronized` internally to manage state safely, ensuring that only one computation runs at a time even when multiple fibers call the cached effect concurrently. This guarantees thread-safe, consistent behavior.
:::

## ZIO Aspect

There are two types of concerns in an application, _core concerns_, and _cross-cutting concerns_. Cross-cutting concerns are shared among different parts of our application. We usually find them scattered and duplicated across our application, or they are tangled up with our primary concerns. This reduces the level of modularity of our programs.

A cross-cutting concern is more about _how_ we do something than _what_ we are doing. For example, when we are downloading a bunch of files, creating a socket to download each one is the core concern because it is a question of _what_ rather than the _how_, but the following concerns are cross-cutting ones:

- Downloading files _sequentially_ or in _parallel_
- _Retrying_ and _timing out_ the download process
- _Logging_ and _monitoring_ the download process

So they don't affect the return type of our workflows, but they add some new aspects or change their behavior.

To increase the modularity of our applications, we can separate cross-cutting concerns from the main logic of our programs. ZIO supports this programming paradigm, which is called _aspect-oriented programming_.

The `ZIO` effect has a data type called `ZIOAspect`, which allows modifying a `ZIO` effect and converting it into a specialized `ZIO` effect. We can add a new aspect to a `ZIO` effect with `@@` syntax like this:

```scala

val myApp: ZIO[Any, Throwable, String] =
  ZIO.attempt("Hello!") @@ ZIOAspect.debug
```

As we see, the `debug` aspect doesn't change the return type of our effect, but it adds a new debugging aspect to our effect.

`ZIOAspect` is like a transformer of the `ZIO` effect, which takes a `ZIO` effect and converts it to another `ZIO` effect. We can think of a `ZIOAspect` as a function of type `ZIO[R, E, A] => ZIO[R, E, A]`.

To compose multiple aspects, we can use `@@` operator:

```scala

def download(url: String): ZIO[Any, Throwable, Chunk[Byte]] = ZIO.succeed(???)

ZIO.foreachPar(List("zio.dev", "google.com")) { url =>
  download(url) @@
    ZIOAspect.retry(Schedule.fibonacci(1.seconds)) @@
    ZIOAspect.loggedWith[Chunk[Byte]](file => s"Downloaded $url file with size of ${file.length} bytes")
}
```

The order of aspect composition matters. Therefore, if we change the order, the behavior may change.

---

## ZIOApp

The `ZIOApp` trait is an entry point for a ZIO application that allows sharing layers between applications. It also
provides us the ability to compose multiple ZIO applications.

There is another simpler version of `ZIOApp` called `ZIOAppDefault`. We usually use `ZIOAppDefault` which uses the default ZIO environment (`ZEnv`).

## Running a ZIO effect

The `ZIOAppDefault` has a `run` function, which is the main entry point for running a ZIO application on the JVM:

```scala

object MyApp extends ZIOAppDefault {
  def run = for {
    _ <- Console.printLine("Hello! What is your name?")
    n <- Console.readLine
    _ <- Console.printLine("Hello, " + n + ", good to meet you!")
  } yield ()
}
```

## Accessing Command-line Arguments

ZIO has a service that contains command-line arguments of an application called `ZIOAppArgs`. We can access command-line arguments using the built-in `getArgs` method:

```scala

object HelloApp extends ZIOAppDefault {
  def run = for {
    args <- getArgs
    _ <-
      if (args.isEmpty)
        Console.printLine("Please provide your name as an argument")
      else
        Console.printLine(s"Hello, ${args.head}!")
  } yield ()
}
```

## Customized Runtime

In the ZIO app, by overriding its `bootstrap` value, we can map the current runtime to a customized one. Let's customize it by introducing our own executor:

```scala

object CustomizedRuntimeZIOApp extends ZIOAppDefault {
  override val bootstrap = Runtime.setExecutor(
    Executor.fromThreadPoolExecutor(
      new ThreadPoolExecutor(
        5,
        10,
        5000,
        TimeUnit.MILLISECONDS,
        new LinkedBlockingQueue[Runnable]()
      )
    )
  )

  def run = myAppLogic
}
```

A detailed explanation of the ZIO runtime system can be found on the [runtime](runtime.md) page.

## Installing Low-level Functionalities

We can hook into the ZIO runtime to install low-level functionalities into the ZIO application, such as _logging_, _profiling_, and other similar foundational pieces of infrastructure.

A detailed explanation can be found on the [runtime](runtime.md) page.

## Composing ZIO Applications

To compose ZIO applications, we can use `<>` operator:

```scala

object MyApp1 extends ZIOAppDefault {    
  def run = ZIO.succeed(???)
}

object MyApp2 extends ZIOAppDefault {
  override val bootstrap: ZLayer[Any, Any, Any] =
    asyncProfiler ++ slf4j ++ loggly ++ newRelic

  def run = ZIO.succeed(???)
}

object Main extends ZIOApp.Proxy(MyApp1 <> MyApp2)
```

The `<>` operator combines the layers of the two applications and then runs the two applications in parallel.

## Graceful Shutdown Timeout

When a ZIO application (e.g. one extending `ZIOAppDefault`) receives an external interruption signal such as **SIGINT** when pressing **Ctrl+C**, the runtime will attempt to run all finalizers (cleanup logic) before exiting. By default, `gracefulShutdownTimeout` set to `Duration.Infinity`, which means ZIO will wait indefinitely for finalizers unless you override it.

Below are two examples: one where cleanup finishes within the timeout and one where cleanup deliberately exceeds it.

### Example 1: Finalizer completes within the timeout

```scala

object MyApp extends ZIOAppDefault {
  // Wait at most 30 seconds for all finalizers to complete on SIGINT
  override def gracefulShutdownTimeout: Duration = 30.seconds

  val run: ZIO[ZIOAppArgs with Scope, Any, Any] =
    ZIO.acquireReleaseWith(
      acquire = ZIO.logInfo("Acquiring resource...").as("MyResource")
    )(release =
      _ =>
        ZIO.logInfo("Releasing resource (3s) ...") *> ZIO.sleep(3.seconds) *>
          ZIO.logInfo("Cleanup done")
    ) { resource =>
      ZIO.logInfo(s"Running with $resource, press Ctrl+C to interrupt") *> ZIO.never
    }
}
```

In this example, `MyApp` starts and logs `Acquiring resource...`. When you press Ctrl+C (sending SIGINT), ZIO interrupts the main fiber and immediately runs the finalizer which logs `Releasing resource (3s) ...` and then sleeps for three seconds. Because the finalizer completes its work well within the 30s timeout, the runtime finishes cleanup and the process exits normally.

### Example 2: Finalizer exceeds the timeout
```scala

object MyAppTimeout extends ZIOAppDefault {
  // Wait at most 5 seconds for finalizers to complete on SIGINT
  override def gracefulShutdownTimeout: Duration = 5.seconds

  val run: ZIO[ZIOAppArgs with Scope, Any, Any] =
    ZIO.acquireReleaseWith(
      acquire = ZIO.logInfo("Acquiring resource...").as("MyResource")
    )(release =
      _ =>
        ZIO.logInfo("Releasing resource (20s) ...") *> ZIO.sleep(20.seconds) *>
          ZIO.logInfo("Cleanup done")
    ) { resource =>
      ZIO.logInfo(s"Running with $resource, press Ctrl+C to interrupt") *> ZIO.never
    }
}
```
Here, `MyAppTimeout` starts and logs `Acquiring resource...`. If you press Ctrl+C after a few seconds, ZIO interrupts the main fiber and starts running the finalizer, logging `Releasing resource (20s) ...` before sleeping for twenty seconds. However, `gracefulShutdownTimeout` is set to just five seconds, ZIO waits those five seconds and then prints exactly:

```bash
**** WARNING ****
Timed out waiting for ZIO application to shut down after 5 seconds. You can adjust your application's shutdown timeout by overriding the `shutdownTimeout` method
```

At that point, the JVM process exits immediately even though the 20-second finalizer has not yet finished.

:::note
Currently, `gracefulShutdownTimeout` is implemented for the **JVM** and **Scala Native** only
:::

---

## Automatic Layer Construction

ZIO also has an automatic layer construction facility, which takes care of building dependency graphs from the individual layers and building blocks. So instead of manually composing layers together to build the final layer, we can only provide individual layers to the ZIO application, and it will do the rest.

The automatic layer construction takes place at the _compile-time_, so if there is a problem in providing a layer, we will receive an error or warning message. So it helps us to diagnose the problem. Additionally, it has a way to print the dependency graph using built-in debug layers.

## Providing Individual Layers to a ZIO Application

When we provide individual layers using `ZIO#provide`, `ZIO#provideCustom`, or `ZIO#provideSome` to a ZIO application, the compiler will create the dependency graph automatically from the provided layers:

:::info
We have a [separate section](dependency-propagation.md) that describes different methods for providing layers to the ZIO application.
:::

Assume we have written the following services (`Cake`, `Chocolate`, `Flour`, and `Spoon`):

```scala

trait Cake

object Cake {
  val live: ZLayer[Chocolate & Flour, Nothing, Cake] =
    for {
      _ <- ZLayer.environment[Chocolate & Flour]
      cake <- ZLayer.succeed(new Cake {})
    } yield cake
}

trait Spoon

object Spoon {
  val live: ULayer[Spoon] =
    ZLayer.succeed(new Spoon {})
}

trait Chocolate

object Chocolate {
  val live: ZLayer[Spoon, Nothing, Chocolate] =
    ZLayer.service[Spoon].project(_ => new Chocolate {})
}

trait Flour

object Flour {
  val live: ZLayer[Spoon, Nothing, Flour] =
    ZLayer.service[Spoon].project(_ => new Flour {})
}
```

The `Cake` service has the following dependency graph:

```
          Cake
          /   \
   Chocolate   Flour
       |         |
     Spoon     Spoon
```

Now we can write an application that uses the `Cake` service as below:

```scala

val myApp: ZIO[Cake, IOException, Unit] = for {
  cake <- ZIO.service[Cake]
  _    <- Console.printLine(s"Yay! I baked a cake with flour and chocolate: $cake")
} yield ()
```

The type of `myApp` indicates we should provide `Cake` to this ZIO application to run it. Let's give it that and see what happens:

```scala
object MainApp extends ZIOAppDefault {
  def run =
    myApp.provide(Cake.live)
}

// error:
// 
// ──── ZLAYER ERROR ────────────────────────────────────────────────────
// 
//  Please provide layers for the following 2 types:
// 
//    Required by Cake.live
//    1. Chocolate
//    2. Flour
//    
// ──────────────────────────────────────────────────────────────────────
```

Here are the errors that will be printed:

```
──── ZLAYER ERROR ────────────────────────────────────────────────────

 Please provide layers for the following 2 types:

   Required by Cake.live
   1. Chocolate
   2. Flour
   
──────────────────────────────────────────────────────────────────────
```

It says that we missed providing `Chocolate` and `Flour` layers. Now let's add these two missing layers:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    myApp.provide(
      Cake.live,
      Chocolate.live,
      Flour.live
    )
}

// error:
// 
// ──── ZLAYER ERROR ────────────────────────────────────────────────────
// 
// Please provide a layer for the following type:
// 
// Required by Flour.live
// 1. Spoon
// 
// Required by Chocolate.live
// 1. Spoon
// 
// ──────────────────────────────────────────────────────────────────────
```

Again, the compiler asks us to provide another dependency called `Spoon`:

```
──── ZLAYER ERROR ────────────────────────────────────────────────────

Please provide a layer for the following type:

Required by Flour.live
1. Spoon

Required by Chocolate.live
1. Spoon

──────────────────────────────────────────────────────────────────────
```

Finally, our application compiles without any errors:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    myApp.provide(
      Cake.live,
      Chocolate.live,
      Flour.live,
      Spoon.live  
    )
}
```

Note that the order of dependencies doesn't matter. We can provide them in any order.

Now, let's compare the automatic layer construction with the manual one:

```scala

object MainApp extends ZIOAppDefault {

  val layers: ULayer[Cake] =
      (((Spoon.live >>> Chocolate.live) ++ (Spoon.live >>> Flour.live)) >>> Cake.live)

  def run = myApp.provideLayer(layers)

}
```

## Automatically Assembling Layers

1. **ZLayer.make[R]** — Using `ZLayer.make[R]`, we can provide a type `R` and then provide individual layers as arguments, it will automatically assemble these layers to create a layer of type `R`.

For example, we can create a `Cake` layer as below:

```scala

val cakeLayer: ZLayer[Any, Nothing, Cake] =
  ZLayer.make[Cake](
    Cake.live,
    Chocolate.live,
    Flour.live,
    Spoon.live
  )
```

We can also create a layer for intersections of services:

```scala

val chocolateAndFlourLayer: ZLayer[Any, Nothing, Chocolate & Flour] =
  ZLayer.make[Chocolate & Flour](
    Chocolate.live,
    Flour.live,
    Spoon.live
  )
```

2. **ZLayer.makeSome[R0, R]** — Automatically constructs a layer for the provided type `R`, leaving a remainder `R0`:

```scala

val cakeLayer: ZLayer[Spoon, Nothing, Cake] =
  ZLayer.makeSome[Spoon, Cake](
    Cake.live,
    Chocolate.live,
    Flour.live
  )
```

## ZLayer Debugging

To debug ZLayer construction, we have two built-in layers, i.e., `ZLayer.Debug.tree` and `ZLayer.Debug.mermaid`.

Let's include the `ZLayer.Debug.tree` layer into the layer construction:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    myApp.provide(
      Cake.live,
      Chocolate.live,
      Flour.live,
      Spoon.live,
      ZLayer.Debug.tree
    )
}
```

The following debug messages will be generated by the compiler:

```
[info]   ZLayer Wiring Graph
[info]
[info] ◉ Cake.live
[info] ├─◑ Chocolate.live
[info] │ ╰─◑ Spoon.live
[info] ╰─◑ Flour.live
[info]   ╰─◑ Spoon.live
[info] 
```

If we use the `ZLayer.Debug.mermaid` layer, it will generate the following debug messages:

```
[info]   ZLayer Wiring Graph  
[info] 
[info] ◉ Cake.live
[info] ├─◑ Chocolate.live
[info] │ ╰─◑ Spoon.live
[info] ╰─◑ Flour.live
[info]   ╰─◑ Spoon.live
[info] 
[info] Mermaid Live Editor Link
[info] https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGhcbiAgICBDb25zb2xlLmxpdmVcbiAgICBDYWtlLmxpdmUgLS0+IENob2NvbGF0ZS5saXZlXG4gICAgQ2FrZS5saXZlIC0tPiBGbG91ci5saXZlXG4gICAgRmxvdXIubGl2ZSAtLT4gU3Bvb24ubGl2ZVxuICAgIFNwb29uLmxpdmVcbiAgICBDaG9jb2xhdGUubGl2ZSAtLT4gU3Bvb24ubGl2ZVxuICAgICIsIm1lcm1haWQiOiAie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwgInVwZGF0ZUVkaXRvciI6IHRydWUsICJhdXRvU3luYyI6IHRydWUsICJ1cGRhdGVEaWFncmFtIjogdHJ1ZX0=
```

---

## Building Dependency Graph

We have two options to build a dependency graph:

1. [Manual layer construction](manual-layer-construction.md)— This method uses ZIO's composition operators such as horizontal (`++`) and vertical (`>>>`) compositions.
2. [Automatic layer construction](automatic-layer-construction.md)— It uses metaprogramming to automatically create the dependency graph at compile time.

Assume we have the following dependency graph with two top-level dependencies:

```
           DocRepo                ++          UserRepo
      ____/   |   \____                       /     \
     /        |        \                     /       \
 Logging  Database  BlobStorage          Logging   Database
                         |                  
                      Logging            
```

Now, assume that we have written an application that finally needs two services: `DocRepo` and `UserRepo`:

```scala
val myApp: ZIO[DocRepo with UserRepo, Throwable, Unit] = ZIO.attempt(???)
```

1. To create the dependency graph for this ZIO application manually, we can use the following code:

```scala
val appLayer: URLayer[Any, DocRepo with UserRepo] =
  ((Logging.live ++ Database.live ++ (Logging.live >>> BlobStorage.live)) >>> DocRepo.live) ++
    ((Logging.live ++ Database.live) >>> UserRepo.live)
    
val res: ZIO[Any, Throwable, Unit] = myApp.provideLayer(appLayer)
```

2. As the development of our application progress, the number of layers will grow, and maintaining the dependency graph manually could be tedious and hard to debug. So, we can automatically construct dependencies with friendly compile-time hints, using `ZIO#provide` operator:

```scala
val res: ZIO[Any, Throwable, Unit] =
  myApp.provide(
    Logging.live,
    Database.live,
    BlobStorage.live,
    DocRepo.live,
    UserRepo.live
  )
```

The order of dependencies doesn't matter:

```scala
val res: ZIO[Any, Throwable, Unit] =
  myApp.provide(
    DocRepo.live,
    BlobStorage.live,
    Logging.live,
    Database.live,
    UserRepo.live
  )
```

If we miss some dependencies, it doesn't compile, and the compiler gives us the clue:

```scala
val app: ZIO[Any, Throwable, Unit] =
  myApp.provide(
    DocRepo.live,
    BlobStorage.live,
//    Logging.live,
    Database.live,
    UserRepo.live
  )
```

```
  ZLayer Wiring Error  

❯ missing Logging
❯     for DocRepo.live

❯ missing Logging
❯     for UserRepo.live
```

:::note
The `ZIO#provide` method, together with its variant `ZIO#provideSome`, is default and easier way of injecting dependencies to the environmental effect. We do not require creating the dependency graph manually, it will be automatically generated.

In contrast, the `ZIO#provideLayer`, and its variant `ZIO#provideSomeLayer`, is useful for low-level and custom cases.
:::

---

## Getting Started With Dependency Injection in ZIO

:::caution
In this page, we will focus on essential parts of dependency injection in ZIO. So in some examples we are not going to cover all the best practices for writing ZIO services.

In real world applications, we encourage to use [service pattern](../service-pattern/service-pattern.md) to write ZIO services.
:::

## Essential Steps of Dependency Injection in ZIO

We can achieve dependency injection through these three simple steps:

1. Accessing services from the ZIO environment through the `ZIO.serviceXYZ` operations.
2. Writing application logic using services and composing them together.
3. Building the dependency graph using manual or automatic layer construction (optional).
4. Providing dependencies to the ZIO environment through the `ZIO.provideXYZ` operations.

### Step 1: Accessing Services From The ZIO Environment

To write application logic, we need to access services from the ZIO environment. We can do this by using the `ZIO.serviceXYZ` operation.

For example, assume we have the following services:

```scala

final class A {
  def foo: UIO[String] = ZIO.succeed("Hello!")
}

final class B {
  def bar: UIO[Int] = ZIO.succeed(42)
}
```

When we call `ZIO.service[A]`, we are asking the ZIO environment for the `A` service. So then we can access all the functionality of the `A` service:

```scala
val effect: ZIO[A, Nothing, String] =
  for {
    a <- ZIO.service[A] 
    r <- a.foo
  } yield r
```

The signature of the above effect, says that in order to produce a value of type `String`, I need the `A` service from the ZIO environment.

We can also use `ZIO.serviceWith`/`ZIO.serviceWithZIO` to directly access one of the service functionalities:

```scala
object A {
  def foo: ZIO[A, Nothing, String] = ZIO.serviceWithZIO[A](_.foo) 
}

object B {
  def bar: ZIO[B, Nothing, Int] = ZIO.serviceWithZIO[B](_.bar)
}
```

### Step 2: Writing Application Logic Using Services

ZIO is a composable data type on its environment type parameter. So when we have an effect that requires the `A` service, and also we have another effect that requires the `B` service; when we compose these two services together, the resulting effect requires both `A` and `B` services:

```scala
// Sequential Composition Example
val myApp: ZIO[A with B, Nothing, (String, Int)] =
  for {
    a <- A.foo
    b <- B.bar
  } yield (a, b)
```

```scala
// Parallel Composition Example
val myApp: ZIO[A with B, Nothing, (String, Int)] = A.foo <&> B.bar
```

Now the `myApp` effect requires `A` and `B` services to fulfill its functionality. We can see that we are writing application logic, we are not concerned about how services will be created! We are focused on using services to write the application logic.

In the next step, we are going to build a dependency graph that holds two `A` and `B` services.

### Step 3: Building The Dependency Graph (Optional)

To be able to run our application, we need to build the dependency graph that it needs. This can be done using the `ZLayer` data type. It allows us to build up the whole application's dependency graph by composing layers manually or automatically.

Assume each of these services has its own layer like the below:

```scala
object A {
  def foo: ZIO[A, Nothing, String] = 
    ZIO.serviceWithZIO[A](_.foo) 
  
  val layer: ZLayer[Any, Nothing, A] = 
    ZLayer.succeed(new A) 
}

object B {
  def bar: ZIO[B, Nothing, Int] = 
    ZIO.serviceWithZIO[B](_.bar)
  
  val layer: ZLayer[Any, Nothing, B] = 
    ZLayer.succeed(new B)
}
```

In the previous example, the `myApp` application requires the `A` and `B` services. We can build that manually by composing two `A` and `B` layers horizontally:

```scala
val appLayer: ZLayer[Any, Nothing, A with B] = 
  A.layer ++ B.layer
```

Or we can use automatic layer construction:

```scala
val appLayer: ZLayer[Any, Nothing, A with B] =
  ZLayer.make[A with B](A.layer, B.layer)
```

:::note
Automatic layer construction is useful when the dependency graph is large and complex. So in simple cases, it doesn't demonstrate the power of automatic layer construction.
:::

### Step 4: Providing Dependencies to the ZIO Environment

To run our application, we need to provide (inject) all dependencies to the ZIO environment. This can be done by using one of the `ZIO.provideXYZ` operations. This allows us to propagate dependencies from button to top:

Let's provide our application with the `appLayer`:

```scala
val result: ZIO[Any, Nothing, (String, Int)] = myApp.provideLayer(appLayer)
```

Here the `ZLayer` data types act as a dependency/environment eliminator. By providing required dependencies to our ZIO application, `ZLayer` eliminates all dependencies from the environment of our application.

That's it! Now we can run our application:

```scala

object MainApp extends ZIOAppDefault {
  def run = result
}
```

Usually, when we use automatic layer construction, we skip the second step and instead provide all dependencies directly to the `ZIO.provide` operation. It takes care of building the dependency graph and providing the dependency graph to our ZIO application:

```scala

object MainApp extends ZIOAppDefault {
  def run = myApp.provide(A.layer, B.layer)
}
```

## Dependency Injection and Service Pattern

### Dependency Injection When Writing Services

When writing services, we might want to use other services. In such cases, we would like dependent services injected into our service. This is where we need to use dependency injection in order to write services.

In ZIO, when we write services, we use class constructors to pass dependencies to the service. This is similar to the object-oriented style.

For example, assume we have written the following `A` and `B` services:

```scala

final class A {
  def foo: ZIO[Any, Nothing, String] = ZIO.succeed("Hello!")
}

object A {
  def foo: ZIO[A, Nothing, String] = ZIO.serviceWithZIO[A](_.foo)

  val layer: ZLayer[Any, Nothing, A] = ZLayer.succeed(new A)
}

final class B {
  def bar: ZIO[Any, Nothing, Int] = ZIO.succeed(42)
}

object B {
  def bar: ZIO[B, Nothing, Int] = ZIO.serviceWithZIO[B](_.bar)

  val layer: ZLayer[Any, Nothing, B] = ZLayer.succeed(new B)
}
```

In order to write a service that depends on `A` and `B` services, we use the class constructor to pass dependencies to our service:

```scala
final case class C(a: A, b: B) {
  def baz: ZIO[Any, Nothing, Unit] =
    for {
      _ <- a.foo
      _ <- b.bar
    } yield ()
}
```

To write a layer for our `C` service, we can use `ZIO.service` to access dependent services effectfully and pass them to the service's constructor:

```scala
object C {
  def baz: ZIO[C, Nothing, Unit] = ZIO.serviceWithZIO[C](_.baz)

  val layer: ZLayer[A with B, Nothing, C] =
    ZLayer {
      for {
        a <- ZIO.service[A]
        b <- ZIO.service[B]
      } yield C(a, b)
    }
}
```

Now, assume we have the following application logic:

```scala
val myApp: ZIO[A with B with C, Nothing, Unit] =
  for {
    _ <- A.foo
    _ <- C.baz
  } yield ()
```

In order to run the application, we should provide the `A`, `B` and `C` services:

```scala

object MainApp extends ZIOAppDefault {
  def run = myApp.provide(A.layer, B.layer, C.layer)
}
```

### Dependency Injection When Writing Services Using Interfaces

Although dependency injection is not about coding to the interface, it is a good pattern to have testable and configurable programs. 

When we code to the interface, our application logic will not dependent on any concrete implementation. So we can replace implementations, without changing the application. This is what [Service Pattern](../service-pattern/service-pattern.md) encourages us when writing services.

Let's try an example. Assume we want to implement service `C` which is implemented in terms of `A` and `B` services. We want to keep our code modular and testable.

The first step is to define interfaces for each service. This gives us the contract for how our services work together and lets us figure out our architecture and divide and conquer:

```scala

trait A {
  def foo: ZIO[Any, Nothing, Int]
}

trait B {
  def bar: ZIO[Any, Nothing, String]
}

trait C {
  def baz: ZIO[Any, Nothing, Unit]
}
```

The next step is to create implementations of our services taking their dependencies as constructor parameters. It's just constructor-based dependency injection:

```scala
final case class ALive() extends A {
  def foo = ZIO.succeed(42)
}

final case class BLive() extends B {
  def bar: ZIO[Any, Nothing, String] = ZIO.succeed("Hello!")
}

final case class CLive(a: A, b: B) extends C {
  def baz: ZIO[Any, Nothing, Unit] =
    for {
      _ <- a.foo
      _ <- b.bar
    } yield ()
}
```

Now, we need to create layers for each of our implementations. This lets ZIO automatically wire them together. It also lets us take care of any setup or tear down. We use `ZIO.service` to grab things from the environment:

```scala modc:silent

object ALive {
  val layer: ZLayer[Any, Nothing, ALive] = ZLayer.succeed(ALive())
}

object BLive {
  val layer: ZLayer[Any, Nothing, BLive] = ZLayer.succeed(BLive())
}

object CLive {
  val layer: ZLayer[B with A, Nothing, CLive] =
    ZLayer {
      for {
        a <- ZIO.service[A]
        b <- ZIO.service[B]
      } yield CLive(a, b)
    }
}
```

Finally, it is time to write our application logic in terms of our services.  We use `ZIO.service` once more in our main application to actually access the service that contains our main application logic and call it:

```scala

val myApp: ZIO[A with C, Nothing, Unit] =
  for {
    a <- ZIO.service[A]
    _ <- a.foo
    c <- ZIO.service[C]
    _ <- c.baz
  } yield ()
```

Now, in order to run our application, we wire all of our services together with `ZIO#provide` and inject them to our application:

```scala

object MainApp extends ZIOAppDefault {
  def run = myApp.provide(
    ALive.layer,
    BLive.layer,
    CLive.layer
  )
}
```

For any purpose, if we decided to use another implementation for the `A` service, we can replace it easily without changing our application logic:

```scala

final case class ACustom() extends A {
  def foo = ZIO.succeed(84)
}

object ACustom {
  val layer: ZLayer[Any, Nothing, A] = ZLayer.succeed(ACustom())
}

object MainApp extends ZIOAppDefault {
  def run = myApp.provide(
    ACustom.layer,
    BLive.layer,
    CLive.layer
  )
}
```

## Conclusion

Following is a summary of some essential points when using dependency injection in ZIO:

1. For each service, we should write a layer that contains the recipe for creating the service.
2. We use class constructors to pass dependencies to our services. So inside the service, it is not idiomatic to use `ZIO.service` to access dependent services.
3. When writing a layer for a service that is dependent on other services, we use `ZIO.service` to access required services from the environment and then pass them to the service's constructor.
4. We use layers to compose and wire them together to create the dependency graph.

---

## Layers Are Shared by Default

Layer memoization allows a layer to be created once and used multiple times in the dependency graph. So if we use the same layer twice, e.g. `(a >>> b) ++ (a >>> c)`, then the `a` layer will be allocated only once.

## Layers are Memoized by Default When Providing Globally

One important feature of a ZIO application is that layers are shared by default, meaning that if the same layer is used twice, and if we provide the layer [globally](overriding-dependency-graph.md#global-environment) the layer will only be allocated a single time. For every layer in our dependency graph, there is only one instance of it that is shared between all the layers that depend on it.

For example, assume we have the three `A`, `B`, and `C` services. The implementation of both `B` and `C` are dependent on the `A` service:

```scala

trait A
trait B
trait C

case class BLive(a: A) extends B
case class CLive(a: A) extends C

val a: ZLayer[Any, Nothing, A] =
  ZLayer(ZIO.succeed(new A {}).debug("initialized"))

val b: ZLayer[A, Nothing, B] =
  ZLayer {
    for {
      a <- ZIO.service[A]
    } yield BLive(a)
  }

val c: ZLayer[A, Nothing, C] =
  ZLayer {
    for {
      a <- ZIO.service[A]
    } yield CLive(a)
  }
```

Although both `b` and `c` layers require the `a` layer, the `a` layer is instantiated only once. It is shared with both `b` and `c`:

```scala

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[B & C, Nothing, Unit] =
    for {
      _ <- ZIO.service[B]
      _ <- ZIO.service[C]
    } yield ()
    
  // alternative: myApp.provideLayer((a >>> b) ++ (a >>> c))
  def run = myApp.provide(a, b, c) 
}
// Output:
// initialized: MainApp3$$anon$32@62c8b8d3
```

## Acquiring a Fresh Version

If we don't want to share a module, we should create a fresh, non-shared version of it through `ZLayer#fresh`.

```scala

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[B & C, Nothing, Unit] =
    for {
      _ <- ZIO.service[B]
      _ <- ZIO.service[C]
    } yield ()

  def run = myApp.provideLayer((a.fresh >>> b) ++ (a.fresh >>> c))
}
// Output:
// initialized: MainApp$$anon$22@7eb282da
// initialized: MainApp$$anon$22@6397a26a
```

## Layers Are Not Memoized When Providing Locally

If we don't provide a layer globally but instead provide them [locally](overriding-dependency-graph.md#local-environment), that layer doesn't support memoization by default.

In the following example, we provided the `A` layer two times locally and the ZIO doesn't memoize the construction of the `A` layer. So, it will be initialized two times:

```scala

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[Any, Nothing, Unit] =
    for {
      _ <- ZIO.service[A].provide(a) // providing locally
      _ <- ZIO.service[A].provide(a) // providing locally
    } yield ()

  def run = myApp
}
// The output:
// initialized: MainApp$$anon$1@cd60bde
// initialized: MainApp$$anon$1@a984546
```

## Manual Memoization

We can memoize the `A` layer manually using the `ZLayer#memoize` operator. It will return a scoped effect that, if evaluated, will return the lazily computed result of this layer:

```scala

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[Any, Nothing, Unit] =
    ZIO.scoped {
      a.memoize.flatMap { aLayer =>
        for {
          _ <- ZIO.service[A].provide(aLayer)
          _ <- ZIO.service[A].provide(aLayer)
        } yield ()
      }
    }
    
  def run = myApp
}
// The output:
// initialized: MainApp$$anon$1@2bfc2bcc
```

---

## Dependency Propagation

When we write an application, our application has a lot of dependencies. We need a way to provide implementations and to feed and propagate all dependencies throughout the whole application. We can solve the propagation problem by using _ZIO environment_.

During the development of an application, we don't care about implementations. Incrementally, when we use various effects with different requirements on their environment, all parts of our application compose together, and at the end of the day we have a ZIO effect which requires some services as an environment. Before running this effect by `unsafeRun` we should provide an implementation of these services into the ZIO Environment of that effect.

ZIO has some facilities for doing this. `ZIO#provide` is the core function that allows us to _feed_ an `R` to an effect that requires an `R`.

Notice that the act of `provide`ing an effect with its environment, eliminates the environment dependency in the resulting effect type, represented by type `Any` of the resulting environment.

## Using `ZIO#provideEnvironment` Method

The `ZIO#provideEnvironment` takes an instance of `ZEnvironment[R]` and provides it to the `ZIO` effect which eliminates its dependency on `R`:

```scala
trait ZIO[-R, +E, +A] {
  def provideEnvironment(r: => ZEnvironment[R]): IO[E, A]
}
```

This is similar to dependency injection, and the `provide*` function can be thought of as _inject_.

Assume we have the following services:

```scala
trait EmailService {
  def send(email: String, content: String): UIO[Unit]
}

object EmailService {
  def send(email: String, content: String) = ZIO.serviceWithZIO[EmailService](_.send(email, content))
}
```

Let's write a simple program using `EmailService` service:

```scala
val app: ZIO[EmailService, Nothing, Unit] = EmailService.send("john@doe.com", "Hello John!")
```

We can `provide` implementation of `EmailService` service into the `app` effect:

```scala
val loggingImpl = new EmailService {
  override def send(email: String, content: String): UIO[Unit] = ???
}

val effect = app.provideEnvironment(ZEnvironment(loggingImpl))
```

Most of the time, we don't use `ZIO#provideEnvironment` directly to provide our services; instead, we use `ZLayer` to construct the dependency graph of our application, then we use methods like `ZIO#provide`, `ZIO#provideSome`, `ZIO#provideSomeAuto` and `ZIO#provideCustom` to propagate dependencies into the environment of our ZIO effect.

## Using `ZIO#provide` Method

Unlike the `ZIO#provideEnvironment` which takes a `ZEnvironment[R]`, the `ZIO#provide` takes a `ZLayer` to the ZIO effect and translates it to another level.

Assume we have written this piece of program that requires `Foo` and `Bar` services:

```scala
trait Foo {
  def foo(): UIO[String]
}

object Foo {
  def foo(): ZIO[Foo, Nothing, String] = ZIO.serviceWithZIO[Foo](_.foo())
}

case class FooLive() extends Foo {
  override def foo(): UIO[String] = ZIO.succeed("foo")
}

object FooLive {
  val layer: ULayer[Foo] = ZLayer.succeed(FooLive())
}

trait Bar {
  def bar(): UIO[Int]
}

object Bar {
  def bar(): ZIO[Bar, Nothing, Int] = ZIO.serviceWithZIO[Bar](_.bar())
}

case class BarLive() extends Bar {
  override def bar(): UIO[Int] = ZIO.succeed(1)
}

object BarLive {
  val layer: ULayer[Bar] = ZLayer.succeed(BarLive())
}

val myApp: ZIO[Foo & Bar, Nothing, Unit] = for {
  foo  <- Foo.foo()
  bar  <- Bar.bar()
  _    <- ZIO.debug(s"foo: $foo, bar: $bar")
} yield ()
```

We provide implementations of `Foo`, `Bar` services to the `myApp` effect by using `ZIO#provide` method:

```scala
val mainEffect: ZIO[Any, Nothing, Unit] = 
  myApp.provide(FooLive.layer, BarLive.layer)
```

As we see, the type of our effect converted from `ZIO[Foo & Bar, Nothing, Unit]` which requires two services to `ZIO[Any, Nothing, Unit]` effect which doesn't require any services.

## Using `ZIO#provideSome` Method

Sometimes we have written a program, and we don't want to provide all its requirements. In these cases, we can use `ZIO#provideSome` to partially apply some layers to the `ZIO` effect.

In the previous example, if we just want to provide the `Foo`, we should use `ZIO#provideSome`:

```scala
val mainEffectSome: ZIO[Bar, Nothing, Unit] = 
  myApp.provideSome(FooLive.layer)
```

:::caution
When using `ZIO#provideSome[R0]`, we should provide the remaining type as `R0` type parameter. This workaround helps the compiler to infer the proper types.
:::

## Using `ZIO#provideSomeAuto` Method

In Scala 3 enhanced version of `ZIO#provideSome` is introduced. The `ZIO#provideSomeAuto` method automatically infers the remaining type of the effect.

```scala
val mainEffectSomeAuto =
  myApp.provideSomeAuto(FooLive.layer) // No need to provide `Bar` anywhere
```

---

## Examples

## An Example of a ZIO Application with Multiple Config Layers

In the following example, we have an application that requires `AppConfig` layer, which itself requires `DBConfig` and `ServerConfig` layers:

```scala

case class ServerConfig(host: String, port: Int)
object ServerConfig {
  val layer: ULayer[ServerConfig] =
    ZLayer.succeed(ServerConfig("localhost", 8080))
}

case class DBConfig(name: String)
object DBConfig {
  val layer: ULayer[DBConfig] =
    ZLayer.succeed(DBConfig("my-test-db"))
}

case class AppConfig(db: DBConfig, serverConfig: ServerConfig)
object AppConfig {
  val layer: ZLayer[DBConfig with ServerConfig, Nothing, AppConfig] =
    ZLayer {
      for {
        db     <- ZIO.service[DBConfig]
        server <- ZIO.service[ServerConfig]
      } yield AppConfig(db, server)
    }
}

object MainApp extends ZIOAppDefault {
  val myApp =
    for {
      c <- ZIO.service[AppConfig]
      _ <- ZIO.debug(s"Application started with config: ${c}")
    } yield ()

  def run = myApp.provide(AppConfig.layer, DBConfig.layer, ServerConfig.layer)
}
```

## An Example of Manually Generating a Dependency Graph

Suppose we have defined the `UserRepo`, `DocumentRepo`, `Database`, `BlobStorage`, and `Cache` services and their respective implementations as follows:

```scala

case class User(email: String, name: String)

trait UserRepo {
  def save(user: User): Task[Unit]

  def get(email: String): Task[User]
}

object UserRepo {
  def save(user: User): ZIO[UserRepo, Throwable, Unit] =
    ZIO.serviceWithZIO(_.save(user))

  def get(email: String): ZIO[UserRepo, Throwable, User] =
    ZIO.serviceWithZIO(_.get(email))
}

case class UserRepoLive(cache: Cache, database: Database) extends UserRepo {
  override def save(user: User): Task[Unit] = ???

  override def get(email: String): Task[User] = ???
}

object UserRepoLive {
  val layer: URLayer[Cache & Database, UserRepo] =
    ZLayer {
      for {
        cache    <- ZIO.service[Cache]
        database <- ZIO.service[Database]
      } yield UserRepoLive(cache, database)
    }
}

trait Database

case class DatabaseLive() extends Database

object DatabaseLive {
  val layer: ZLayer[Any, Nothing, Database] =
    ZLayer.succeed(DatabaseLive())
}

trait Cache {
  def save(key: String, value: Array[Byte]): Task[Unit]

  def get(key: String): Task[Array[Byte]]

  def remove(key: String): Task[Unit]
}

class InmemeoryCache() extends Cache {
  override def save(key: String, value: Array[Byte]): Task[Unit] = ???

  override def get(key: String): Task[Array[Byte]] = ???

  override def remove(key: String): Task[Unit] = ???
}

object InmemoryCache {
  val layer: ZLayer[Any, Throwable, Cache] =
    ZLayer(ZIO.attempt(new InmemeoryCache).debug("initialized"))
}

class PersistentCache() extends Cache {
  override def save(key: String, value: Array[Byte]): Task[Unit] = ???

  override def get(key: String): Task[Array[Byte]] = ???

  override def remove(key: String): Task[Unit] = ???
}

object PersistentCache {
  val layer: ZLayer[Any, Throwable, Cache] =
    ZLayer(ZIO.attempt(new PersistentCache).debug("initialized"))
}

case class Document(title: String, author: String, body: String)

trait DocumentRepo {
  def save(document: Document): Task[Unit]

  def get(id: String): Task[Document]
}

object DocumentRepo {
  def save(document: Document): ZIO[DocumentRepo, Throwable, Unit] =
    ZIO.serviceWithZIO(_.save(document))

  def get(id: String): ZIO[DocumentRepo, Throwable, Document] =
    ZIO.serviceWithZIO(_.get(id))
}

case class DocumentRepoLive(cache: Cache, blobStorage: BlobStorage) extends DocumentRepo {
  override def save(document: Document): Task[Unit] = ???

  override def get(id: String): Task[Document] = ???
}

object DocumentRepoLive {
  val layer: ZLayer[Cache & BlobStorage, Nothing, DocumentRepo] =
    ZLayer {
      for {
        cache       <- ZIO.service[Cache]
        blobStorage <- ZIO.service[BlobStorage]
      } yield DocumentRepoLive(cache, blobStorage)
    }
}

trait BlobStorage {
  def store(key: String, value: Array[Byte]): Task[Unit]
}

case class BlobStorageLive() extends BlobStorage {
  override def store(key: String, value: Array[Byte]): Task[Unit] = ???
}

object BlobStorageLive {
  val layer: URLayer[Any, BlobStorage] =
    ZLayer.succeed(BlobStorageLive())
}
```

And then assume we have the following ZIO application:

```scala

def myApp: ZIO[DocumentRepo & UserRepo, Throwable, Unit] =
  for {
    _ <- UserRepo.save(User("john@doe", "john"))
    _ <- DocumentRepo.save(Document("introduction to zio", "john", ""))
    _ <- UserRepo.get("john@doe").debug("retrieved john@doe user")
    _ <- DocumentRepo.get("introduction to zio").debug("retrieved article about zio")
  } yield ()
```

The `myApp` requires `DocumentRepo` and `UserRepo` services to run. So we need to create a `ZLayer` which requires no services and produces `DocumentRepo` and `UserRepo`. We can manually create this layer using [vertical and horizontal layer composition](manual-layer-construction.md#vertical-and-horizontal-composition):

```scala

object MainApp extends ZIOAppDefault {

  val layers: ZLayer[Any, Any, DocumentRepo with UserRepo] =
    (BlobStorageLive.layer ++ InmemoryCache.layer ++ DatabaseLive.layer) >>>
      (DocumentRepoLive.layer >+> UserRepoLive.layer)

  def run = myApp.provideLayer(layers)
}
```

## An Example of Automatically Generating a Dependency Graph

Instead of creating the required layer manually, we can use the `ZIO#provide`. ZIO internally creates the dependency graph automatically based on all dependencies provided:

```scala

object MainApp extends ZIOAppDefault {

  def run =
    myApp.provide(
      InmemoryCache.layer,
      DatabaseLive.layer,
      UserRepoLive.layer,
      BlobStorageLive.layer,
      DocumentRepoLive.layer
    )
    
}
```

## An Example of Providing Different Implementations of the Same Service

Let's say we want to provide different versions of the same service to different services. In this example, both `UserRepo` and `DocumentRepo` services require the `Cache` service. However, we want to provide different cache implementations for these two services. Our goal is to provide an `InmemoryCache` layer for `UserRepo` and a `PersistentCache` layer for the `DocumentRepo` service:

```scala

object MainApp extends ZIOAppDefault {

  val layers: ZLayer[Any, Throwable, UserRepo with DocumentRepo] =
    ((InmemoryCache.layer ++ DatabaseLive.layer) >>> UserRepoLive.layer) ++
      ((PersistentCache.layer ++ BlobStorageLive.layer) >>> DocumentRepoLive.layer)

  def run = myApp.provideLayer(layers)
}

// Output:
// initialized: zio.examples.PersistentCache@6e899128
// initialized: zio.examples.InmemeoryCache@852e20a
```

## An Example of How to Get Fresh Layers

Having covered the topic of [acquiring fresh layers](../../reference/di/dependency-memoization.md#acquiring-a-fresh-version), let's see an example of using the `ZLayer#fresh` operator.

`DocumentRepo` and `UserRepo` services are dependent on an in-memory cache service. On the other hand, let's assume the cache service is quite simple, and we might be prone to cache conflicts between services. While sharing the cache service may cause some problems for our business logic, we should separate the cache service for both `DocumentRepo` and `UserRepo`:

```scala

object MainApp extends ZIOAppDefault {

  val layers: ZLayer[Any, Throwable, UserRepo & DocumentRepo] =
    ((InmemoryCache.layer.fresh ++ DatabaseLive.layer) >>> UserRepoLive.layer) ++
      ((InmemoryCache.layer.fresh ++ BlobStorageLive.layer) >>> DocumentRepoLive.layer)

  def run = myApp.provideLayer(layers)
}

// Output:
// initialized: zio.examples.InmemoryCache@13c9672b
// initialized: zio.examples.InmemoryCache@26d79027
```

## An Example of Pass-through Dependencies

Notice that in the previous examples, both `UserRepo` and `DocuemntRepo` have some [hidden dependencies](../../reference/di/manual-layer-construction.md#hidden-versus-passed-through-dependencies), such as `Cache`, `Database`, and `BlobStorage`.  So these hidden dependencies are no longer expressed in the type signature of the `layers`. From the perspective of a caller, `layers` just outputs a `UserRepo` and `DocuemntRepo` and requires no inputs. The caller does not need to be concerned with the internal implementation details of how the `UserRepo` and `DocumentRepo` are constructed.

An upstream dependency that is used by many other services can be "passed-through" and included in a layer's output. This can be done with the `>+>` operator, which provides the output of one layer to another layer, returning a new layer that outputs the services of _both_.

The following example shows how to passthrough all dependencies to the final layer:

```scala

object MainApp extends ZIOAppDefault {

  // passthrough all dependencies
  val layers: ZLayer[Any, Throwable, Database & BlobStorage & Cache & DocumentRepo & UserRepo] =
    DatabaseLive.layer >+>
      BlobStorageLive.layer >+>
      InmemoryCache.layer >+>
      DocumentRepoLive.layer >+>
      UserRepoLive.layer

  // providing all passthrough dependencies to the ZIO application
  def run = myApp.provideLayer(layers)
}
```

## An Example of Updating Hidden Dependencies

One of the use cases of having explicit all dependencies in the final layer is that we can [update](../../reference/di/examples.md#an-example-of-updating-hidden-dependencies) those hidden layers using `ZLayer#update`. In the following example, we are replacing the `InmemoryCache` with another implementation called `PersistentCache`:

```scala

object MainApp extends ZIOAppDefault {

  def myApp: ZIO[DocumentRepo & UserRepo, Nothing, Unit] =
    for {
      _ <- ZIO.service[UserRepo]
      _ <- ZIO.service[DocumentRepo]
    } yield ()

  val layers: ZLayer[Any, Throwable, Database & BlobStorage & Cache & DocumentRepo & UserRepo] =
    DatabaseLive.layer >+>
      BlobStorageLive.layer >+>
      InmemoryCache.layer >+>
      DocumentRepoLive.layer >+>
      UserRepoLive.layer

  def run =
    myApp.provideLayer(
      layers.update[Cache](_ => new PersistentCache)
    )
}
```

---

## Introduction to Dependency Injection in ZIO

## What is a Dependency?

When we implement a service, we might need to use other services. So a dependency is just another service that is required to fulfill its functionality:

```scala
class Editor {
 val formatter = new Formatter
 val compiler = new Compiler
 
 def formatAndCompile(code: String): UIO[String] = 
 formatter.format(code).flatMap(compiler.compile)
}
```

## What is Dependency Injection?

Dependency injection is a pattern for decoupling the usage of dependencies from their actual creation process. In other words, it is a process of injecting dependencies of service from the outside world. The service itself doesn't know how to create its dependencies.

The following example shows an `Editor` service that depends on `Formatter` and `Compiler` services. It doesn't use dependency injection:

```scala

class Editor {
 private val formatter = new Formatter
 private val compiler = new Compiler
 
 def formatAndCompile(code: String): UIO[String] =
 formatter.format(code).flatMap(compiler.compile)
}
```

The `Editor` class in the above example is responsible for creating the `Formatter` and `Compiler` services. The client of the `Editor` class doesn't have any control over these services. The client can't use a different implementation for the `Formatter` and `Compiler` services. So it makes it hard to test the `Editor` class.

Let's try to change the above example to use the constructor-based dependency injection pattern:

```scala

class Editor(formatter: Formatter, compiler: Compiler) {
 def formatAndCompile(code: String): UIO[String] = ???
}
```

In this example, the `Editor` service is not responsible for creating its dependencies. Instead, they are expected to be injected from the caller site. The `Editor` service does not know how its dependencies are created, they are just injected into its constructor.

So dependency injection is a very simple concept and can be implemented with simple constructs. In a lot of situations, we are not required to use any tools or frameworks.

In the [motivation page](motivation.md) we explain why applications should use the dependency injection pattern in more detail.

## ZIO's Built-in Dependency Injection

ZIO has a full solution to the dependency injection problem. It provides a built-in approach to dependency injection using the following tools in combination together:

1. **ZIO Environment**

    1. We use the `ZIO.serviceXYZ` to access services inside the ZIO environment, without having any knowledge of how the services are created or implemented. Using `ZIO.serviceXYZ` helps us to decouple our usage of services from the implementation of the services.
  
     Consequently, all dependencies will be encoded inside the `R` type parameter of our ZIO application. This specifies which services are required to fulfill the application's functionality.
  
    2. We use the `ZIO.provideXYZ` to provide services to the ZIO environment. This is the opposite operation of `ZIO.serviceXYZ`. It allows us to inject all dependencies into the ZIO environment.
    
2. **ZLayer**— We use layers to create the dependency graph that our application depends on.

We can have dependency injection through three simple steps:

1. Accessing services from the ZIO environment
2. Building the dependency graph
3. Providing services to the ZIO environment

We will discuss them in more detail throughout [this page](dependency-injection-in-zio.md).

## ZIO's Dependency Injection Features

Dependency injection in ZIO is very powerful, which increases developer productivity. Let's recap some important features of dependency injection in ZIO:

1. **Composable**

    1. **Composable Environment**— Because of the very composable nature of the `ZIO` data type, its environment type parameter is also composable. So when we compose multiple `ZIO` effects, where each one requires a specific service, we finally get a `ZIO` effect that requires all the required services that each of the composed effects requires. 
  
     For example, if we `zip` two effects of type `ZIO[A, Nothing, Int]` and `ZIO[B, Throwable, String]`, the result of this operation will become `ZIO[A with B, Throwable, (Int, String)]`. The result operation requires both `A` and `B` services.

    2. **Composable Dependencies**— The `ZLayer` is also composable, as well as ZIO's environment type parameter. So we can compose multiple layers to [create a complex dependency graph](building-dependency-graph.md).

2. **Type-Safe**— All the required dependencies should be provided at compile time. If we forget to provide the required services at compile time, we will get a compile error. So if our program compiles successfully, we can be sure that we won't have runtime errors due to missing dependencies.

3. **Effectful**— We build dependency graphs using `ZLayer`. Since `ZLayer` is effectful, we can create a dependency graph in an effectful way.

4. **Resourceful**— It also helps us to have resourceful dependencies, where we can manage the creation and release phases of the dependencies.

5. **Parallelism**— All dependencies are created in parallel, and will be provided to our application.

## Other Frameworks

Using `ZLayer` along with the ZIO environment to use dependency injection is optional. While we encourage users to use ZIO's idiomatic dependency injection, it is not mandatory.

We can still use other DI solutions. Here are some other options:
- [Guice](https://github.com/google/guice)
- [izumi distage](https://izumi.7mind.io/distage/index.html)
- [MacWire](https://github.com/softwaremill/macwire)

---

## Manual Layer Construction

We said that we can think of the `ZLayer` as a more powerful _constructor_. Constructors are not composable, because they are not values. While a constructor is not composable, `ZLayer` has a nice facility to compose with other `ZLayer`s. So we can say that a `ZLayer` turns a constructor into values.

:::note
In a regular ZIO application we are not required to build the dependency graph through composing layers tougher. Instead, we can provide all dependencies to the ZIO application using `ZIO#provide`, and the ZIO will create the dependency graph manually under the hood. Therefore, use manual layer composition if you know what you're doing.
:::

## Vertical and Horizontal Composition

Assume we have several services with their dependencies, and we need a way to compose and wire up these dependencies to create the dependency graph of our application. `ZLayer` is a ZIO solution for this problem, it allows us to build up the whole application dependency graph by composing layers horizontally and vertically.

### Horizontal Composition

Layers can be composed together horizontally with the `++` operator. When we compose layers horizontally, the new layer requires all the services that both of them require and produces all services that both of them produce. Horizontal composition is a way of composing two layers side-by-side. It is useful when we combine two layers that don't have any relationship with each other.

We can compose `fooLayer` and `barLayer` _horizontally_ to build a layer that has the requirements of both, to provide the capabilities of both, through `fooLayer ++ barLayer`:

```scala

val fooLayer: ZLayer[A, Throwable, B] = ???        // A ==> B
val barLayer: ZLayer[C, Nothing  , D] = ???        // C ==> D

val horizontal: ZLayer[A & C, Throwable, B & D] =  // A & C ==> B & D
  fooLayer ++ barLayer
```

### Vertical Composition

We can also compose layers _vertically_ using the `>>>` operator, meaning the output of one layer is used as input for the subsequent layer, resulting in one layer with the requirement of the first, and the output of the second.

For example if we have a layer that requires `A` and produces `B`, we can compose this with another layer that requires `B` and produces `C`; this composition produces a layer that requires `A` and produces `C`. The feed operator, `>>>`, stack them on top of each other by using vertical composition. This sort of composition is like _function composition_, feeding an output of one layer to an input of another:

```scala

val fooLayer: ZLayer[A, Throwable, B] = ???  // A ==> B
val barLayer: ZLayer[B, Nothing  , C] = ???  // B ==> C

val horizontal: ZLayer[A, Throwable, C] =    // A ==> C
  fooLayer >>> barLayer
```

## Hidden Versus Passed-through Dependencies

ZLayer has a `passthrough` operator which returns a new layer that produces the outputs of this layer but also passes-through the inputs:

```scala

val fooLayer: ZLayer[A, Nothing, B] = ???  // A ==> B

val result1 : ZLayer[A, Nothing, A & B] =  // A ==> A & B
  fooLayer.passthrough
  
val result2 : ZLayer[A, Nothing, A & B] =  // A ==> A & B
  ZLayer.service[A] ++ fooLayer
 
// (A ==> A) ++ (A ==> B)
// (A ==> A & B)
```

By default, the `ZLayer` hides intermediate dependencies when composing vertically. For example, when we compose `fooLayer` with `barLayer` vertically, the output would be a `ZLayer[A, Throwable, C]`. This hides the dependency on the `B` layer. By using the above technique, we can pass through hidden dependencies.

Let's include the `B` service into the upstream dependencies of the final layer using the `ZIO.service[B]`. We can think of `ZIO.service[B]` as an _identity function_ (`B ==> B`).

```scala

val fooLayer: ZLayer[A, Throwable, B] = ???  // A  ==> B
val barLayer: ZLayer[B, Throwable, C] = ???  // B  ==> C

val finalLayer: ZLayer[A & B, Throwable, C] = // A & B ==> C
  (fooLayer ++ ZLayer.service[B]) >>> barLayer

// ((A ==> B) ++ (B ==> B)) >>> (B ==> C)
// (A & B ==> B) >> (B ==> C)
// (A & B ==> C)
```

Or we may want to include the middle services in the output channel of the final layer, resulting in a new layer with the inputs of the first layer and the outputs of both layers:

```scala

val fooLayer: ZLayer[A, Throwable, B] = ??? // A  ==> B
val barLayer: ZLayer[B, Throwable, C] = ??? // B  ==> C

val finalLayer: ZLayer[A, Throwable, B & C] = // A ==> B & C
  fooLayer >>> (ZLayer.service[B] ++ barLayer)
  
// (A ==> B) >>> ((B ==> B) ++ (B ==> C))
// (A ==> B) >>> (B ==> B & C)
// (A ==> B & C)
```

We can do the same with the `>+>` operator:

```scala

val fooLayer: ZLayer[A, Throwable, B] = ??? // A  ==> B
val barLayer: ZLayer[B, Throwable, C] = ??? // B  ==> C

val finalLayer: ZLayer[A, Throwable, B & C] = // A ==> B & C
  fooLayer >+> barLayer
```

This technique is useful when we want to defer the creation of some intermediate services and require them as part of the input of the final layer. For example, assume we have these two layers:

```scala

val fooLayer: ZLayer[A    , Throwable, B] = ???   // A     ==> B
val barLayer: ZLayer[B & C, Throwable, D] = ???   // B & C ==> D

val finalLayer: ZLayer[A & C, Throwable, D] = // A & C ==> B & D
  fooLayer >>> barLayer
```

So we can defer the creation of the `C` layer using `ZLayer.service[C]`:

```scala

val fooLayer: ZLayer[A    , Throwable, B] = ??? // A ==> B 
val barLayer: ZLayer[B & C, Throwable, D] = ??? // B & C ==> D

val layer: ZLayer[A & C, Throwable, D] =        // A & C ==> D
  (fooLayer ++ ZLayer.service[C]) >>> barLayer

// ((A ==> B) ++ (C ==> C)) >>> (B & C ==> D)
// (A & C ==> B & C) >>> (B & C ==> D)
// (A & C ==> D)
```

Here is an example in which we passthrough all requirements to bake a `Cake` so all the requirements are available to all the downstream services:

```scala

trait Baker 
trait Ingredients
trait Oven
trait Dough
trait Cake

lazy val baker      : ZLayer[Any, Nothing, Baker] = ???
lazy val ingredients: ZLayer[Any, Nothing, Ingredients] = ???
lazy val oven       : ZLayer[Any, Nothing, Oven] = ???
lazy val dough      : ZLayer[Baker & Ingredients, Nothing, Dough] = ???
lazy val cake       : ZLayer[Baker & Oven & Dough, Nothing, Cake] = ???

lazy val all: ZLayer[Any, Nothing, Baker & Ingredients & Oven & Dough & Cake] =
  baker >+>       // Baker
  ingredients >+> // Baker & Ingredients
  oven >+>        // Baker & Ingredients & Oven
  dough >+>       // Baker & Ingredients & Oven & Dough
  cake            // Baker & Ingredients & Oven & Dough & Cake
```

This allows a style of composition where the `>+>` operator is used to build a progressively larger set of services, with each new service able to depend on all the services before it. If we passthrough dependencies and later want to hide them we can do so through a simple type ascription:

```scala
lazy val hidden: ZLayer[Any, Nothing, Cake] = all
```

The `ZLayer` makes it easy to mix and match these styles. If we build our dependency graph more explicitly, we can be confident that dependencies used in multiple parts of the dependency graph will only be created once due to memoization and sharing.

Using these simple operators we can build complex dependency graphs.

## Updating Local Dependencies

Given a layer, it is possible to update one or more components it provides. We update a dependency in two ways:

1. **Using the `update` Method** — This method allows us to replace one requirement with a different implementation:

```scala

val origin: ZLayer[Any, Nothing, String & Int & Double] = 
  ZLayer.succeedEnvironment(ZEnvironment[String, Int, Double]("foo", 123, 1.3))

val updated1 = origin.update[String](_ + "bar")
val updated2 = origin.update[Int](_ + 5)
val updated3 = origin.update[Double](_ - 0.3)
```

Here is an example of updating a config layer:

```scala

case class AppConfig(poolSize: Int)

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[AppConfig, IOException, Unit] =
    for {
      config <- ZIO.service[AppConfig]
      _ <- Console.printLine(s"Application config after the update operation: $config")
    } yield ()

  val appLayers: ZLayer[Any, Nothing, AppConfig] =
    ZLayer(ZIO.succeed(AppConfig(5)).debug("Application config initialized"))

  val updatedConfig: ZLayer[Any, Nothing, AppConfig] =
    appLayers.update[AppConfig](c =>
      c.copy(poolSize = c.poolSize + 10)
    )

  def run = myApp.provide(updatedConfig)
}

// Output:
// Application config initialized: AppConfig(5)
// Application config after the update operation: AppConfig(15)
```

2. **Using Horizontal Composition** — Another way to update a requirement is to horizontally compose in a layer that provides the updated service. The resulting composition will replace the old layer with the new one:

```scala

val origin: ZLayer[Any, Nothing, String & Int & Double] =
  ZLayer.succeedEnvironment(ZEnvironment[String, Int, Double]("foo", 123, 1.3))

val updated = origin ++ ZLayer.succeed(321)
```

Let's see an example of updating a config layer:

```scala

case class AppConfig(poolSize: Int)

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[AppConfig, IOException, Unit] =
    for {
      config <- ZIO.service[AppConfig]
      _      <- Console.printLine(s"Application config after the update operation: $config")
    } yield ()

  val appLayers: ZLayer[Any, Nothing, AppConfig] =
    ZLayer(ZIO.succeed(AppConfig(5)).debug("Application config initialized"))

  val updatedConfig: ZLayer[Any, Nothing, AppConfig] =
    appLayers ++ ZLayer.succeed(AppConfig(8))

  def run = myApp.provide(updatedConfig)
}
// Output:
// Application config initialized: AppConfig(5)
// Application config after the update operation: AppConfig(8)
```

## Cyclic Dependencies

The `ZLayer` mechanism makes it impossible to build cyclic dependencies, making the initialization process very linear, by construction.

---

## Motivation

:::caution
In this section, we are going to study how ZIO supports dependency injection by providing pedagogical examples. Examples provided in these sections are not idiomatic and are not meant to be used as a reference. We will discuss the idiomatic way to use dependency injection in ZIO later.

So feel free to skip reading this section if you are not interested to learn the underlying concepts in detail.
:::

Assume we have two services called `Formatter` and `Compiler` like the below:

```scala

class Formatter {
  def format(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}

class Compiler {
  def compile(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}
```

We want to create an editor service, which uses these two services. Hence, we are going to instantiate the required services inside the `Editor` class:

```scala
class Editor {
  private val formatter: Formatter = new Formatter()
  private val compiler: Compiler   = new Compiler()

  def formatAndCompile(code: String): UIO[String] =
    formatter.format(code).flatMap(compiler.compile)
}
```

There are some problems with this approach:
1. Users of the `Editor` service haven't any control over how dependencies will be created.
2. Users of the `Editor` service cannot use different implementations of `Formatter` and `Compiler` services. For example, we would like to test the `Editor` service with a mock version of `Formatter` and `Compiler`. With this approach, mocking these dependencies is hard.
3. The `Editor` service is tightly coupled with `Formatter` and `Compiler`. This means any change to these services, may introduce a new change in the `Editor` class.
4. Creating the object graph is a manual process.

Let's see how we can provide a solution to these problems. In the following sections, we will step by step solve these problems, and finally, we will see how ZIO solves the dependency injection problem.

## Step 1: Inversion of Control

On solution to the first problem is inverting the control to the user of the `Editor` service, which is called _Inversion of Control_.

Now lets instead of instantiating the dependencies inside the `Editor` service, create them outside the `Editor` service and pass them to the `Editor` service:

```scala
class Editor(formatter: Formatter, compiler: Compiler) {
  def formatAndCompile(code: String): UIO[String] =
    formatter.format(code).flatMap(compiler.compile)
}
```

Now the `Editor` service is decoupled from how the `Formatter` and `Compiler` services are created. The client of the `Editor` service can instantiate the `Formatter` and `Compiler` services and pass them to the `Editor` service:

```scala
val formatter = new Formatter() // creating formatter
val compiler  = new Compiler()  // creating compiler
val editor = new Editor(formatter, compiler) // assembling formatter and compiler into editor

editor.formatAndCompile("println(\"Hello, world!\")")
``` 

## Step 2: Decoupling from Implementations

In the previous step, we delegated the creation of dependencies to the client of the `Editor` service. This decouples the `Editor` service from the creation of the dependencies. But it is not enough. We still coupled to the concrete classes called `Formatter` and `Compiler`. The user of the `Editor` service cannot use different implementations rather than the `Formatter` and `Compiler` services. This is where the object-oriented approach comes into play. By programming to interfaces, we can encapsulate the `Editor` service and make it independent of concrete implementations:

```scala
trait Formatter {
  def format(code: String): UIO[String]
}

class ScalaFormatter extends Formatter {
  def format(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}

trait Compiler {
  def compile(code: String): UIO[String]
}

class ScalaCompiler extends Compiler {
  def compile(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}

trait Editor {
  def formatAndCompile(code: String): UIO[String]
}

class EditorLive(formatter: Formatter, compiler: Compiler) extends Editor {
  def formatAndCompile(code: String): UIO[String] =
    formatter.format(code).flatMap(compiler.compile)
}

val formatter = new ScalaFormatter() // Creating Formatter
val compiler  = new ScalaCompiler()  // Creating Compiler
val editor    = new EditorLive(formatter, compiler) // Assembling formatter and compiler into CodeEditor

editor.formatAndCompile("println(\"Hello, world!\")")
```

Now, we can test the `Editor` service easily without having to worry about the implementation of the `Formatter` and `Compiler` services. To test the `Editor` service, we can use a mock implementation of its dependencies:

```scala
class MockFormatter extends Formatter {
  def format(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}

class MockCompiler extends Compiler {
  def compile(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}

val formatter = new MockFormatter() // Creating mock formatter
val compiler  = new MockCompiler()  // Creating mock compiler
val editor    = new EditorLive(formatter, compiler) // Assembling formatter and compiler into CodeEditor

val expectedOutput = ???
for {
  r <- editor.formatAndCompile("println(\"Hello, world!\")") 
} yield assertTrue(r == expectedOutput)
```

## Step 3: Binding Interfaces to their Implementations

In the previous step, we successfully decoupled the `Editor` service from concrete dependencies. However, there is still a problem. When the application grows, the number of dependencies might increase. So, instead of injecting the dependencies manually whenever needed, we would like to maintain a mapping from interfaces to their implementations in a container, and then whenever needed, we can ask for the required dependency from the container.

So we need a container that maintains this mapping. ZIO has a type-level map, called `ZEnvironment`, which can do that for us:

```scala
val scalaFormatter = new ScalaFormatter() // Creating Formatter
val scalaCompiler  = new ScalaCompiler() // Creating Compiler
val myEditor       = // Assembling Formatter and Compiler into an Editor
  new EditorLive(
    scalaFormatter,
    scalaCompiler
  )

val environment = ZEnvironment[Formatter, Compiler, Editor](scalaFormatter, scalaCompiler, myEditor)
// Map(
//  Formatter -> scalaFormatter,
//  Compiler  -> scalaCompiler
//  Editor    -> myEditor
//)
```

Now, whenever we need an object of type `Formatter`, `Compiler`, or `Editor`, we can ask the `environment` for them.

```scala
object MainApp extends ZIOAppDefault {
  def run = 
    environment.get[Editor].formatAndCompile("println(\"Hello, world!\")")
}
```

Here is another example:

```scala
val workflow: ZIO[Any, Nothing, Unit] =
  for {
    f <- environment.get[Formatter].format("println(\"Hello, world!\")")
    _ <- environment.get[Compiler].compile(f)
  } yield ()
```

## Step 4: Effectful Constructors

Until now, we discussed the creation of services where the creation process was not effectful. But, assume in order to implement the `Editor` service, we need the `Counter` service, and the creation of `Counter` itself is effectful:

```scala
trait Counter {
  def inc: UIO[Unit]
  def dec: UIO[Unit]
  def get: UIO[Int]
}

case class CounterLive(ref: Ref[Int]) extends Counter {
  def inc: UIO[Unit] = ref.update(_ + 1)
  def dec: UIO[Unit] = ref.update(_ - 1)
  def get: UIO[Int]  = ref.get
}

object CounterLive {
  // Effectful constructor
  def make: UIO[Counter] = Ref.make(0).map(new CounterLive(_))
}

class EditorLive(
    formatter: Formatter,
    compiler: Compiler,
    counter: Counter
) extends Editor {
  def formatAndCompile(code: String): UIO[String] = ???
}
```

To instantiate `EditorLive` we can't use the same technique as before:

```scala
val scalaFormatter = new ScalaFormatter() // Creating Formatter
val scalaCompiler  = new ScalaCompiler()  // Creating Compiler
val myEditor       =                      // Assembling Formatter and Compiler into an Editor
  new EditorLive(
    scalaFormatter,
    scalaCompiler,
    CounterLive.make // Compiler Error: Type mismatch: expected: Counter, found: UIO[Counter]
  )
```

We can use `ZIO#flatMap` to create the dependency graph but to make it easier, we have a special data type called `ZLayer`. It is effectful, so we can use it to create the dependency graph effectfully:

```scala
trait Formatter {
  def format(code: String): UIO[String]
}

case class ScalaFormatter() extends Formatter {
  def format(code: String): UIO[String] = 
    ZIO.succeed(code) // dummy implementation
}

object ScalaFormatter {
  val layer: ULayer[Formatter] = ZLayer.succeed(ScalaFormatter())
}

trait Compiler {
  def compile(code: String): UIO[String]
}

case class ScalaCompiler() extends Compiler {
  def compile(code: String): UIO[String] = ZIO.succeed(code)
}
object ScalaCompiler {
  val layer = ZLayer.succeed(ScalaCompiler())
}

trait Editor {
  def formatAndCompile(code: String): UIO[String]
}

trait Counter {
  def inc: UIO[Unit]
  def dec: UIO[Unit]
  def get: UIO[Int]
}

case class CounterLive(ref: Ref[Int]) extends Counter {
  def inc: UIO[Unit] = ref.update(_ + 1)
  def dec: UIO[Unit] = ref.update(_ - 1)
  def get: UIO[Int]  = ref.get
}

object CounterLive {
  // Effectful constructor
  def make: UIO[Counter] = Ref.make(0).map(new CounterLive(_))

  val layer: ULayer[Counter] = ZLayer.fromZIO(CounterLive.make)
}

case class EditorLive(
    formatter: Formatter,
    compiler: Compiler,
    counter: Counter
) extends Editor {
  def formatAndCompile(code: String): UIO[String] = ???
}

object EditorLive {
  val layer: ZLayer[Counter with Compiler with Formatter, Nothing, Editor] =
    ZLayer {
      for {
        // we will discuss ZIO.service later
        formatter <- ZIO.service[Formatter] 
        compiler  <- ZIO.service[Compiler]
        counter   <- ZIO.service[Counter]
      } yield EditorLive(formatter, compiler, counter)
    }
}

object MainApp extends ZIOAppDefault {
  val environment =
    ((ScalaFormatter.layer ++ ScalaCompiler.layer ++ CounterLive.layer) >>> EditorLive.layer).build

  def run =
    for {
      editor <- environment.map(_.get[Editor])
      _      <- editor.formatAndCompile("println(\"Hello, world!\")")
    } yield ()
}
```

:::note
`ZLayer` is not only an effectful constructor, but also it supports concurrency and resource safety when constructing layers.
:::

## Step 5: Using ZIO Environment To Declare Dependencies

So far, we learned that the `ZEnvironment` can act as an IoC container. Whenever we need a dependency, we can ask for it from the environment:

```scala
val workflow: ZIO[Scope, Nothing, Unit] =
  for {
    env <- (ScalaFormatter.layer ++ ScalaCompiler.layer).build
    f   <- env.get[Formatter].format("println(\"Hello, world!\")")
    _   <- env.get[Compiler].compile(f)
  } yield ()
```

While this is a pretty good solution, there is a problem with it. Every time we need a dependency, we are asking for that instantly. In a large codebase, this imperative style of asking for dependencies can be tedious. This is an imperative style. It's better to make this declarative. So instead of **asking for dependencies** it is better to **declare dependencies**.
Accordingly, we can use the `R` type-parameter of the `ZIO` data type which supports the declarative style:

```scala
val workflow: ZIO[Compiler with Formatter, Nothing, String] =
 for {
   f  <- ZIO.service[Formatter] 
   r1 <- f.format("println(\"Hello, world!\")")
   c  <- ZIO.service[Compiler]
   r1 <- c.compile(r1)
 } yield r1
```

This is a much better solution. We just declare that we need the `Compiler` and the `Formatter` services using `ZIO.service` and then we compose pieces of our program to create the final application. The final workflow has all requirements in its type signature. For example, the `ZIO[Compiler with Formatter, Nothing, String]` type says that I need the `Compiler` and the `Formatter` services to produce the final result as a `String`.

Finally, we can provide all the dependencies through the `ZIO#provideEnvironment` method:

```scala
workflow.provideLayer(ScalaCompiler.layer ++ ScalaFormatter.layer)
```

## Step 6: Automatic Dependency Graph Generation

For large applications, it can be tedious to manually create the dependency graph. ZIO has a built-in mechanism empowered by using macros to automatically generate the dependency graph. To use this feature, we can use the `ZIO#provide` method:

```scala
workflow.provide(ScalaCompiler.layer, ScalaFormatter.layer)
```

We should provide all required dependencies and then the ZIO will construct the dependency graph and provide that to our application.

---

## Overriding Dependency Graph

We can create a ZIO application by providing a local or a global environment, or a combination:

## Global Environment

It is usual when writing ZIO applications to provide layers at the end of the world. Then we provide layers to the whole ZIO application all at once. This pattern uses a single global environment for all ZIO applications:

```scala

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[ServiceA & ServiceB & ServiceC & ServiceD, Throwable, Unit] = ???
    
  def run = myApp.provide(a, b, c, d)
}
```

## Local Environment

Occasionally, we may need to provide different environments for different parts of our application, or it may be necessary to provide a single global environment for the entire application except for some inner layers.

Providing a layer locally is analogous to overriding a method in an object-oriented paradigm. So we can think of that as overriding the global environment:

```scala

object MainApp extends ZIOAppDefault {
  def myApp: ZIO[A & B & C, Throwable, Unit] = {
    def innerApp1: ZIO[A & B & C, Throwable, Unit] = ???
    def innerApp2: ZIO[A & C,     Throwable, Unit] = ???

    innerApp1.provideSomeLayer[A & B](localC) *> innerApp2
  }

  def run = myApp.provide(globalA, globalB, globalC)
}
```

ZIO Test's [Live service](../test/services/live.md) uses this pattern to provide real environment to a single part of an effect.

---

## Providing Different Implementation of a Service

One of the benefits of using dependency injection is that, we can write our application in a way that without modifying the application logic, we can provide different implementations of services to our application.

## Example 1: Config Service

In the next example, we have a ZIO application that uses the `AppConfig` service:

```scala

case class AppConfig(poolSize: Int)

object AppConfig {
  def poolSize: ZIO[AppConfig, Nothing, Int] =
    ZIO.serviceWith[AppConfig](_.poolSize)

  val appArgsLayer: ZLayer[ZIOAppArgs, Nothing, AppConfig] =
    ZLayer {
      ZIOAppArgs.getArgs
        .map(_.headOption.map(_.toInt).getOrElse(8))
        .map(poolSize => AppConfig(poolSize))
    }

  val systemEnvLayer: ZLayer[Any, SecurityException, AppConfig] =
    ZLayer.fromZIO(
      System
        .env("POOL_SIZE")
        .map(_.headOption.map(_.toInt).getOrElse(8))
        .map(poolSize => AppConfig(poolSize))
    )
}

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[AppConfig, Nothing, Unit] =
    for {
      poolSize <- AppConfig.poolSize
      _        <- ZIO.debug(s"Application started with $poolSize pool size.")
    } yield ()

  def run = myApp.provideSome[ZIOAppArgs](AppConfig.appArgsLayer)
}
```

The `AppConfig` has two layers, `appArgsLayer` and `systemEnvLayer`. The first one uses command-line arguments to create the `AppConfig` and the second one uses environment variables. As we can see, without changing the core logic of our application, we can easily change the way we get the configuration:

```diff
object MainApp extends ZIOAppDefault {
  val myApp: ZIO[AppConfig, Nothing, Unit] =
    for {
      poolSize <- AppConfig.poolSize
      _        <- ZIO.debug(s"Application started with $poolSize pool size.")
    } yield ()

-  def run = myApp.provideSome[ZIOAppArgs](AppConfig.appArgsLayer)
+  def run = myApp.provide(AppConfig.systemEnvLayer)
}
```

## Example 2: Logging Service

In this example, we have a ZIO application that uses the `Logging` service. And we provided two implementations of the `Logging` service: `SimpleLogger` and `DateTimeLogger`:

```scala

trait Logging {
  def log(msg: String): ZIO[Any, IOException, Unit]
}

object Logging {
  def log(msg: String): ZIO[Logging, IOException, Unit] =
    ZIO.serviceWithZIO[Logging](_.log(msg))
}

case class DateTimeLogger() extends Logging {
  override def log(msg: String): ZIO[Any, IOException, Unit] =
    for {
      dt <- Clock.currentDateTime
      _  <- Console.printLine(s"$dt: $msg")
    } yield ()
}

object DateTimeLogger {
  val live: ULayer[DateTimeLogger] =
    ZLayer.succeed(DateTimeLogger())
}

case class SimpleLogger() extends Logging {
  override def log(msg: String): ZIO[Any, IOException, Unit] =
    Console.printLine(msg)
}
object SimpleLogger {
  val live: ULayer[SimpleLogger] =
    ZLayer.succeed(SimpleLogger())
}
```

Now, let's write a ZIO application that uses the `Logging` service:

```scala

val myApp: ZIO[Logging, IOException, Unit] =
  for {
    _ <- Logging.log("Application started.")
    _ <- Logging.log("Application ended.")
  } yield ()
```

Now, we can run our application, just by providing one of the implementations of the `Logging` service. Let's run it with the `SimpleLogger` implementation:

```scala
object MainApp extends ZIOAppDefault {
  def run = myApp.provide(SimpleLogger.live)
}
```

Now, we can see that, without changing the core logic of our application, we can easily change the logger implementation:

```scala
object MainApp extends ZIOAppDefault {
  def run = myApp.provide(DateTimeLogger.live)
}
```

---

## ZLayer: Constructor as a Value

Before jumping into the next section, which will explain dependency injection in ZIO, let's take a look at the philosophy behind the `ZLayer` data type.

In the [motivation](motivation.md) section, we find out that the ordinary Scala constructors are not powerful enough to help us to build the dependency graph easily. So `ZLayer` was created to overcome scala constructors' limitations.

We can think of `ZLayer` as an alternative to constructors but with the following powerful features:
- Composable with a nice ergonomic API
- Asynchronous so it doesn't block the thread
- Effectful and resourceful
- Support for concurrency and parallelism

Let's see the following example written using scala constructors:

```scala
class Editor(formatter: Formatter, compiler: Compiler) {
  // ...
}

class Compiler() {
  // ...
}

class Formatter() {
  // ...
}
```

We can say that each constructor is a function that takes some arguments as dependencies and returns a new instance of the class:
- `() => Formatter`
- `() => Compiler`
- `(Formatter, Compiler) => Editor`

The `ZLayer` reifies the conceptual idea of scala constructor and turned it into typed value which is equipped with lots of compositional operators and also supporting asynchronous operations.

So in other words, `ZLayer` is a type-safe data type that describes the asynchronous, effectful and resourceful process of building the dependency graph. We can say that a `ZLayer[Input, E, Output]` is a recipe that takes some services as input and returns some services as output.

For example, a `ZLayer` of type `ZLayer[Any, Nothing, Formatter]` is a constructor that doesn't take any services from the input and returns `Formatter` as output. Also, a `ZLayer` of type `ZLayer[Formatter with Compiler, Nothing, Editor]` is a constructor that takes `Formatter` and `Compiler` services from the input and returns `Editor` as output:

```scala

object Formatter {
  val layer: ZLayer[Any, Nothing, Formatter] =
    ZLayer.succeed(new Formatter())
}

object Compiler {
  val layer: ZLayer[Any, Nothing, Compiler] =
    ZLayer.succeed(new Compiler())
}

object Editor {
  val layer: ZLayer[Formatter with Compiler, Nothing, Editor] =
    ZLayer {
      for {
        formatter <- ZIO.service[Formatter]
        compiler  <- ZIO.service[Compiler]
      } yield new Editor(formatter, compiler) 
    }
}
```

## Composable Constructors

With scala constructors we compose services like the below to create the dependency graph:

```scala
val formatter = new Formatter()
val compiler = new Compiler()
val editor = new Editor(formatter, compiler)
```

While Scala constructors are a type of Scala function. Composable functions in Scala are not as ergonomic as ZLayer for constructing dependency graphs.

With ZLayer we can compose them using operators like `++` and `>>>`:

```scala
val editor: ZLayer[Formatter with Compiler, Nothing, Editor] = 
  (Formatter.layer ++ Compiler.layer) >>> Editor.layer
```

Also, we can compose `Formatter` and `Editor` layers to create a new layer that takes the `Compiler` service and returns the` Editor` service:

```scala
val editor: ZLayer[Compiler, Nothing, Editor] =
  Formatter.layer >>> Editor.layer
```

## Effectful Constructors

If we have a dependency that requires an effectful computation to be initialized, we can't model easily such an operation using ordinary Scala constructors.

In the following example, without the help of `ZIO#flatMap` or `ZLayer`, we can't easily create an instance of the `Editor` class:

```scala

case class Counter(ref: Ref[Int]) {
  def inc: UIO[Unit] = ref.update(_ + 1)
  def dec: UIO[Unit] = ref.update(_ - 1)
  def get: UIO[Int]  = ref.get
}

object Counter {
  // Effectful constructor
  def make: UIO[Counter] = Ref.make(0).map(new Counter(_))
}

class Editor(formatter: Formatter, compiler: Compiler, counter: Counter) {
  // ...
}

object Formatter {
  def make = new Formatter() 
}

object Compiler {
  def make = new Compiler()
}

val editor    =   
  new Editor(
    Formatter.make,
    Compiler.make,
    Counter.make // Compiler Error: Type mismatch: expected: Counter, found: UIO[Counter]
  )
```

Let's see how we can use `ZIO#flatMap` to create `Editor`:

```scala
val editor: ZIO[Any, Nothing, Editor] =
  Counter.make.map { counter =>
    new Editor(
      Formatter.make,
      Compiler.make,
      counter
    )
  }
```

While with `ZLayer`, we can easily have an effectful constructor. We can create `ZLayer` from any `ZIO` effect by using `ZLayer.fromZIO`/`ZLayer.apply` constructor:

```scala
case class Counter(ref: Ref[Int]) {
  def inc: UIO[Unit] = ref.update(_ + 1)
  def dec: UIO[Unit] = ref.update(_ - 1)
  def get: UIO[Int]  = ref.get
}

object Counter {
  val layer: ZLayer[Any, Nothing, Counter] = 
    ZLayer {
      Ref.make(0).map(new Counter(_))
    }
}

class Formatter {
  def format(code: String): UIO[String] = ???
}

object Formatter {
  val layer: ZLayer[Any, Nothing, Formatter] =
    ZLayer.succeed(new Formatter())
}

class Compiler {
  def compile(code: String): UIO[String] = ???
}

object Compiler {
  val layer: ZLayer[Any, Nothing, Compiler] = 
    ZLayer.succeed(new Compiler())
}

class Editor(formatter: Formatter, compiler: Compiler, counter: Counter) {
  def formatAndCompile(code: String): UIO[String] = ???
}

object Editor {
  val layer: ZLayer[Formatter with Compiler with Counter, Nothing, Editor] =
    ZLayer {
      for {
        formatter <- ZIO.service[Formatter]
        compiler  <- ZIO.service[Compiler]
        counter   <- ZIO.service[Counter]
      } yield new Editor(formatter, compiler, counter) 
    }
}
```

Let's try another example. Assume we have a `ZIO` effect that reads the application config from a file, we can create a layer from that:

```scala

case class AppConfig(poolSize: Int)
  
object AppConfig {
  private def loadConfig : Task[AppConfig] = 
    ZIO.attempt(???) // loading config from a file
    
  val layer: TaskLayer[AppConfig] = 
    ZLayer(loadConfig)  // or ZLayer.fromZIO(loadConfig)
}
```

## Resourceful Constructors

Some components of our applications need to be scoped, meaning they undergo a resource acquisition phase before usage, and a resource release phase after usage (e.g. when the application shuts down). As we stated before, the construction of ZIO layers can be effectful and resourceful, this means they can be acquired and safely released when the services are done being utilized.

The `ZLayer` relies on the powerful `Scope` data type and this makes this process extremely simple. We can lift any scoped `ZIO` to `ZLayer` by providing a scoped resource to the `ZLayer.scoped` constructor:

```scala

case class A(a: Int)
object A {
  val layer: ZLayer[Any, Nothing, A] =
    ZLayer.scoped {
      ZIO.acquireRelease(acquire = ZIO.debug("Initializing A") *> ZIO.succeed(A(5)))(
        release = _ => ZIO.debug("Releasing A")
      )
    }
}

object ZIOApp extends ZIOAppDefault {
  val myApp: ZIO[A, Nothing, Int] =
    for {
      a <- ZIO.serviceWith[A](_.a)
    } yield a * a

  def run =
    myApp
      .debug("result")
      .provide(A.layer)
}
```

The output:

```
Initializing A
result: 25
Releasing A
```

We can see that the `A` service is initialized and carefull released when the application is shut down.

Here is another example that uses auto closeable resources:

```scala

val fileLayer: ZLayer[Any, Throwable, BufferedSource] =
  ZLayer.scoped {
    ZIO.fromAutoCloseable(
      ZIO.attemptBlocking(scala.io.Source.fromFile("file.txt"))
    )
  }
```

Finally, let's see a real-world example of creating a layer from scoped resources. Assume we have the following `UserRepository` service:

```scala

trait DBConfig
trait Transactor
trait User

def dbConfig: Task[DBConfig] = ZIO.attempt(???)
def initializeDb(config: DBConfig): Task[Unit] = ZIO.attempt(???)
def makeTransactor(config: DBConfig): ZIO[Scope, Throwable, Transactor] = ZIO.attempt(???)

trait UserRepository {
  def save(user: User): Task[Unit]
}

case class UserRepositoryLive(xa: Transactor) extends UserRepository {
  override def save(user: User): Task[Unit] = ZIO.attempt(???)
}
```

Assume we have written a scoped `UserRepository`:

```scala
def scoped: ZIO[Scope, Throwable, UserRepository] = 
  for {
    cfg <- dbConfig
    _   <- initializeDb(cfg)
    xa  <- makeTransactor(cfg)
  } yield new UserRepositoryLive(xa)
```

We can convert that to `ZLayer` with `ZLayer.scoped`:

```scala
object UserRepositoyLive {
  val layer : ZLayer[Any, Throwable, UserRepository] =
    ZLayer.scoped(scoped)
}
```

## Asynchronous Constructors

We should avoid using blocking operations inside Scala constructors:

```scala
class ProducerInput

class KafkaProducer(input: ProducerInput) {
  def send(message: String): Unit = ???
}

object KafkaProducer {
  def apply() = {
    // Blocking operation, we should avoid it inside constructors
    val input = doSomeBlockingOperation()
    new KafkaProducer(input)
  }

  private def doSomeBlockingOperation(): ProducerInput = ???
}
```

While with `ZLayer`, we can easily use blocking operations:

```scala

class ProducerInput

class KafkaProducer(input: ProducerInput) {
    def send(message: String): Task[Unit] = ???
}

object KafkaProducer {
  val layer =
    ZLayer {
      for {
        input <- ZIO.attemptBlocking(doSomeBlockingOperation())
      } yield (new KafkaProducer(input))
    }

  private def doSomeBlockingOperation(): ProducerInput = ???
}
```

## Parallel Constructors

With `Zlayer` all layers in the dependency graph are executed in parallel:

```scala

case class A(a: Int)
object A {
  val layer: ZLayer[Any, Nothing, A] =
    ZLayer.fromZIO {
      for {
        _ <- ZIO.debug("Initializing A")
        _ <- ZIO.sleep(3.seconds)
        _ <- ZIO.debug("Initialized A")
      } yield A(1)
    }
}

case class B(b: Int)
object B {
  val layer: ZLayer[Any, Nothing, B] =
    ZLayer.fromZIO {
      for {
        _ <- ZIO.debug("Initializing B")
        _ <- ZIO.sleep(2.seconds)
        _ <- ZIO.debug("Initialized B")
      } yield B(2)
    }
}

object ZIOApp extends ZIOAppDefault {
  val myApp: ZIO[A with B, Nothing, Int] =
    for {
      a <- ZIO.serviceWith[A](_.a)
      b <- ZIO.serviceWith[B](_.b)
    } yield a + b

  def run =
    myApp
      .debug("result")
      .provide(A.layer, B.layer)
}
```

The output:

```
Initializing A
Initializing B
Initialized B
Initialized A
result: 3
```

---

## Model Domain Errors Using Algebraic Data Types

It is best to use _algebraic data types (ADTs)_ when modeling errors within the same domain or subdomain.

Sealed traits allow us to introduce an error type as a common supertype and all errors within a domain are part of that error type by extending that:

```scala
sealed trait UserServiceError extends Exception

case class InvalidUserId(id: ID) extends UserServiceError
case class ExpiredAuth(id: ID)   extends UserServiceError
```

In this case, the super error type is `UserServiceError`. We sealed that trait, and we extend it by two cases, `InvalidUserId` and `ExpiredAuth`. Because it is sealed, if we have a reference to a `UserServiceError` we can match against it and the Scala compiler knows there are two possibilities for a `UserServiceError`:

```scala
userServiceError match {
  case InvalidUserId(id) => ???
  case ExpiredAuth(id)   => ???
}
```

This is a sum type, and also an enumeration. The Scala compiler knows only two of these `UserServiceError` exist. If we don't match on all of them, it is going to warn us. We can add the `-Xfatal-warnings` compiler option which treats warnings as errors. By turning on the fatal warning, we will have type-safety control on expected errors. So sealing these traits gives us great power.

Also extending all of our errors from a common supertype helps the ZIO's combinators like flatMap to auto widen to the most specific error type.

Let's say we have this for-comprehension here that calls the `userAuth` function, and it can fail with `ExpiredAuth`, and then we call `userProfile` that fails with `InvalidUserID`, and then we call `generateEmail` that can't fail at all, and finally we call `sendEmail` which can fail with `EmailDeliveryError`. We have got a lot of different errors here:

```scala
val myApp: IO[Exception, Receipt] =
  for {
    service <- userAuth(token)                // IO[ExpiredAuth, UserService]
    profile <- service.userProfile(userId)    // IO[InvalidUserId, Profile]
    body    <- generateEmail(orderDetails)    // IO[Nothing, String]
    receipt <- sendEmail("Your order detail",
       body, profile.email)                   // IO[EmailDeliveryError, Unit]
  } yield receipt
```

In this example, the flatMap operations auto widens the error type to the most specific error type possible. As a result, the inferred error type of this for-comprehension will be `Exception` which gives us the best information we could hope to get out of this. We have lost information about the particulars of this. We no longer know which of these error types it is. We know it is some type of `Exception` which is more information than nothing.

---

## Don't Type Unexpected Errors

When we first discover typed errors, it may be tempting to put every error into the error type parameter. That is a mistake because we can't recover from all types of errors. When we encounter unexpected errors we can't do anything in those cases. We should let the application die. Let it crash is the erlang philosophy. It is a good philosophy for all unexpected errors. At best, we can sandbox it, but we should let it crash.

The context of a domain determines whether an error is expected or unexpected. When using typed errors, sometimes it is necessary to make a typed-error un-typed because in that case, we can't handle the error, and we should let the application crash.

For example, in the following example, we don't want to handle the `IOException` so we can call `ZIO#orDie` to make the effect's failure unchecked. This will translate effect's failure to the death of the fiber running it:

```scala

Console.printLine("Hello, World") // ZIO[Any, IOException, Unit]
  .orDie                          // ZIO[Any, Nothing, Unit]
```

If we have an effect that fails for some `Throwable` we can pick certain recoverable errors out of that, and then we can just let the rest of them kill the fiber that is running that effect. The ZIO effect has a method called `ZIO#refineOrDie` that allows us to do that.

In the following example, calling `ZIO#refineOrDie` on an effect that has an error type `Throwable` allows us to refine it to have an error type of `TemporaryUnavailable`:

```scala

val response: ZIO[Any, Nothing, Response] =
  ZIO
    .attemptBlocking(
      httpClient.fetchUrl(url)
    ) // ZIO[Any, Throwable, Response]
    .refineOrDie[TemporaryUnavailable] {
      case e: TemporaryUnavailable => e
    } // ZIO[Any, TemporaryUnavailable, Response]
    .retry(
      Schedule.fibonacci(1.second)
    ) // ZIO[Any, TemporaryUnavailable, Response]
    .orDie // ZIO[Any, Nothing, Response]
```

In this example, we are importing the `fetchUrl` which is a blocking operation into a `ZIO` value. We know that in case of a service outage it will throw the `TemporaryUnavailable` exception. This is an expected error, so we want that to be typed. We are going to reflect that in the error type. We only expect it, so we know how to recover from it.

Also, this operation may throw unexpected errors like `OutOfMemoryError`, `StackOverflowError`, and so forth. Therefore, we don't include these errors since we won't be handling them at runtime. They are defects, and in case of unexpected errors, we should let the application crash.

Therefore, it is quite common to import a code that may throw exceptions, whether that uses expected errors for error handling or can fail for a wide variety of unexpected errors like disk unavailable, service unavailable, and so on. Generally, importing these operations end up represented as a `Task` (`ZIO[Any, Throwable, A]`). So in order to make recoverable errors typed, we use the `ZIO#refineOrDie` method.

---

## Don't Reflexively Log Errors

In modern async concurrent applications with a lot of subsystems, if we do not type errors, we are not able to see what section of our code fails with what error. Therefore, this can be very tempting to log errors when they happen. So when we lose type-safety in the whole application it makes us be more sensitive and program defensively. Therefore, whenever we are calling an API we tend to catch its errors, log them as below:

```scala

sealed trait UploadError extends Exception
case class FileExist(name: String)          extends UploadError
case class FileNotExist(name: String)       extends UploadError
case class StorageLimitExceeded(limit: Int) extends UploadError

/**
 * This API fail with `FileExist` failure when the provided file name exist.
 */
def upload(name: String): Task[Unit] = {
    if (...)
      ZIO.fail(FileExist(name))
    else if (...)
      ZIO.fail(StorageLimitExceeded(limit)) // This error is undocumented unintentionally
    else
      ZIO.attempt(...)
}

upload("contacts.csv").catchAll {
  case FileExist(name) => delete("contacts.csv") *> upload("contacts.csv")
  case _ =>
    for {
      _ <- ZIO.log(error.toString) // logging the error
      _ <- ZIO.fail(error) // failing again (just like rethrowing exceptions in OOP)
    } yield ()
}
```

In the above code when we see the `upload`'s return type we can't find out what types of error it may fail with. So as a programmer we need to read the API documentation, and see in what cases it may fail. Due to the fact that the documents may be outdated, and they may not provide all error cases, we tend to add another case to cover all the other errors. Expert developers may prefer to read the implementation to find out all expected errors, but it is a tedious task to do.

We don't want to lose any errors. So if we do not use typed errors, it makes us defensive to log every error, regardless of whether they will occur or not.

When we are programming with typed errors, that allows us to never lose any errors. Even if we don't handle all, the error channel of our effect type demonstrate the type of remaining errors:

```scala
val myApp: ZIO[Any, UploadError, Unit] =
  upload("contacts.csv")
    .catchSome {
      case FileExist(name) => delete(name) *> upload(name)
    }
```

It is still going to be sent an unhandled error type as a result. Therefore, there is no way to lose any errors, and they propagate automatically through all the different subsystems in our application, which means we don't have to be fearful anymore. It will be handled by higher-level code, or if it doesn't it will be passed off to something that can.

If we handle all errors using `ZIO#catchAll` the type of error channel become `Nothing` which means there is no expected error remaining to handle:

```scala
val myApp: ZIO[Any, Nothing, Unit] =
  upload("contacts.csv")
    .catchAll {
      case FileExist(name) =>
        ZIO.unit // handling FileExist error case
      case StorageLimitExceeded(limit) =>
        ZIO.unit // handling StorageLimitExceeded error case
    }
```

When we type errors, we know that they can't be lost. So typed errors give us the ability to log less.

---

## Use Union Types to Be More Specific About Error Types

In Scala 3, we have an exciting new feature called union types. By using the union operator, we can encode multiple error types. Using this facility, we can have more precise information on typed errors.

Let's see an example of `Storage` service which have `upload`, `download` and `delete` API:

```scala

type Name = String

enum StorageError extends Exception {
  case ObjectExist(name: Name)            extends StorageError
  case ObjectNotExist(name: Name)         extends StorageError
  case PermissionDenied(cause: String)    extends StorageError
  case StorageLimitExceeded(limit: Int)   extends StorageError
  case BandwidthLimitExceeded(limit: Int) extends StorageError
}

trait Storage {
  def upload(
      name: Name,
      obj: Array[Byte]
  ): ZIO[Any, ObjectExist | StorageLimitExceeded, Unit]

  def download(
      name: Name
  ): ZIO[Any, ObjectNotExist | BandwidthLimitExceeded, Array[Byte]]

  def delete(name: Name): ZIO[Any, ObjectNotExist | PermissionDenied, Unit]
}
```

Union types allow us to get rid of the requirement to extend some sort of common error types like `Exception` or `Throwable`. This allows us to have completely unrelated error types.

In the following example, the `FooError` and `BarError` are two distinct error. They have no super common type like `FooBarError` and also they are not extending `Exception` or `Throwable` classes:

```scala

// Two unrelated errors without having a common supertype
trait FooError
trait BarError

def foo: IO[FooError, Nothing] = ZIO.fail(new FooError {})
def bar: IO[BarError, Nothing] = ZIO.fail(new BarError {})

val myApp: ZIO[Any, FooError | BarError, Unit] = for {
  _ <- foo
  _ <- bar
} yield ()
```

---

## Imperative vs. Declarative Error Handling

To figure out the benefit of typed errors in declarative error handling, we need to understand the drawbacks of the imperative approach and then see how the declarative approach can be used to solve the same problem.

## Imperative Error Handling

In the imperative style, when we encounter a wrong state, we throw an exception, and to handle exceptions, we have to use the `try`/`catch` language construct. Whenever we encounter an exception inside a `try` block, the control flow will jump to the `catch` block. In the catch block, we can handle the exception and decide what to do next. This is a very common pattern in imperative programming.

For example, we can write the `divide` function like this:

```scala
def divide(a: Int, b: Int): Int =
  if (b == 0)
    throw new IllegalArgumentException("Division by zero")
  else
    a / b
```

As we know that this function throws an exception when `b` is zero, so we need to handle the exception when we call this function:

```scala
def readFromConsole: (Int, Int) = ???

val (a, b) = readFromConsole

try {
  Some(divide(a, b))
} catch {
  case _: IllegalArgumentException => None
}
```

## Declarative Error Handling

In declarative error handling, we treat errors as values instead of throwing exceptions. So instead of breaking the flow of the program, we can return a value that represents the error. When we have a workflow of type `ZIO[R, E, A]`, the `E` type parameter is used to represent that our workflow may fail with an error of type `E`.

For example, the following program written with ZIO may fail with an error of type `AgeValidationException`:

```scala

sealed trait AgeValidationException extends Exception
case class NegativeAgeException(age: Int) extends AgeValidationException
case class IllegalAgeException(age: Int)  extends AgeValidationException

def validate(age: Int): ZIO[Any, AgeValidationException, Int] =
  if (age < 0)
    ZIO.fail(NegativeAgeException(age))
  else if (age < 18)
    ZIO.fail(IllegalAgeException(age))
  else ZIO.succeed(age)
```

We can handle errors using `catchAll`/`catchSome` methods instead of using `try`/`catch` blocks:

```scala
validate(17).catchAll {
  case NegativeAgeException(age) => ???
  case IllegalAgeException(age)  => ???
}
```

## Imperative vs. Declarative

### Referential Transparency

We say that an expression is referentially transparent if it can be replaced with its value without changing the behavior of the program. This property helps us to reason about a program easily. Also writing tests for our programs becomes much easier.

Unfortunately, when we throw an exception, we lose the ability to reason about our programs. Exceptions break the referential transparency of our programs. For example, let's say we have the following function:

```scala
def divide10By(b: Int): Option[Int] = {
  val result = divide(10, b)
  try {
    Some(result)
  } catch {
    case _: IllegalArgumentException => None
  }
}
```

If we call `divide10By(0)`, we will get an exception (`IllegalArgumentException`). Now, let's see what happens if we replace the result with its value like this:

```scala
def divide10By(b: Int): Option[Int] = 
  try {
    Some(divide(10, b))
  } catch {
    case _: IllegalArgumentException => None
  }
```

In this case, if we call `divide10By(0)`, we will get the `None` value. The behavior of the function will be changed. We cannot reason about the behavior of our program by substituting expressions with their values. In this style of error handling, the behavior of the program is dependent on where we call our expressions, inside or outside the `try` block.

When we model our programs with the `ZIO`, we sure that our programs are referentially transparent. We can reason about our programs very easily without having to worry about changing the behavior of our programs.

### Type-safety

There is no way to know what errors can be thrown by looking at the function signature. The only way to find out in which circumstance a method may throw an exception is to read and investigate its implementation. So the compiler cannot prevent us from writing unsafe codes. It is also hard for a developer to read the documentation event through reading the documentation is not sufficient as it may be obsolete, or it may don't reflect the exact exceptions.

In ZIO when we see the type of the effect, we can determine what kind of error it can fail with. This helps us to have compile-time type-safety on our programs not only on success values but also on failure values.

### Exhaustivity Checking

When we use `try`/`catch` the compiler doesn't know about errors at compile time, so if we forgot to handle one of the exceptions the compiler doesn't help us to write total functions. This code will crash at runtime because we forgot to handle the `IllegalAgeException` case:

```scala
try {
  validate(17)
} catch {
  case NegativeAgeException(age) => ???
  //  case IllegalAgeException(age) => ???
}
```

When we are using typed errors we can have exhaustive checking support from the compiler. For example, when we are catching all errors if we forgot to handle one of the cases, the compiler warns us about that:

```scala
validate(17).catchAll {
  case NegativeAgeException(age) => ???
}

// match may not be exhaustive.
// It would fail on the following input: IllegalAgeException(_)
```

In the example above, if we only handle `NegativeAgeException`, the compiler will complain about the `IllegalAgeException` being unhandled. This helps us cover all cases and write _total functions_ easily.

> **Note:**
>
> When a function is defined for all possible input values, it is called a _total function_ in functional programming.

### Error Model

The error model based on the `try`/`catch`/`finally` statement is lossy and broken. Because if we have the combinations of these statements we can throw many exceptions, and then we are only able to catch one of them. All the other ones are lost. They are swallowed into a black hole, and also the one that we catch is the wrong one. It is not the primary cause of the failure.

To be more specific, if the `try` block throws an exception, and the `finally` block throws an exception as well, then, if these are caught at a higher level, only the finalizer's exception will be caught normally, not the exception from the try block.

In the following example, we are going to show this behavior:

```scala
 try {
    try throw new Error("e1")
    finally throw new Error("e2")
 } catch {
   case e: Error => println(e)
 }

// Output:
// e2
```

The above program just prints the `e2` while it is not the primary cause of failure. That is why we say the `try`/`catch` model is lossy.

In ZIO, all the errors will still be reported. So even though we are only able to catch one error, the other ones will be reported which we have full control over them. They don't get lost.

Let's write a ZIO version:

```scala
ZIO.fail("e1")
  .ensuring(ZIO.succeed(throw new Exception("e2")))
  .catchAll {
    case "e1" => Console.printLine("e1")
    case "e2" => Console.printLine("e2")
  }

// Output:
// e1
```

ZIO guarantees that no errors are lost. It has a _lossless error model_. This guarantee is provided via a hierarchy of supervisors and information made available via data types such as `Exit` and `Cause`. All errors will be reported. If there's a bug in the code, ZIO enables us to find out about it.

---

## Error Accumulation

Sequential combinators such as `ZIO#zip` and `ZIO.foreach` stop when they reach the first error and return immediately. So their policy on error management is to fail fast.

In the following example, we can see that the `ZIO#zip` operator will fail as soon as it reaches the first failure. As a result, we only see the first error in the stack trace:

```scala

object MainApp extends ZIOAppDefault {
  val f1: ZIO[Any, Nothing, Int] = ZIO.succeed(1)
  val f2: ZIO[Any, String, Int]  = ZIO.fail("Oh uh!").as(2)
  val f3: ZIO[Any, Nothing, Int] = ZIO.succeed(3)
  val f4: ZIO[Any, String, Int]  = ZIO.fail("Oh no!").as(4)

  val myApp: ZIO[Any, String, (Int, Int, Int, Int)] =
    f1 zip f2 zip f3 zip f4

  def run = myApp.debug
}

// Output:
// <FAIL> Oh uh!
// timestamp=2022-03-13T09:26:03.447149388Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
// 	at <empty>.MainApp.f2(MainApp.scala:5)
// 	at <empty>.MainApp.myApp(MainApp.scala:10)
//	at <empty>.MainApp.run(MainApp.scala:12)"
```

There is also the `ZIO.foreach` operator that takes a collection and an effectful operation, then tries to apply the transformation to all elements of the collection. This operator also has the same error management behavior. It fails when it encounters the first error:

```scala

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[Any, String, List[Int]] =
    ZIO.foreach(List(1, 2, 3, 4, 5)) { n =>
      if (n < 4)
        ZIO.succeed(n)
      else
        ZIO.fail(s"$n is not less that 4")
    }

  def run = myApp.debug
}

// Output:
// <FAIL> 4 is not less that 4
// timestamp=2022-03-13T08:28:53.865690767Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: 4 is not less that 4
//	at <empty>.MainApp.myApp(MainApp.scala:9)
//	at <empty>.MainApp.run(MainApp.scala:12)"
```

There are some situations when we need to collect all potential errors in a computation rather than failing fast. In this section, we will discuss operators that accumulate errors as well as successes.

## `ZIO#validate`

It is similar to the `ZIO#zip` operator, it sequentially zips two ZIO effects together, if both effects fail, it combines their causes with `Cause.Then`:

```scala
trait ZIO[-R, +E, +A] {
  def validate[R1 <: R, E1 >: E, B](that: => ZIO[R1, E1, B]): ZIO[R1, E1, (A, B)]
}
```

If any of effecful operations doesn't fail, it results like the `zip` operator. Otherwise, when it reaches the first error it won't stop, instead, it will continue the zip operation until reach the final effect while combining:

```scala

object MainApp extends ZIOAppDefault {
  val f1 = ZIO.succeed(1).debug
  val f2 = ZIO.succeed(2) *> ZIO.fail("Oh uh!")
  val f3 = ZIO.succeed(3).debug
  val f4 = ZIO.succeed(4) *> ZIO.fail("Oh error!")
  val f5 = ZIO.succeed(5).debug

  val myApp: ZIO[Any, String, (Int, Int, Int)] =
    f1 validate f2 validate f3 validate f4 validate f5

  def run = myApp.cause.debug.uncause
}

// Output:
// 1
// 3
// 5
// Then(Fail(Oh uh!,Trace(None,Chunk())),Fail(Oh error!,Trace(None,Chunk())))
// timestamp=2022-03-14T08:53:42.389942626Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
// 	at <empty>.MainApp.run(MainApp.scala:13)
// 	Suppressed: java.lang.String: Oh error!
//		at <empty>.MainApp.run(MainApp.scala:13)"
```

The `ZIO#validatePar` operator is similar to the `ZIO#validate` operator zips two effects but in parallel. As this operator doesn't fail fast, unlike the `ZIO#zipPar` if it reaches a failure, it won't interrupt another running effect. If both effects fail, it will combine their causes with `Cause.Both`:

```scala

object MainApp extends ZIOAppDefault {
  val f1 = ZIO.succeed(1).debug
  val f2 = ZIO.succeed(2) *> ZIO.fail("Oh uh!")
  val f3 = ZIO.succeed(3).debug
  val f4 = ZIO.succeed(4) *> ZIO.fail("Oh error!")
  val f5 = ZIO.succeed(5).debug

  val myApp: ZIO[Any, String, ((((Int, Int), Int), Int), Int)] =
    f1 validatePar f2 validatePar f3 validatePar f4 validatePar f5

  def run = myApp.cause.map(_.untraced).debug.uncause
}

// One possible output:
// 3
// 1
// 5
// Both(Fail(Oh uh!,Trace(None,Chunk())),Fail(Oh error!,Trace(None,Chunk())))
// timestamp=2022-03-14T09:16:00.670444190Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
//  	at <empty>.MainApp.run(MainApp.scala:13)
// Exception in thread "zio-fiber-2" java.lang.String: Oh error!
//  	at <empty>.MainApp.run(MainApp.scala:13)"
```

In addition, it has a `ZIO#validateWith` variant, which is useful for providing combiner function (`f: (A, B) => C`) to combine pair values.

## `ZIO.validate`

This operator is very similar to the `ZIO.foreach` operator. It transforms all elements of a collection using the provided effectful operation, but it collects all errors in the error channel, as well as the success values in the success channel.

It is similar to the `ZIO.partition` but it is an exceptional operator which means it collects errors in the error channel and success in the success channel:

```scala
object ZIO {
  def validate[R, E, A, B](in: Collection[A])(
    f: A => ZIO[R, E, B]
  ): ZIO[R, ::[E], Collection[B]]

  def validate[R, E, A, B](in: NonEmptyChunk[A])(
    f: A => ZIO[R, E, B]
  ): ZIO[R, ::[E], NonEmptyChunk[B]]
}
```

Another difference is that this operator is lossy, which means if there are errors all successes will be lost.

In the lossy scenario, it will collect all errors in the error channel, which cause the failure:

```scala

object MainApp extends ZIOAppDefault {
  val res: ZIO[Any, ::[String], List[Int]] =
    ZIO.validate(List.range(1, 7)){ n =>
      if (n < 5)
        ZIO.succeed(n)
      else
        ZIO.fail(s"$n is not less that 5")
    }
  def run = res.debug
}

// Output:
// <FAIL> List(5 is not less that 5, 6 is not less that 5)
// timestamp=2022-03-12T07:34:36.510227783Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" scala.collection.immutable.$colon$colon: List(5 is not less that 5, 6 is not less that 5)
//	at <empty>.MainApp.res(MainApp.scala:5)
//	at <empty>.MainApp.run(MainApp.scala:11)"
```

In the success scenario when we have no errors at all, all the successes will be collected in the success channel:

```scala

object MainApp extends ZIOAppDefault {
  val res: ZIO[Any, ::[String], List[Int]] =
    ZIO.validate(List.range(1, 4)){ n =>
      if (n < 5)
        ZIO.succeed(n)
      else
        ZIO.fail(s"$n is not less that 5")
    }
  def run = res.debug
}

// Ouput:
// List(1, 2, 3)
```

Two more notes:

1. The `ZIO.validate` operator is sequential, so we can use the `ZIO.validatePar` version to do the computation in parallel.
2. The `ZIO.validateDiscard` and `ZIO.validateParDiscard` operators are mostly similar to their non-discard versions, except they discard the successes. So the type of the success channel will be `Unit`.

## `ZIO.validateFirst`

Like the `ZIO.validate` in the success scenario, it will collect all errors in the error channel except in the success scenario it will return only the first success:

```scala
object ZIO {
  def validateFirst[R, E, A, B](in: Collection[A])(
    f: A => ZIO[R, E, B]
  ): ZIO[R, Collection[E], B]
}
```

In the failure scenario, it will collect all errors in the failure channel, and it causes the failure:

```scala

object MainApp extends ZIOAppDefault {
  val res: ZIO[Any, List[String], Int] =
    ZIO.validateFirst(List.range(5, 10)) { n =>
      if (n < 5)
        ZIO.succeed(n)
      else
        ZIO.fail(s"$n is not less that 5")
    }
  def run = res.debug
}
// Output:
// <FAIL> List(5 is not less that 5, 6 is not less that 5, 7 is not less that 5, 8 is not less that 5, 9 is not less that 5)
// timestamp=2022-03-12T07:50:15.632883494Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" scala.collection.immutable.$colon$colon: List(5 is not less that 5, 6 is not less that 5, 7 is not less that 5, 8 is not less that 5, 9 is not less that 5)
// 	at <empty>.MainApp.res(MainApp.scala:5)
//	at <empty>.MainApp.run(MainApp.scala:11)"
```

In the success scenario it will return the first success value:

```scala

object MainApp extends ZIOAppDefault {
  val res: ZIO[Any, List[String], Int] =
    ZIO.validateFirst(List.range(1, 4)) { n =>
      if (n < 5)
        ZIO.succeed(n)
      else
        ZIO.fail(s"$n is not less that 5")
    }
  def run = res.debug
}

// Output:
// 1
```

## `ZIO.partition`

The partition operator takes an iterable and effectful function that transforms each value of the iterable and finally creates a tuple of both failures and successes in the success channel.

```scala
object ZIO {
  def partition[R, E, A, B](in: => Iterable[A])(
    f: A => ZIO[R, E, B]
  ): ZIO[R, Nothing, (Iterable[E], Iterable[B])]
}
```

Note that this operator is an unexceptional effect, which means the type of the error channel is `Nothing`. So using this operator, if we reach a failure case, the whole effect doesn't fail. This is similar to the `List#partition` in the standard library:

Let's try an example of collecting even numbers from the range of 0 to 7:

```scala

val res: ZIO[Any, Nothing, (Iterable[String], Iterable[Int])] =
  ZIO.partition(List.range(0, 7)){ n =>
    if (n % 2 == 0)
      ZIO.succeed(n)
    else
      ZIO.fail(s"$n is not even")
  }
res.debug

// Output:
// (List(1 is not even, 3 is not even, 5 is not even),List(0, 2, 4, 6))
```

---

## Examples(Error-management)

Let's write an application that takes numerator and denominator from the user and then print the result back to the user:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      a <- readNumber("Enter the first number  (a): ")
      b <- readNumber("Enter the second number (b): ")
      r <- divide(a, b)
      _ <- Console.printLine(s"a / b: $r")
    } yield ()

  def readNumber(msg: String): ZIO[Any, IOException, Int] =
    Console.print(msg) *> Console.readLine.map(_.toInt)

  def divide(a: Int, b: Int): ZIO[Any, Nothing, Int] =
    if (b == 0)
      ZIO.die(new ArithmeticException("divide by zero")) // unexpected error
    else
      ZIO.succeed(a / b)
}
```

Now let's try to enter the zero for the second number and see what happens:

```scala
Please enter the first number  (a): 5
Please enter the second number (b): 0
timestamp=2022-02-14T09:39:53.981143209Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.ArithmeticException: divide by zero
at MainApp$.$anonfun$divide$1(MainApp.scala:16)
at zio.ZIO$.$anonfun$die$1(ZIO.scala:3384)
at zio.internal.FiberContext.runUntil(FiberContext.scala:255)
at zio.internal.FiberContext.run(FiberContext.scala:115)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:831)
at <empty>.MainApp.divide(MainApp.scala:16)"
```

As we see, because we entered the zero for the denominator, the `ArithmeticException` defect, makes the application crash.

Defects are any _unexpected errors_ that we are not going to handle. They will propagate through our application stack until they crash the whole.

Defects have many roots, most of them are from a programming error. Errors will happen when we haven't written the application with best practices. For example, one of these practices is that we should validate the inputs before providing them to the `divide` function. So if the user entered the zero as the denominator, we can retry and ask the user to return another number:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      a <- readNumber("Enter the first number  (a): ")
      b <- readNumber("Enter the second number (b): ").repeatUntil(_ != 0)
      r <- divide(a, b)
      _ <- Console.printLine(s"a / b: $r")
    } yield ()

  def readNumber(msg: String): ZIO[Any, IOException, Int] =
    Console.print(msg) *> Console.readLine.map(_.toInt)

  def divide(a: Int, b: Int): ZIO[Any, Nothing, Int] = ZIO.succeed(a / b)
}
```

Another note about defects is that they are invisible, and they are not typed. We cannot expect what defects will happen by observing the typed error channel. In the above example, when we run the application and enter noninteger input, another defect, which is called `NumberFormatException` will crash the application:

```scala
Enter the first number  (a): five
timestamp=2022-02-18T06:36:25.984665171Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.NumberFormatException: For input string: "five"
	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
	at java.base/java.lang.Integer.parseInt(Integer.java:660)
	at java.base/java.lang.Integer.parseInt(Integer.java:778)
	at scala.collection.StringOps$.toInt$extension(StringOps.scala:910)
	at MainApp$.$anonfun$readNumber$3(MainApp.scala:16)
	at MainApp$.$anonfun$readNumber$3$adapted(MainApp.scala:16)
  ...
	at <empty>.MainApp.run(MainApp.scala:9)"
```

The cause of this defect is also a programming error, which means we haven't validated input when parsing it. So let's try to validate the input, and make sure that it is a number. We know that if the entered input does not contain a parsable `Int` the `String#toInt` throws the `NumberFormatException` exception. As we want this exception to be typed, we import the `String#toInt` function using the `ZIO.attempt` constructor. Using this constructor the function signature would be as follows:

```scala

def parseInput(input: String): ZIO[Any, Throwable, Int] =
  ZIO.attempt(input.toInt)
```

Since the `NumberFormatException` is an expected error, and we want to handle it. So we type the error channel as `NumberFormatException`.

To be more specific, we would like to narrow down the error channel to the `NumberFormatException`, so we can use the `refineToOrDie` operator:

```scala

def parseInput(input: String): ZIO[Any, NumberFormatException, Int] =
  ZIO.attempt(input.toInt)                 // ZIO[Any, Throwable, Int]
    .refineToOrDie[NumberFormatException]  // ZIO[Any, NumberFormatException, Int]
```

The same result can be achieved by succeeding the `String#toInt` and then widening the error channel using the `ZIO#unrefineTo` operator:

```scala

def parseInput(input: String): ZIO[Any, NumberFormatException, Int] =
  ZIO.succeed(input.toInt)                 // ZIO[Any, Nothing, Int]
    .unrefineTo[NumberFormatException]     // ZIO[Any, NumberFormatException, Int]
```

Now, let's refactor the example with recent changes:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      a <- readNumber("Enter the first number  (a): ")
      b <- readNumber("Enter the second number (b): ").repeatUntil(_ != 0)
      r <- divide(a, b)
      _ <- Console.printLine(s"a / b: $r")
    } yield ()

  def parseInput(input: String): ZIO[Any, NumberFormatException, Int] =
    ZIO.attempt(input.toInt).refineToOrDie[NumberFormatException]

  def readNumber(msg: String): ZIO[Any, IOException, Int] =
    (Console.print(msg) *> Console.readLine.flatMap(parseInput))
      .retryUntil(!_.isInstanceOf[NumberFormatException])
      .refineToOrDie[IOException]

  def divide(a: Int, b: Int): ZIO[Any, Nothing, Int] = ZIO.succeed(a / b)
}
```

---

## Exceptional and Unexceptional Effects

Besides the `IO` type alias, ZIO has four different type aliases which can be categorized into two different categories:

- **Exceptional Effect**— `Task` and `RIO` are two effects whose error parameter is fixed to `Throwable`, so we call them exceptional effects.
- **Unexceptional Effect**— `UIO` and `URIO` have error parameters that are fixed to `Nothing`, indicating that they are unexceptional effects. So they can't fail, and the compiler knows about it.

So when we compose different effects together, at any point of the codebase we can determine this piece of code can fail or cannot. As a result, typed errors offer a compile-time transition point between this can fail and this can't fail.

For example, the `ZIO.acquireReleaseWith` API asks us to provide three different inputs: _acquire_, _release_, and _use_. The `release` parameter requires a function from `A` to `URIO[R, Any]`. So, if we put an exceptional effect, it will not compile:

```scala
object ZIO {
  def acquireReleaseWith[R, E, A, B](
    acquire: => ZIO[R, E, A],
    release: A => URIO[R, Any],
    use: A => ZIO[R, E, B]
  ): ZIO[R, E, B]
}
```

---

## Expected and Unexpected Errors

Inside an application, there are two distinct categories of errors:
- **Expected errors** are those that are expected to occur, and we tend to recover them. They are also known as _recoverable errors_ or _declared errors_.
- **Unexpected errors** are those that are not expected to occur, and they are not recoverable. They are also known as _non-recoverable errors_ or _defects_. 

## Expected Errors

Expected errors are those errors in which we expected them to happen in normal circumstances, and we can't prevent them. They can be predicted upfront, and we can plan for them. We know when, where, and why they occur. So we know when, where, and how to handle these errors. By handling them we can recover from the failure, this is why we say they are _recoverable errors_. All domain errors, business errors are expected once because we talk about them in workflows and user stories, so we know about them in the context of business flows.

For example, when accessing an external database, that database might be down for some short period of time, so we retry to connect again, or after some number of attempts, we might decide to use an alternative solution, e.g. using an in-memory database.

## Unexpected Errors

We know there is a category of things that we are not going to expect and plan for. These are the things we don't expect but of course, we know they are going to happen. We don't know what is the exact root of these errors at runtime, so we have no idea how to handle them. They are actually going to bring down our production application, and then we have to figure out what went wrong to fix them.

For example, the corrupted database file will cause an unexpected error. We can't handle that in runtime. It may be necessary to shut down the whole application in order to prevent further damage.

Most of the unexpected errors are rooted in programming errors. This means, we have just tested the _happy path_, so in case of _unhappy path_ we encounter a defect. When we have defects in our code we have no way of knowing about them otherwise we investigate, test, and fix them.

One of the common programming errors is forgetting to validate unexpected errors that may occur when we expect an input but the input is not valid, while we haven't validated the input. When the user inputs the invalid data, we might encounter the divide by zero exception or might corrupt our service state or a cause similar defect. 

These kinds of defects are common when we upgrade our service with the new data model for its input, while one of the other services is not upgraded with the new data contract and is calling our service with the deprecated data model. If we haven't a validation phase, they will cause defects!

Another example of defects is _memory errors_ like _buffer overflows_, _stack overflows_, _out-of-memory_, _invalid access to null pointers_, and so forth. Most of the time these unexpected errors are occurs when we haven't written a memory-safe and resource-safe program, or they might occur due to hardware issues or uncontrollable external problems. We as a developer don't know how to cope with these types of errors at runtime. We should investigate to find the exact root cause of these defects.

As we cannot handle unexpected errors, we should instead log them with their respective stack traces and contextual information. So later we could investigate the problem and try to fix them. The best we can do with unexpected errors is to _sandbox_ them to limit the damage that they do to the overall application. For example, an unexpected error in browser extension shouldn't crash the whole browser.

## Best Practices

So the best practice for each of these errors is as follows:

### Expected Errors

we handle expected errors with the aid of the Scala compiler, by pushing them into the type system. In ZIO there is the error type parameter called `E`, and this error type parameter is for modeling all the expected errors in the application.

A ZIO value has a type parameter `E` which is the type of _declared errors_ it can fail with. `E` only covers the errors which were specified at the outset. The same ZIO value could still throw exceptions in unforeseen ways. These unforeseen situations are called _defects_ in a ZIO program, and they lie outside `E`.

Bringing abnormal situations from the domain of defects into that of `E` enables the compiler to help us keep a tab on error conditions throughout the application, at compile time. This helps ensure the handling of domain errors in domain-specific ways.

### Unexpected Errors

We encode unexpected errors by not reflecting them to the type system because there is no way we could do it, and it wouldn't provide any value if we could. At best as we can, we simply sandbox that to some well-defined area of the application.

Note that _defects_, can creep silently to higher levels in our application, and, if they get triggered at all, their handling might eventually be in more general ways.

So for ZIO, expected errors are reflected in the type of the ZIO effect, whereas unexpected errors are not so reflective, and that is the distinction.

That is the best practice. It helps us write better code. The code that we can reason about its error properties and potential expected errors. We can look at the ZIO effect and know how it is supposed to fail.

## Conclusion

1. Unexpected errors are impossible to recover, and they will eventually shut down the application but expected errors can be recovered by handling them.
2. We do not type unexpected errors, but we type expected errors either explicitly or using general `Throwable` error type.
3. Unexpected errors mostly is a sign of programming errors, but expected errors part of domain errors.
4. Even though we haven't any clue on how to handle defects, we might still need to do some operation, before letting them crash the application. So in such a situation, we can [catch defects](recovering/catching.md) do following operations, and then rethrow them again:
  - logging the defect to a log aggregator
  - sending an email to alert developers
  - displaying a nice "unexpected error" message to the user
  - etc.

---

## Introduction to Error Management in ZIO

As well as providing first-class support for typed errors, ZIO has a variety of facilities for catching, propagating, and transforming errors in a typesafe manner. In this section, we will learn about different types of errors in ZIO and how we can manage them.

---

## Chaining Effects Based on Errors

Unlike `ZIO#flatMap` the `ZIO#flatMapError` combinator chains two effects, where the second effect is dependent on the error channel of the first effect:

```scala
trait ZIO[-R, +E, +A] {
  def flatMapError[R1 <: R, E2](
    f: E => ZIO[R1, Nothing, E2]
  ): ZIO[R1, E2, A]
}
```

In the following example, we are trying to find a random prime number between 1000 and 10000. We will use the `ZIO#flatMapError` to collect all errors inside a `Ref` of type `List[String]`:

```scala

object MainApp extends ZIOAppDefault {
  def isPrime(n: Int): Boolean =
    if (n <= 1) false else (2 until n).forall(i => n % i != 0)

  def findPrimeBetween(
      minInclusive: Int,
      maxExclusive: Int
  ): ZIO[Any, List[String], Int] =
    for {
      errors <- Ref.make(List.empty[String])
      number <- Random
        .nextIntBetween(minInclusive, maxExclusive)
        .reject {
          case n if !isPrime(n) =>
            s"non-prime number rejected: $n"
        }
        .flatMapError(error => errors.updateAndGet(_ :+ error))
        .retryUntil(_.length >= 5)
    } yield number

  val myApp: ZIO[Any, Nothing, Unit] =
    findPrimeBetween(1000, 10000)
      .flatMap(prime => Console.printLine(s"found a prime number: $prime").orDie)
      .catchAll { (errors: List[String]) =>
        Console.printLine(
          s"failed to find a prime number after 5 attempts:\n  ${errors.mkString("\n  ")}"
        )
      }
      .orDie

  def run = myApp
}
```

---

## Converting Defects to Failures

Both `ZIO#resurrect` and `ZIO#absorb` are symmetrical opposite of the `ZIO#orDie` operator. The `ZIO#orDie` takes failures from the error channel and converts them into defects, whereas the `ZIO#absorb` and `ZIO#resurrect` take defects and convert them into failures:

```scala
trait ZIO[-R, +E, +A] {
  def absorb(implicit ev: E IsSubtypeOfError Throwable): ZIO[R, Throwable, A]
  def absorbWith(f: E => Throwable): ZIO[R, Throwable, A]
  def resurrect(implicit ev1: E IsSubtypeOfError Throwable): ZIO[R, Throwable, A]
}
```

Below are examples of the `ZIO#absorb` and `ZIO#resurrect` operators:

```scala

val effect1 =
  ZIO.fail(new IllegalArgumentException("wrong argument"))  // ZIO[Any, IllegalArgumentException, Nothing]
    .orDie                                                  // ZIO[Any, Nothing, Nothing]
    .absorb                                                 // ZIO[Any, Throwable, Nothing]
    .refineToOrDie[IllegalArgumentException]                // ZIO[Any, IllegalArgumentException, Nothing]

val effect2 =
  ZIO.fail(new IllegalArgumentException("wrong argument"))  // ZIO[Any, IllegalArgumentException , Nothing]
    .orDie                                                  // ZIO[Any, Nothing, Nothing]
    .resurrect                                              // ZIO[Any, Throwable, Nothing]
    .refineToOrDie[IllegalArgumentException]                // ZIO[Any, IllegalArgumentException, Nothing]
```

So what is the difference between `ZIO#absorb` and `ZIO#resurrect` operators?

The `ZIO#absorb` can recover from both `Die` and `Interruption` causes. Using this operator we can absorb failures, defects and interruptions using `ZIO#absorb` operation. It attempts to convert all causes into a failure, throwing away all information about the cause of the error:

```scala

object MainApp extends ZIOAppDefault {
  val effect1 =
    ZIO.dieMessage("Boom!") // ZIO[Any, Nothing, Nothing]
      .absorb               // ZIO[Any, Throwable, Nothing]
      .ignore
  val effect2 =
    ZIO.interrupt           // ZIO[Any, Nothing, Nothing]
      .absorb               // ZIO[Any, Throwable, Nothing]
      .ignore

  def run =
    (effect1 <*> effect2)
      .debug("application exited successfully")
}
```

The output would be as below:

```scala
application exited successfully: ()
```

Whereas, the `ZIO#resurrect` will only recover from `Die` causes:

```scala

object MainApp extends ZIOAppDefault {
  val effect1 =
    ZIO
      .dieMessage("Boom!") // ZIO[Any, Nothing, Nothing]
      .resurrect           // ZIO[Any, Throwable, Nothing]
      .ignore
  val effect2 =
    ZIO.interrupt          // ZIO[Any, Nothing, Nothing]
      .resurrect           // ZIO[Any, Throwable, Nothing]
      .ignore

  def run =
    (effect1 <*> effect2)
      .debug("couldn't recover from fiber interruption")
}
```

And, here is the output:

```scala
timestamp=2022-02-18T14:21:52.559872464Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.InterruptedException: Interrupted by thread "zio-fiber-"
	at <empty>.MainApp.effect2(MainApp.scala:10)
	at <empty>.MainApp.effect2(MainApp.scala:11)
	at <empty>.MainApp.effect2(MainApp.scala:12)
	at <empty>.MainApp.run(MainApp.scala:15)
	at <empty>.MainApp.run(MainApp.scala:16)"
```

---

## Error Refinement

ZIO has some operators useful for converting defects into failures. So we can take part in non-recoverable errors and convert them into the typed error channel and vice versa.

Note that both `ZIO#refine*` and `ZIO#unrefine*` do not alter the error behavior, but only change the error model. That is to say, if an effect fails or die, then after `ZIO#refine*` or `ZIO#unrefine*`, it will still fail or die; and if an effect succeeds, then after `ZIO#refine*` or `ZIO#unrefine*`, it will still succeed; only the manner in which it signals the error will be altered by these two methods:

1. The `ZIO#refine*` pinches off a piece of _failure_ of type `E`, and converts it into a _defect_.
2. The `ZIO#unrefine*` pinches off a piece of a _defect_, and converts it into a _failure_ of type `E`.

## Refining

### `ZIO#refineToOrDie`

This operator **narrows** down the type of the error channel from `E` to the `E1`. It leaves the rest errors untyped, so everything that doesn't fit is turned into a defect. So it makes the error space **smaller**.

```scala
ZIO[-R, +E, +A] {
  def refineToOrDie[E1 <: E]: ZIO[R, E1, A]
}
```

In the following example, we are going to implement `parseInt` by importing `String#toInt` code from the standard scala library using `ZIO#attempt` and then refining the error channel from `Throwable` to the `NumberFormatException` error type:

```scala

def parseInt(input: String): ZIO[Any, NumberFormatException, Int] =
  ZIO.attempt(input.toInt)                 // ZIO[Any, Throwable, Int]
    .refineToOrDie[NumberFormatException]  // ZIO[Any, NumberFormatException, Int]
```

In this example, if the `input.toInt` throws any other exceptions other than `NumberFormatException`, e.g. `IndexOutOfBoundsException`, will be translated to the ZIO defect.

### `ZIO#refineOrDie`

It is the more powerful version of the previous operator. Instead of refining to one specific error type, we can refine to multiple error types using a partial function:

```scala
trait ZIO[-R, +E, +A] {
  def refineOrDie[E1](pf: PartialFunction[E, E1]): ZIO[R, E1, A]
}
```

In the following example, we excluded the `Baz` exception from recoverable errors, so it will be converted to a defect. In another word, we narrowed `DomainError` down to just `Foo` and `Bar` errors:

```scala

sealed abstract class DomainError(msg: String)
  extends Exception(msg)
    with Serializable
    with Product
case class Foo(msg: String) extends DomainError(msg)
case class Bar(msg: String) extends DomainError(msg)
case class Baz(msg: String) extends DomainError(msg)

object MainApp extends ZIOAppDefault {
  val effect: ZIO[Any, DomainError, Unit] =
    ZIO.fail(Baz("Oh uh!"))

  val refined: ZIO[Any, DomainError, Unit] =
    effect.refineOrDie {
      case foo: Foo => foo
      case bar: Bar => bar
    }

  def run = refined.catchAll(_ => ZIO.unit).debug
}
```

### `ZIO#refineOrDieWith`

In the two previous refine combinators, we were dealing with exceptional effects whose error channel type was `Throwable` or a subtype of that. The `ZIO#refineOrDieWith` operator is a more powerful version of refining operators. It can work with any exceptional effect whether they are `Throwable` or not. When we narrow down the failure space, some failures become defects. To convert those failures to defects, it takes a function from `E` to `Throwable`:

```scala
trait ZIO[-R, +E, +A] {
  def refineOrDieWith[E1](pf: PartialFunction[E, E1])(f: E => Throwable): ZIO[R, E1, A]
}
```

In the following example, we excluded the `BazError` from recoverable errors, so it will be converted to a defect. In another word, we narrowed the whole space of `String` errors down to just "FooError" and "BarError":

```scala

object MainApp extends ZIOAppDefault {
  def effect(i: String): ZIO[Any, String, Nothing] = {
    if (i == "foo") ZIO.fail("FooError")
    else if (i == "bar") ZIO.fail("BarError")
    else ZIO.fail("BazError")
  }

  val refined: ZIO[Any, String, Nothing] =
    effect("baz").refineOrDieWith {
      case "FooError" | "BarError" => "Oh Uh!"
    }(e => new Throwable(e))

  def run = refined.catchAll(_ => ZIO.unit)
}
```

## Unrefining

### `ZIO#unrefineTo[E1 >: E]`

This operator **broadens** the type of the error channel from `E` to the `E1` and embeds some defects into it. So it is going from some fiber failures back to errors and thus making the error type **larger**:

```scala
trait ZIO[-R, +E, +A] {
  def unrefineTo[E1 >: E]: ZIO[R, E1, A]
}
```

In the following example, we are going to implement `parseInt` by importing `String#toInt` code from the standard scala library using `ZIO#succeed` and then unrefining the error channel from `Nothing` to the `NumberFormatException` error type:

```scala

def parseInt(input: String): ZIO[Any, NumberFormatException, Int] =
  ZIO.succeed(input.toInt)              // ZIO[Any, Nothing, Int]
    .unrefineTo[NumberFormatException]  // ZIO[Any, NumberFormatException, Int]
```

### `ZIO#unrefine`

It is a more powerful version of the previous operator. It takes a partial function from `Throwable` to `E1` and converts those defects to recoverable errors:

```scala
trait ZIO[-R, +E, +A] {
  def unrefine[E1 >: E](pf: PartialFunction[Throwable, E1]): ZIO[R, E1, A]
}
```

```scala

case class Foo(msg: String) extends Throwable(msg)
case class Bar(msg: String) extends Throwable(msg)
case class Baz(msg: String) extends Throwable(msg)

object MainApp extends ZIOAppDefault {
  def unsafeOpThatMayThrows(i: String): String =
    if (i == "foo")
      throw Foo("Oh uh!")
    else if (i == "bar")
      throw Bar("Oh Error!")
    else if (i == "baz")
      throw Baz("Oh no!")
    else i

  def effect(i: String): ZIO[Any, Nothing, String] =
    ZIO.succeed(unsafeOpThatMayThrows(i))

  val unrefined: ZIO[Any, Foo, String] =
    effect("foo").unrefine { case e: Foo => e }

  def run = unrefined.catchAll(_ => ZIO.unit)
}
```

Using `ZIO#unrefine` we can have more control to unrefine a ZIO effect that may die because of some defects, for example in the following example we are going to convert both `Foo` and `Bar` defects to recoverable errors and remain `Baz` unrecoverable:

```scala
val unrefined: ZIO[Any, Throwable, String] =
  effect("foo").unrefine {
    case e: Foo => e
    case e: Bar => e
  }
```

### `ZIO#unrefineWith`

This is the most powerful version of unrefine operators. It takes a partial function, as the previous operator, and then tries to broaden the failure space by converting some of the defects to typed recoverable errors. If it doesn't find any defect, it will apply the `f` which is a function from `E` to `E1`, and map all typed errors using this function:

```scala
trait ZIO[-R, +E, +A] {
  def unrefineWith[E1](pf: PartialFunction[Throwable, E1])(f: E => E1): ZIO[R, E1, A]
}
```

```scala

object MainApp extends ZIOAppDefault {
  case class Foo(msg: String) extends Exception(msg)
  case class Bar(msg: String) extends Exception(msg)

  val effect: ZIO[Any, Foo, Nothing] =
    ZIO.ifZIO(Random.nextBoolean)(
      onTrue = ZIO.fail(Foo("Oh uh!")),
      onFalse = ZIO.die(Bar("Boom!"))
    )

  val unrefined: ZIO[Any, String, Nothing] =
    effect
      .unrefineWith {
        case e: Bar => e.getMessage
      }(e => e.getMessage)

  def run = unrefined.cause.debug
}
```

---

## Exposing Errors in The Success Channel

Before taking into `ZIO#either` and `ZIO#absolve`, let's see their signature:

```scala
trait ZIO[-R, +E, +A] {
  def either(implicit ev: CanFail[E]): URIO[R, Either[E, A]]
  def absolve[E1 >: E, B](implicit ev: A IsSubtypeOfOutput Either[E1, B]): ZIO[R, E1, B]
}
```

Before continuing, let's take a look again at the `validate` function we have written earlier:

```scala

sealed trait AgeValidationException extends Exception
case class NegativeAgeException(age: Int) extends AgeValidationException
case class IllegalAgeException(age: Int)  extends AgeValidationException

def validate(age: Int): ZIO[Any, AgeValidationException, Int] =
  if (age < 0)
    ZIO.fail(NegativeAgeException(age))
  else if (age < 18)
    ZIO.fail(IllegalAgeException(age))
  else ZIO.succeed(age)
```

Now we are ready to use `ZIO#either` and `ZIO#absolve` operations:

## `ZIO#either`

The `ZIO#either` convert a `ZIO[R, E, A]` effect to another effect in which its failure (`E`) and success (`A`) channel have been lifted into an `Either[E, A]` data type as success channel of the `ZIO` data type:

```scala

val age: Int = ???

val res: URIO[Any, Either[AgeValidationException, Int]] = validate(age).either
```

The resulting effect is an unexceptional effect and cannot fail, because the failure case has been exposed as part of the `Either` success case. The error parameter of the returned `ZIO` is `Nothing`, since it is guaranteed the `ZIO` effect does not model failure.

This method is useful for recovering from `ZIO` effects that may fail:

```scala

val myApp: ZIO[Any, IOException, Unit] =
  for {
    _ <- Console.print("Please enter your age: ")
    age <- Console.readLine.map(_.toInt)
    res <- validate(age).either
    _ <- res match {
      case Left(error) => ZIO.debug(s"validation failed: $error")
      case Right(age) => ZIO.debug(s"The $age validated!")
    }
  } yield ()
```

## `ZIO#absolve`/`ZIO.absolve`

The `ZIO#abolve` operator and the `ZIO.absolve` constructor perform the inverse. They submerge the error case of an `Either` into the `ZIO`:

```scala

val age: Int = ???
validate(age) // ZIO[Any, AgeValidationException, Int]
  .either     // ZIO[Any, Either[AgeValidationException, Int]]
  .absolve    // ZIO[Any, AgeValidationException, Int]
```

Here is another example:

```scala

def sqrt(input: ZIO[Any, Nothing, Double]): ZIO[Any, String, Double] =
  ZIO.absolve(
    input.map { value =>
      if (value < 0.0)
        Left("Value must be >= 0.0")
      else
        Right(Math.sqrt(value))
    }
  )
```

---

## Exposing the Cause in The Success Channel

Using the `ZIO#cause` operation we can expose the cause, and then by using `ZIO#uncause` we can reverse this operation:

```scala
trait ZIO[-R, +E, +A] {
  def cause: URIO[R, Cause[E]]
  def uncause[E1 >: E](implicit ev: A IsSubtypeOfOutput Cause[E1]): ZIO[R, E1, Unit]
}
```

In the following example, we expose and then untrace the underlying cause:

```scala

object MainApp extends ZIOAppDefault {
  val f1: ZIO[Any, String, Int] =
    ZIO.fail("Oh uh!").as(1)

  val f2: ZIO[Any, String, Int] =
    ZIO.fail("Oh error!").as(2)

  val myApp: ZIO[Any, String, (Int, Int)] = f1 zipPar f2

  def run = myApp.cause.map(_.untraced).debug
}
```

Sometimes the [`ZIO#mapErrorCause`](map-operations.md#ziomaperrorziomaperrorcause) operator is a better choice when we just want to map the underlying cause without exposing the cause.

---

## Filtering the Success Channel

ZIO has a variety of operators that can filter values on the success channel based on a given predicate, and if the predicate fails, we can use different strategies:

- Failing the original effect (`ZIO#filterOrFail`)
- Dying the original effect (`ZIO#filterOrDie` and `ZIO#filterOrDieMessage`)
- Running an alternative ZIO effect (`ZIO#filterOrElse` and `ZIO#filterOrElseWith`)

```scala

def getNumber: ZIO[Any, Nothing, Int] =
  (Console.print("Please enter a non-negative number: ") *>
    Console.readLine.mapAttempt(_.toInt))
    .retryUntil(!_.isInstanceOf[NumberFormatException]).orDie

val r1: ZIO[Any, String, Int] =
  Random.nextInt.filterOrFail(_ >= 0)("random number is negative")

val r2: ZIO[Any, Nothing, Int] =
  Random.nextInt.filterOrDie(_ >= 0)(
    new IllegalArgumentException("random number is negative")
  )

val r3: ZIO[Any, Nothing, Int] =
  Random.nextInt.filterOrDieMessage(_ >= 0)("random number is negative")

val r4: ZIO[Any, Nothing, Int] =
  Random.nextInt.filterOrElse(_ >= 0)(getNumber)

val r5: ZIO[Any, Nothing, Int] =
  Random.nextInt.filterOrElseWith(_ >= 0)(x => ZIO.succeed(-x))
```

---

## Flattening Optional Error Types

If we have an optional error of type `E` in the error channel, we can flatten it to the `E` type using the `ZIO#flattenErrorOption` operator:

```scala

def parseInt(input: String): ZIO[Any, Option[String], Int] =
  if (input.isEmpty)
    ZIO.fail(Some("empty input"))
  else
    try {
      ZIO.succeed(input.toInt)
    } catch {
      case _: NumberFormatException => ZIO.fail(None)
    }

def flattenedParseInt(input: String): ZIO[Any, String, Int] =
  parseInt(input).flattenErrorOption("non-numeric input")

val r1: ZIO[Any, String, Int] = flattenedParseInt("zero")
val r2: ZIO[Any, String, Int] = flattenedParseInt("")
val r3: ZIO[Any, String, Int] = flattenedParseInt("123")
```

---

## Flipping Error and Success Channels

Sometimes, we would like to apply some methods on the error channel which are specific for the success channel, or we want to apply some methods on the success channel which are specific for the error channel. Therefore, we can flip the error and success channel and before flipping back, we can perform the right operator on flipped channels:

```scala
trait ZIO[-R, +E, +A] {
  def flip: ZIO[R, A, E]
  def flipWith[R1, A1, E1](f: ZIO[R, A, E] => ZIO[R1, A1, E1]): ZIO[R1, E1, A1]
}
```

Assume we have the following example:

```scala

val evens: ZIO[Any, List[String], List[Int]] =
  ZIO.validate(List(1, 2, 3, 4, 5)) { n =>
    if (n % 2 == 0)
      ZIO.succeed(n)
    else
      ZIO.fail(s"$n is not even")
  }
```

We want to reverse the order of errors. In order to do that instead of using `ZIO#mapError`, we can map the error channel by using flip operators:

```scala

val r1: ZIO[Any, List[String], List[Int]] = evens.mapError(_.reverse)
val r2: ZIO[Any, List[String], List[Int]] = evens.flip.map(_.reverse).flip
val r3: ZIO[Any, List[String], List[Int]] = evens.flipWith(_.map(_.reverse))
```

---

## Map Operations

Other than `ZIO#map` and `ZIO#flatMap`, ZIO has several other operators to manage errors while mapping:

## `ZIO#mapError`/`ZIO#mapErrorCause`

Let's begin with `ZIO#mapError` and `ZIO#mapErrorCause`. These operators help us to access the error channel as a raw error value or as a type of `Cause` and map their values:

```scala
trait ZIO[-R, +E, +A] {
  def mapError[E2](f: E => E2): ZIO[R, E2, A]
  def mapErrorCause[E2](h: Cause[E] => Cause[E2]): ZIO[R, E2, A]
}
```

Here are two simple examples for these operators:

```scala

def parseInt(input: String): ZIO[Any, NumberFormatException, Int] = ???

// mapping the error of the original effect to its message
val r1: ZIO[Any, String, Int] =
  parseInt("five")                // ZIO[Any, NumberFormatException, Int]
    .mapError(e => e.getMessage)  // ZIO[Any, String, Int]

// mapping the cause of the original effect to be untraced
val r2 = parseInt("five")         // ZIO[Any, NumberFormatException, Int]
  .mapErrorCause(_.untraced)      // ZIO[Any, NumberFormatException, Int]
```

> _**Note:**_
>
> Note that mapping over an effect's success or error channel does not change the success or failure of the effect, in the same way that mapping over an `Either` does not change whether the `Either` is `Left` or `Right`.

## `ZIO#mapAttempt`

The `ZIO#mapAttempt` returns an effect whose success is mapped by the specified side-effecting `f` function, translating any thrown exceptions into typed failed effects. So it converts an unchecked exception to a checked one by returning the `RIO` effect.

```scala
  trait ZIO[-R, +E, +A] {
    def map[B](f: A => B): ZIO[R, E, B]
    def mapAttempt[B](f: A => B): ZIO[R, Throwable, B]
  }
```

Using operations that can throw exceptions inside of `ZIO#map` such as `effect.map(_.unsafeOpThatThrows)` will result in a defect (an unexceptional effect that will die). In the following example, when we use the `ZIO#map` operation. So, if the `String#toInt` operation throws `NumberFormatException` it will be converted to a defect:

```scala

val result: ZIO[Any, Nothing, Int] =
  Console.readLine.orDie.map(_.toInt)
```

As a result, when the map operation is unsafe, it may lead to buggy programs that may crash, as shown below:

```scala

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[Any, Nothing, Unit] =
    Console.print("Please enter a number: ").orDie *>
      Console.readLine.orDie
        .map(_.toInt)
        .map(_ % 2 == 0)
        .flatMap {
          case true =>
            Console.printLine("You have entered an even number.").orDie
          case false =>
            Console.printLine("You have entered an odd number.").orDie
        }

  def run = myApp
}
```

Converting literal "five" String to Int by calling `toInt` is a side effecting operation because it will throw `NumberFormatException` exception. So in the previous example, if we enter a non-integer number, e.g. "five", it will die because of a `NumberFormatException` defect:

```scala
Please enter a number: five
timestamp=2022-03-17T14:01:33.323639073Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.NumberFormatException: For input string: "five"
	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
	at java.base/java.lang.Integer.parseInt(Integer.java:660)
	at java.base/java.lang.Integer.parseInt(Integer.java:778)
	at scala.collection.StringOps$.toInt$extension(StringOps.scala:910)
	at MainApp$.$anonfun$myApp$3(MainApp.scala:7)
	at MainApp$.$anonfun$myApp$3$adapted(MainApp.scala:7)
	at zio.ZIO.$anonfun$map$1(ZIO.scala:1168)
	at zio.ZIO$FlatMap.apply(ZIO.scala:6182)
	at zio.ZIO$FlatMap.apply(ZIO.scala:6171)
	at zio.internal.FiberContext.runUntil(FiberContext.scala:885)
	at zio.internal.FiberContext.run(FiberContext.scala:115)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:831)
	at zio.internal.FiberContext.runUntil(FiberContext.scala:538)
	at <empty>.MainApp.myApp(MainApp.scala:8)
	at <empty>.MainApp.myApp(MainApp.scala:9)"
```

We can see that the error channel of `myApp` is typed as `Nothing`, so it's not an exceptional error. If we want typed effects, this behavior is not intended. So instead of `ZIO#map` we can use the `mapAttempt` combinator which is a safe map operator that translates all thrown exceptions into typed exceptional effect.

To prevent converting exceptions to defects, we can use `ZIO#mapAttempt` which converts any exceptions to exceptional effects:

```scala

val result: ZIO[Any, Throwable, Int] =
  Console.readLine.orDie.mapAttempt(_.toInt)
```

Having typed errors helps us to catch errors explicitly and handle them in the right way:

```scala

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[Any, Nothing, Unit] =
    Console.print("Please enter a number: ").orDie *>
      Console.readLine.orDie
        .mapAttempt(_.toInt)
        .map(_ % 2 == 0)
        .flatMap {
          case true =>
            Console.printLine("You have entered an even number.").orDie
          case false =>
            Console.printLine("You have entered an odd number.").orDie
        }.catchAll(_ => myApp)

  def run = myApp
}

// Please enter a number: five
// Please enter a number: 4
// You have entered an even number.
```

## `ZIO#mapBoth`

It takes two map functions, one for the error channel and the other for the success channel, and maps both sides of a ZIO effect:

```scala
trait ZIO[-R, +E, +A] {
  def mapBoth[E2, B](f: E => E2, g: A => B): ZIO[R, E2, B]
}
```

Here is a simple example:

```scala

val result: ZIO[Any, String, Int] =
  Console.readLine.orDie.mapAttempt(_.toInt).mapBoth(
    _ => "non-integer input",
    n => Math.abs(n)
  )
```

---

## Merging the Error Channel into the Success Channel

With `ZIO#merge` we can merge the error channel into the success channel:

```scala

val merged : ZIO[Any, Nothing, String] =
  ZIO.fail("Oh uh!") // ZIO[Any, String, Nothing]
    .merge           // ZIO[Any, Nothing, String]
```

If the error and success channels were of different types, it would choose the supertype of both.

---

## Rejecting Some Success Values

We can reject some success values using the `ZIO#reject` operator:

```scala
trait ZIO[-R, +E, +A] {
  def reject[E1 >: E](pf: PartialFunction[A, E1]): ZIO[R, E1, A]

  def rejectZIO[R1 <: R, E1 >: E](
    pf: PartialFunction[A, ZIO[R1, E1, E1]]
  ): ZIO[R1, E1, A]
}
```

If the `PartialFunction` matches, it will reject that success value and convert that to a failure, otherwise it will continue with the original success value:

```scala

val myApp: ZIO[Any, String, Int] =
  Random
    .nextIntBounded(20)
    .reject {
      case n if n % 2 == 0 => s"even number rejected: $n"
      case 5               => "number 5 was rejected"
    }
    .debug
```

---

## Tapping Errors

Like [tapping for success values](../../core/zio/zio.md#tapping) ZIO has several operators for tapping error values. So we can peek into failures or underlying defects or causes:

```scala
trait ZIO[-R, +E, +A] {
  def tapError[R1 <: R, E1 >: E](f: E => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
  def tapErrorCause[R1 <: R, E1 >: E](f: Cause[E] => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
  def tapErrorTrace[R1 <: R, E1 >: E](f: ((E, Trace)) => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
  def tapDefect[R1 <: R, E1 >: E](f: Cause[Nothing] => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
  def tapBoth[R1 <: R, E1 >: E](f: E => ZIO[R1, E1, Any], g: A => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
  def tapEither[R1 <: R, E1 >: E](f: Either[E, A] => ZIO[R1, E1, Any]): ZIO[R1, E1, A]
}
```

Let's try an example:

```scala

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[Any, NumberFormatException, Int] =
    Console.readLine
      .mapAttempt(_.toInt)
      .refineToOrDie[NumberFormatException]
      .tapError { e =>
        ZIO.debug(s"user entered an invalid input: ${e}").when(e.isInstanceOf[NumberFormatException])
      }

  def run = myApp
}
```

---

## Zooming In on Nested Values

## Option

We can extract a value from a Some using `ZIO#some` and then we can unsome it again using `ZIO#unsome`:

```scala
ZIO.attempt(Option("something")) // ZIO[Any, Throwable, Option[String]]
  .some                          // ZIO[Any, Option[Throwable], String]
  .unsome                        // ZIO[Any, Throwable, Option[String]]
```

## Either

With `Either` ZIO values, we can zoom in or out on the left or right side of an `Either`, as well as we can do the inverse and zoom out:

```scala

val eitherEffect: ZIO[Any, Exception, Either[String, Int]] = ???

eitherEffect // ZIO[Any, Exception, Either[String, Int]]
  .left      // ZIO[Any, Either[Exception, Int], String]
  .unleft    // ZIO[Any, Exception, Either[String, Int]]

eitherEffect // ZIO[Any, Exception, Either[String, Int]]
  .right     // ZIO[Any, Either[String, Exception], Int]
  .unright   // ZIO[Any, Exception, Either[String, Int]]
```

---

## Catching

## Catching Failures

If we want to catch and recover from all _typed error_ and effectfully attempt recovery, we can use the `ZIO#catchAll` operator:

```scala
trait ZIO[-R, +E, +A] {
  def catchAll[R1 <: R, E2, A1 >: A](h: E => ZIO[R1, E2, A1]): ZIO[R1, E2, A1]
}
```

We can recover from all errors while reading a file and then fallback to another operation:

```scala

val z: ZIO[Any, IOException, Array[Byte]] =
  readFile("primary.json").catchAll(_ =>
    readFile("backup.json"))
```

In the callback passed to `ZIO#catchAll`, we may return an effect with a different error type (or perhaps `Nothing`), which will be reflected in the type of effect returned by `ZIO#catchAll`.

When using `ZIO#catchAll` operator, the match cases should be exhaustive. Remember our `validate` function again:

```scala

sealed trait AgeValidationException extends Exception
case class NegativeAgeException(age: Int) extends AgeValidationException
case class IllegalAgeException(age: Int)  extends AgeValidationException

def validate(age: Int): ZIO[Any, AgeValidationException, Int] =
  if (age < 0)
    ZIO.fail(NegativeAgeException(age))
  else if (age < 18)
    ZIO.fail(IllegalAgeException(age))
  else ZIO.succeed(age)
```

In the following example, we covered all the cases for the `catchAll` operator:

```scala

val result: ZIO[Any, Nothing, Int] =
  validate(20)
  .catchAll {
    case NegativeAgeException(age) =>
      ZIO.debug(s"negative age: $age").as(-1)
    case IllegalAgeException(age) =>
      ZIO.debug(s"illegal age: $age").as(-1)
  }
```

If we forget to catch all cases and the match fails, the original **failure** will be lost and replaced by a `MatchError` **defect**:

```scala
object MainApp extends ZIOAppDefault {
  val result: ZIO[Any, Nothing, Int] =
    validate(15)
      .catchAll {
        case NegativeAgeException(age) =>
          ZIO.debug(s"negative age: $age").as(-1)
//        case IllegalAgeException(age) =>
//          ZIO.debug(s"illegal age: $age").as(-1)
      }

  def run = result
}
```

Another important note about `ZIO#catchAll` is that this operator only can recover from _failures_. So it can't recover from defects or fiber interruptions.

Let's try what happens if we `catchAll` on a dying effect:

```scala

object MainApp extends ZIOAppDefault {
  val die: ZIO[Any, String, Nothing] =
    ZIO.dieMessage("Boom!") *> ZIO.fail("Oh uh!")

  def run = die.catchAll(_ => ZIO.unit)
}

// Output:
// timestamp=2022-03-03T11:04:41.209169849Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.RuntimeException: Boom!
// 	at <empty>.MainApp.die(MainApp.scala:6)
//	at <empty>.MainApp.run(MainApp.scala:8)"
```

Also, if we have a fiber interruption, we can't catch that using this operator:

```scala

object MainApp extends ZIOAppDefault {
  val interruptedEffect: ZIO[Any, String, Nothing] =
    ZIO.interrupt *> ZIO.fail("Oh uh!")

  def run = interruptedEffect.catchAll(_ => ZIO.unit)
}

// Output:
// timestamp=2022-03-03T11:10:15.573588420Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.InterruptedException: Interrupted by thread "zio-fiber-"
//	at <empty>.MainApp.die(MainApp.scala:6)
//	at <empty>.MainApp.run(MainApp.scala:8)"
```

If we want to catch and recover from only some types of exceptions and effectfully attempt recovery, we can use the `ZIO#catchSome` method:

```scala
trait ZIO[-R, +E, +A] {
  def catchSome[R1 <: R, E1 >: E, A1 >: A](pf: PartialFunction[E, ZIO[R1, E1, A1]]): ZIO[R1, E1, A1]
}
```

In the following example, we are only catching failure of type `FileNotFoundException`:

```scala

val data: ZIO[Any, IOException, Array[Byte]] =
  readFile("primary.data").catchSome {
    case _ : FileNotFoundException =>
      readFile("backup.data")
  }
```

## Catching Defects

Like catching failures, ZIO has two operators to catch _defects_:

```scala
trait ZIO[-R, +E, +A] {
  def catchAllDefect[R1 <: R, E1 >: E, A1 >: A](h: Throwable => ZIO[R1, E1, A1]): ZIO[R1, E1, A1]

  def catchSomeDefect[R1 <: R, E1 >: E, A1 >: A](pf: PartialFunction[Throwable, ZIO[R1, E1, A1]]): ZIO[R1, E1, A1]
}
```

Let's try the `ZIO#catchAllDefect` operator:

```scala

ZIO.dieMessage("Boom!")
  .catchAllDefect {
    case e: RuntimeException if e.getMessage == "Boom!" =>
      ZIO.debug("Boom! defect caught.")
    case _: NumberFormatException =>
      ZIO.debug("NumberFormatException defect caught.")
    case _ =>
      ZIO.debug("Unknown defect caught.")
  }
```

We should note that using these operators, we can only recover from a dying effect, and it cannot recover from a failure or fiber interruption.

A defect is an error that cannot be anticipated in advance, and there is no way to respond to it. Our rule of thumb is to not recover defects since we don't know about them. We let them crash the application. Although, in some cases, we might need to reload a part of the application instead of killing the entire application.

Assume we have written an application that can load plugins at runtime. During the runtime of the plugins, if a defect occurs, we don't want to crash the entire application; rather, we log all defects and then reload the plugin.

## Catching Causes

So far, we have only studied how to catch _failures_ and _defects_. But what about _fiber interruptions_ or how about the specific combination of these errors?

There are two ZIO operators useful for catching causes:

```scala
trait ZIO[-R, +E, +A] {
  def catchAllCause[R1 <: R, E2, A1 >: A](h: Cause[E] => ZIO[R1, E2, A1]): ZIO[R1, E2, A1]

  def catchSomeCause[R1 <: R, E1 >: E, A1 >: A](pf: PartialFunction[Cause[E], ZIO[R1, E1, A1]]): ZIO[R1, E1, A1]
}
```

With the help of the `ZIO#catchAllCause` operator we can catch all errors of an effect and recover from them:

```scala

val exceptionalEffect = ZIO.attempt(???)

exceptionalEffect.catchAllCause {
  case Cause.Empty =>
    ZIO.debug("no error caught")
  case Cause.Fail(value, _) =>
    ZIO.debug(s"a failure caught: $value")
  case Cause.Die(value, _) =>
    ZIO.debug(s"a defect caught: $value")
  case Cause.Interrupt(fiberId, _) =>
    ZIO.debug(s"a fiber interruption caught with the fiber id: $fiberId")
  case Cause.Stackless(cause: Cause.Die, _) =>
    ZIO.debug(s"a stackless defect caught: ${cause.value}")
  case Cause.Stackless(cause: Cause[_], _) =>
    ZIO.debug(s"an unknown stackless defect caught: ${cause.squashWith(identity)}")
  case Cause.Then(left, right) =>
    ZIO.debug(s"two consequence causes caught")
  case Cause.Both(left, right) =>
    ZIO.debug(s"two parallel causes caught")
}
```

Additionally, there is a partial version of this operator called `ZIO#catchSomeCause`, which can be used when we don't want to catch all causes, but some of them.

## Catching Traces

The two `ZIO#catchAllTrace` and `ZIO#catchSomeTrace` operators are useful to catch the typed error as well as stack traces of exceptional effects:

```scala
trait ZIO[-R, +E, +A] {
  def catchAllTrace[R1 <: R, E2, A1 >: A](
    h: ((E, Trace)) => ZIO[R1, E2, A1]
  ): ZIO[R1, E2, A1]

  def catchSomeTrace[R1 <: R, E1 >: E, A1 >: A](
    pf: PartialFunction[(E, Trace), ZIO[R1, E1, A1]]
  ): ZIO[R1, E1, A1]
}
```

In the below example, let's try to catch a failure on the line number 4:

```scala

ZIO
  .fail("Oh uh!")
  .catchAllTrace {
    case ("Oh uh!", trace)
      if trace.toJava
        .map(_.getLineNumber)
        .headOption
        .contains(4) =>
      ZIO.debug("caught a failure on the line number 4")
    case _ =>
      ZIO.debug("caught other failures")
  }
```

## Catching Non-Fatal

We can use the `ZIO#catchAll` to recover from all non-fatal errors:

```scala
trait ZIO[-R, +E, +A] {
  final def catchAll[R1 <: R, E2, A1 >: A](
    h: E => ZIO[R1, E2, A1]
  )(implicit ev: CanFail[E], trace: Trace): ZIO[R1, E2, A1]
}
```

In case of occurring any [fatal error](#catching-traces), it will die.

```scala
openFile("data.json").catchAll(_ => openFile("backup.json"))
```

---

## Fallback

## `ZIO#orElse`

We can try one effect, or if it fails, try another effect with the `orElse` combinator:

```scala
trait ZIO[-R, +E, +A] {
  def orElse[R1 <: R, E2, A1 >: A](that: => ZIO[R1, E2, A1]): ZIO[R1, E2, A1]
}
```

Let's try an example:

```scala

val primaryOrBackupData: ZIO[Any, IOException, Array[Byte]] =
  readFile("primary.data").orElse(readFile("backup.data"))
```

## `ZIO#orElseEither`

If the original effect fails, this operator tries another effect, and as a result, returns either:

```scala
trait ZIO[-R, +E, +A] {
  def orElseEither[R1 <: R, E2, B](that: => ZIO[R1, E2, B]): ZIO[R1, E2, Either[A, B]]
}
```

This operator is useful when the fallback effect has a different result type than the original effect. So this will unify both in the `Either[A, B]` data type. Here is an example usage of this operator:

```scala

trait LocalConfig
trait RemoteConfig

def readLocalConfig: ZIO[Any, Throwable, LocalConfig] = ???
def readRemoteConfig: ZIO[Any, Throwable, RemoteConfig] = ???

val result: ZIO[Any, Throwable, Either[LocalConfig, RemoteConfig]] =
  readLocalConfig.orElseEither(readRemoteConfig)
```

## `ZIO#orElseSucceed`/`ZIO#orElseFail`

These two operators convert the original failure with constant succeed or failure values:

```scala
trait ZIO[-R, +R, +E] {
  def orElseFail[E1](e1: => E1): ZIO[R, E1, A]

  def orElseSucceed[A1 >: A](a1: => A1): ZIO[R, Nothing, A1]
}
```

The `ZIO#orElseFail` will always replace the original failure with the new one, so `E1` does not have to be a supertype of `E`. It is useful when we have `Unit` as an error, and we want to unify that with something else:

```scala

sealed trait AgeValidationException extends Exception
case class NegativeAgeException(age: Int) extends AgeValidationException
case class IllegalAgeException(age: Int)  extends AgeValidationException

def validate(age: Int): ZIO[Any, AgeValidationException, Int] = {
  if (age < 0)
    ZIO.fail(NegativeAgeException(age))
  else if (age < 18)
    ZIO.fail(IllegalAgeException(age))
  else ZIO.succeed(age)
}

val result: ZIO[Any, String, Int] =
  validate(3).orElseFail("invalid age")
```

The `ZIO#orElseSucceed` will always replace the original failure with a success value so the resulting effect cannot fail. It is useful when we have a constant value that will work in case the effect fails:

```scala
val result: ZIO[Any, Nothing, Int] =
  validate(3).orElseSucceed(0)
```

## `ZIO#orElseOptional`

When dealing with optional failure types, we might need to fall back to another effect when the failure value is `None`. This operator helps to do so:

```scala
trait ZIO[-R, +E, +A] {
  def orElseOptional[R1 <: R, E1, A1 >: A](
      that: => ZIO[R1, Option[E1], A1]
    )(implicit ev: E IsSubtypeOfError Option[E1]): ZIO[R1, Option[E1], A1] =
}
```

In the following example, the `parseInt(" ")` fails with `None`, so then the fallback effect results in a zero:

```scala

def parseInt(input: String): ZIO[Any, Option[String], Int] =
  input.toIntOption match {
    case Some(value) => ZIO.succeed(value)
    case None =>
      if (input.trim.isEmpty)
        ZIO.fail(None)
      else
        ZIO.fail(Some(s"invalid non-integer input: $input"))
  }

val result = parseInt("  ").orElseOptional(ZIO.succeed(0)).debug
```

## `ZIO.firstSuccessOf`/`ZIO#firstSuccessOf`

These two operators make it easy for a user to run an effect, and in case it fails, it will run a series of ZIO effects until one succeeds:

```scala
object ZIO {
  def firstSuccessOf[R, R1 <: R, E, A](
    zio: => ZIO[R, E, A],
    rest: => Iterable[ZIO[R1, E, A]]
  ): ZIO[R1, E, A] =
}

trait ZIO[-R, +E, +A] {
  final def firstSuccessOf[R1 <: R, E1 >: E, A1 >: A](
    rest: => Iterable[ZIO[R1, E1, A1]]
  ): ZIO[R1, E1, A1]
}
```

These methods use `orElse` to reduce the non-empty iterable of effects into a single effect.

In the following example, we are trying to get the config from the master node, and if it fails, we will try successively to retrieve the config from the next available node:

```scala

trait Config

def remoteConfig(name: String): ZIO[Any, Throwable, Config] =
  ZIO.attempt(???)

val masterConfig: ZIO[Any, Throwable, Config] =
  remoteConfig("master")

val nodeConfigs: Seq[ZIO[Any, Throwable, Config]] =
  List("node1", "node2", "node3", "node4").map(remoteConfig)

val config: ZIO[Any, Throwable, Config] =
  ZIO.firstSuccessOf(masterConfig, nodeConfigs)
```

---

## Folding

Scala's `Option` and `Either` data types have `fold`, which let us handle both failure and success at the same time. In a similar fashion, `ZIO` effects also have several methods that allow us to handle both failure and success.

## `ZIO#fold`/`ZIO#foldZIO`

The first fold method, `ZIO#fold`, lets us non-effectfully handle both failure and success, by supplying a non-effectful handler for each case. The second fold method, `ZIO#foldZIO`, lets us effectfully handle both failure and success, by supplying an effectful (but still pure) handler for each case:

```scala
trait ZIO[-R, +E, +A] {
  def fold[B](
    failure: E => B,
    success: A => B
  ): ZIO[R, Nothing, B]

  def foldZIO[R1 <: R, E2, B](
    failure: E => ZIO[R1, E2, B],
    success: A => ZIO[R1, E2, B]
  ): ZIO[R1, E2, B]
}
```

Let's try an example:

```scala

lazy val DefaultData: Array[Byte] = Array(0, 0)

val primaryOrDefaultData: UIO[Array[Byte]] =
  readFile("primary.data").fold(_ => DefaultData, data => data)
```

We can ignore any failure and success values:

```scala

val result: ZIO[Any, Nothing, Unit] =
  ZIO
    .fail("Uh oh!")         // ZIO[Any, String, Int]
    .as(5)                  // ZIO[Any, String, Int]
    .fold(_ => (), _ => ()) // ZIO[Any, Nothing, Unit]
```

It is equivalent to use the `ZIO#ignore` operator instead:

```scala

val result: ZIO[Any, Nothing, Unit] = ZIO.fail("Uh oh!").as(5).ignore
```

Now let's try the effectful version of the fold operation. In this example, in case of failure on reading from the primary file, we will fallback to another effectful operation which will read data from the secondary file:

```scala
val primaryOrSecondaryData: IO[IOException, Array[Byte]] =
  readFile("primary.data").foldZIO(
    failure = _    => readFile("secondary.data"),
    success = data => ZIO.succeed(data)
  )
```

Nearly all error handling methods are defined in terms of `foldZIO`, because it is both powerful and fast.

In the following example, `foldZIO` is used to handle both failure and success of the `readUrls` method:

```scala
val urls: UIO[Content] =
  readUrls("urls.json").foldZIO(
    error   => ZIO.succeed(NoContent(error)),
    success => fetchContent(success)
  )
```

It's important to note that both `ZIO#fold` and `ZIO#foldZIO` operators cannot catch fiber interruptions. So the following application will crash due to `InterruptedException`:

```scala

object MainApp extends ZIOAppDefault {
  def run = (ZIO.interrupt *> ZIO.fail("Uh oh!")).fold(_ => (), _ => ())
}
```

And here is the output:

```scala
timestamp=2022-02-24T13:41:01.696273024Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.InterruptedException: Interrupted by thread "zio-fiber-"
   at <empty>.MainApp.run(MainApp.scala:4)"
```

## `ZIO#foldCause`/`ZIO#foldCauseZIO`

This cause version of the `fold` operator is useful to access the full cause of the underlying fiber. So in case of failure, based on the exact cause, we can determine what to do:

```scala
trait ZIO[-R, +E, +A] {
  def foldCause[B](
    failure: Cause[E] => B,
    success: A => B
  ): ZIO[R, Nothing, B]

  def foldCauseZIO[R1 <: R, E2, B](
    failure: Cause[E] => ZIO[R1, E2, B],
    success: A => ZIO[R1, E2, B]
  ): ZIO[R1, E2, B]
}
```

Among the fold operators, these are the most powerful combinators. They can recover from any error, even fiber interruptions.

In the following example, we are printing the proper message according to what cause occurred due to failure:

```scala

val exceptionalEffect: ZIO[Any, Throwable, Unit] = ???

val myApp: ZIO[Any, IOException, Unit] =
  exceptionalEffect.foldCauseZIO(
    failure = {
      case Cause.Fail(value, _)        => Console.printLine(s"failure: $value")
      case Cause.Die(value, _)         => Console.printLine(s"cause: $value")
      case Cause.Interrupt(failure, _) => Console.printLine(s"${failure.threadName} interrupted!")
      case _                           => Console.printLine("failed due to other causes")
    },
    success = succeed => Console.printLine(s"succeeded with $succeed value")
  )
```

When catching errors using this operator, if our cases were not exhaustive, we may receive a defect of the type `scala.MatchError` :

```scala

object MainApp extends ZIOAppDefault {
  val exceptionalEffect: ZIO[Any, Throwable, Unit] = ZIO.interrupt

  val myApp: ZIO[Any, IOException, Unit] =
    exceptionalEffect.foldCauseZIO(
      failure = {
        case Cause.Fail(value, _) => ZIO.debug(s"failure: $value")
        case Cause.Die(value, _) => ZIO.debug(s"cause: ${value.toString}")
        // case Cause.Interrupt(failure, _) => ZIO.debug(s"${failure.threadName} interrupted!")
      },
      success = succeed => ZIO.debug(s"succeeded with $succeed value")
    )

  def run = myApp
}
```

The output:

```scala
timestamp=2022-02-24T11:05:40.241436257Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" scala.MatchError: Interrupt(Runtime(2,1645700739),Trace(Runtime(2,1645700739),Chunk(<empty>.MainApp.exceptionalEffect(MainApp.scala:6),<empty>.MainApp.myApp(MainApp.scala:9)))) (of class zio.Cause$Interrupt)
	at MainApp$.$anonfun$myApp$1(MainApp.scala:10)
	at zio.ZIO$TracedCont$$anon$33.apply(ZIO.scala:6167)
	at zio.ZIO$TracedCont$$anon$33.apply(ZIO.scala:6165)
	at zio.internal.FiberContext.runUntil(FiberContext.scala:885)
	at zio.internal.FiberContext.run(FiberContext.scala:115)
	at zio.internal.ZScheduler$$anon$1.run(ZScheduler.scala:151)
	at zio.internal.FiberContext.runUntil(FiberContext.scala:538)"
```

## `ZIO#foldTraceZIO`

This version of fold, provide us the facility to access the trace info of the failure:

```scala
trait ZIO[-R, +E, +A] {
  def foldTraceZIO[R1 <: R, E2, B](
    failure: ((E, Trace)) => ZIO[R1, E2, B],
    success: A => ZIO[R1, E2, B]
  )(implicit ev: CanFail[E]): ZIO[R1, E2, B]
}
```

```scala

val result: ZIO[Any, Nothing, Int] =
  validate(5).foldTraceZIO(
    failure = {
      case (_: NegativeAgeException, trace) =>
        ZIO.succeed(0).debug(
          "The entered age is negative\n" +
            s"trace info: ${trace.stackTrace.mkString("\n")}"
        )
      case (_: IllegalAgeException, trace) =>
        ZIO.succeed(0).debug(
          "The entered age in not legal\n" +
            s"trace info: ${trace.stackTrace.mkString("\n")}"
        )
    },
    success = s => ZIO.succeed(s)
  )
```

Note that similar to `ZIO#fold` and `ZIO#foldZIO` this operator cannot recover from fiber interruptions.

---

## Retrying

When we are building applications we want to be resilient in the face of a transient failure. This is where we need to retry to overcome these failures.

There are a number of useful methods on the ZIO data type for retrying failed effects:

## `ZIO#retry`

The most basic of these is `ZIO#retry`, which takes a `Schedule` and returns a new effect that will retry the first effect if it fails, according to the specified policy:

```scala
trait ZIO[-R, +E, +A] {
  def retry[R1 <: R, S](policy: => Schedule[R1, E, S]): ZIO[R1, E, A]
}
```

In this example, we try to read from a file. If we fail to do that, it will try five more times:

```scala

val retriedOpenFile: ZIO[Any, IOException, Array[Byte]] =
  readFile("primary.data").retry(Schedule.recurs(5))
```

## `ZIO#retryN`

In case of failure, a ZIO effect can be retried as many times as specified:

```scala

val file = readFile("primary.data").retryN(5)
```

## `ZIO#retryOrElse`

The next most powerful function is `ZIO#retryOrElse`, which allows specification of a fallback to use, if the effect does not succeed with the specified policy:

```scala
trait ZIO[-R, +E, +A] {
  def retryOrElse[R1 <: R, A1 >: A, S, E1](
    policy: => Schedule[R1, E, S],
    orElse: (E, S) => ZIO[R1, E1, A1]
  ): ZIO[R1, E1, A1] =
}
```

The `orElse` is the recovery function that has two inputs:

1. The last error message
2. Schedule output

So based on these two values, we can decide what to do as the fallback operation. Let's try an example:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    Random
      .nextIntBounded(11)
      .flatMap { n =>
        if (n < 9)
          ZIO.fail(s"$n is less than 9!").debug("failed")
        else
          ZIO.succeed(n).debug("succeeded")
      }
      .retryOrElse(
        policy = Schedule.recurs(5),
        orElse = (lastError, scheduleOutput: Long) =>
          ZIO.debug(s"after $scheduleOutput retries, we couldn't succeed!") *>
            ZIO.debug(s"the last error message we received was: $lastError") *>
            ZIO.succeed(-1)
      )
      .debug("the final result")
}
```

## `ZIO#retryOrElseEither`

This operator is almost the same as the **`ZIO#retryOrElse`** except it will return either result of the original or the fallback operation:

```scala

trait LocalConfig
trait RemoteConfig

def readLocalConfig: ZIO[Any, Throwable, LocalConfig] = ???
def readRemoteConfig: ZIO[Any, Throwable, RemoteConfig] = ???

val result: ZIO[Any, Throwable, Either[RemoteConfig, LocalConfig]] =
  readLocalConfig.retryOrElseEither(
    schedule0 = Schedule.fibonacci(1.seconds),
    orElse = (_, _: Duration) => readRemoteConfig
  )
```

## `ZIO#retryUntil`/`ZIO#retryUntilZIO`

We can retry an effect until a condition on the error channel is satisfied:

```scala
trait ZIO[-R, +E, +A] {
  def retryUntil(f: E => Boolean): ZIO[R, E, A]
  def retryUntilZIO[R1 <: R](f: E => URIO[R1, Boolean]): ZIO[R1, E, A]
}
```

Assume we have defined the following remote service call:

```scala
sealed trait  ServiceError extends Exception
case object TemporarilyUnavailable extends ServiceError
case object DataCorrupted          extends ServiceError

def remoteService: ZIO[Any, ServiceError, Unit] = ???
```

In the following example, we repeat the failed remote service call until we reach the `DataCorrupted` error:

```scala
remoteService.retryUntil(_ == DataCorrupted)
```

To provide an effectful predicate we use the `ZIO#retryUntilZIO` operator.

## `ZIO#retryUntilEqual`

Like the previous operator, it tries until its error is equal to the specified error:

```scala
remoteService.retryUntilEquals(DataCorrupted)
```

## `ZIO#retryWhile`/`ZIO#retryWhileZIO`

Unlike the `ZIO#retryUntil` it will retry the effect while its error satisfies the specified predicate:

```scala
trait ZIO[-R, +E, +A] {
  def retryWhile(f: E => Boolean): ZIO[R, E, A]
  def retryWhileZIO[R1 <: R](f: E => URIO[R1, Boolean]): ZIO[R1, E, A]
}
```

In the following example, we repeat the failed remote service call while we have the `TemporarilyUnavailable` error:

```scala
remoteService.retryWhile(_ == TemporarilyUnavailable)
```

To provide an effectful predicate we use the `ZIO#retryWhileZIO` operator.

## `ZIO#retryWhileEquals`

Like the previous operator, it tries while its error is equal to the specified error:

```scala
remoteService.retryWhileEquals(TemporarilyUnavailable)
```

---

## Sandboxing

We know that a ZIO effect may fail due to a failure, a defect, a fiber interruption, or a combination of these causes. So a ZIO effect may contain more than one cause. Using the `ZIO#sandbox` operator, we can sandbox all errors of a ZIO application, whether the cause is a failure, defect, or a fiber interruption or combination of these. This operator exposes the full cause of a ZIO effect into the error channel:

```scala
trait ZIO[-R, +E, +A] {
  def sandbox: ZIO[R, Cause[E], A]
}
```

We can use the `ZIO#sandbox` operator to uncover the full causes of an _exceptional effect_. So we can see all the errors that occurred as a type of `Cause[E]` at the error channel of the `ZIO` data type. So then we can use normal error-handling operators such as `ZIO#catchSome` and `ZIO#catchAll` operators:

```scala

object MainApp extends ZIOAppDefault {
  val effect: ZIO[Any, String, String] =
    ZIO.succeed("primary result") *> ZIO.fail("Oh uh!")

  val myApp: ZIO[Any, Cause[String], String] =
    effect.sandbox.catchSome {
      case Cause.Interrupt(fiberId, _) =>
        ZIO.debug(s"Caught interruption of a fiber with id: $fiberId") *>
          ZIO.succeed("fallback result on fiber interruption")
      case Cause.Die(value, _) =>
        ZIO.debug(s"Caught a defect: $value") *>
          ZIO.succeed("fallback result on defect")
      case Cause.Fail(value, _) =>
        ZIO.debug(s"Caught a failure: $value") *>
          ZIO.succeed("fallback result on failure")
    }

  val finalApp: ZIO[Any, String, String] = myApp.unsandbox.debug("final result")

  def run = finalApp
}

// Output:
// Caught a failure: Oh uh!
// final result: fallback result on failure
```

Using the `sandbox` operation we are exposing the full cause of an effect. So then we have access to the underlying cause in more detail. After handling exposed causes using `ZIO#catch*` operators, we can undo the `sandbox` operation using the `unsandbox` operation. It will submerge the full cause (`Cause[E]`) again:

```scala

val effect: ZIO[Any, String, String] =
  ZIO.succeed("primary result") *> ZIO.fail("Oh uh!")

effect            // ZIO[Any, String, String]
  .sandbox        // ZIO[Any, Cause[String], String]
  .catchSome(???) // ZIO[Any, Cause[String], String]
  .unsandbox      // ZIO[Any, String, String]
```

There is another version of sandbox called `ZIO#sandboxWith`. This operator helps us to sandbox, then catch all causes, and then unsandbox back:

```scala
trait ZIO[-R, +E, +A] {
  def sandboxWith[R1 <: R, E2, B](f: ZIO[R1, Cause[E], A] => ZIO[R1, Cause[E2], B])
}
```

Let's try the previous example using this operator:

```scala

object MainApp extends ZIOAppDefault {
  val effect: ZIO[Any, String, String] =
    ZIO.succeed("primary result") *> ZIO.fail("Oh uh!")

  val myApp =
    effect.sandboxWith[Any, String, String] { e =>
      e.catchSome {
        case Cause.Interrupt(fiberId, _) =>
          ZIO.debug(s"Caught interruption of a fiber with id: $fiberId") *>
            ZIO.succeed("fallback result on fiber interruption")
        case Cause.Die(value, _) =>
          ZIO.debug(s"Caught a defect: $value") *>
            ZIO.succeed("fallback result on defect")
        case Cause.Fail(value, _) =>
          ZIO.debug(s"Caught a failure: $value") *>
            ZIO.succeed("fallback result on failure")
      }
    }
  def run = myApp.debug
}

// Output:
// Caught a failure: Oh uh!
// fallback result on failure
```

---

## Timing out

## `ZIO#timeout`

ZIO lets us timeout any effect using the `ZIO#timeout` method, which returns a new effect that succeeds with an `Option`. A value of `None` indicates the timeout elapsed before the effect completed. If an effect times out, then instead of continuing to execute in the background, it will be interrupted so no resources will be wasted.

Assume we have the following effect:

```scala

val myApp =
  for {
    _ <- ZIO.debug("start doing something.")
    _ <- ZIO.sleep(2.second)
    _ <- ZIO.debug("my job is finished!")
  } yield "result"
```

We should note that when we use the `ZIO#timeout` operator on the `myApp`, it doesn't return until one of the following situations happens:

1. The original effect returns before the timeout elapses so the output will be `Some` of the produced value by the original effect:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    myApp
      .timeout(3.second)
      .debug("output")
      .timed
      .map(_._1.toMillis / 1000)
      .debug("execution time of the whole program in second")
}

// Output:
// start doing something.
// my job is finished!
// output: Some(result)
// execution time of the whole program in second: 2
```

2. The original effect interrupted after the timeout elapses:

    - If the effect is interruptible it will be immediately interrupted, and finally, the timeout operation produces `None` value.

   ```scala mdoc:compile-only

   object MainApp extends ZIOAppDefault {
     def run =
       myApp
         .timeout(1.second)
         .debug("output")
         .timed
         .map(_._1.toMillis / 1000)
         .debug("execution time of the whole program in second")
   }

   // Output:
   // start doing something.
   // output: None
   // execution time of the whole program in second: 1
   ```

    - If the effect is uninterruptible it will be blocked until the original effect safely finished its work, and then the timeout operator produces the `None` value:

   ```scala mdoc:compile-only

   object MainApp extends ZIOAppDefault {
     def run =
       myApp
         .uninterruptible
         .timeout(1.second)
         .debug("output")
         .timed
         .map(_._1.toMillis / 1000)
         .debug("execution time of the whole program in second")
   }

   // Output:
   // start doing something.
   // my job is finished!
   // output: None
   // execution time of the whole program in second: 2
   ```

Instead of waiting for the original effect to be interrupted, we can use `effect.disconnect.timeout` which first disconnects the effect's interruption signal before performing the timeout. By using this technique, we can return early after the timeout has passed and before an underlying effect has been interrupted.

```scala
object MainApp extends ZIOAppDefault {
  def run =
    myApp
      .uninterruptible
      .disconnect
      .timeout(1.second)
      .debug("output")
      .timed
      .map(_._1.toMillis / 1000)
      .debug("execution time of the whole program in second")
}
```

By using this technique, the original effect will be interrupted in the background.

## `ZIO#timeoutTo`

This operator is similar to the previous one, but it also allows us to manually create the final result type:

```scala

val delayedNextInt: ZIO[Any, Nothing, Int] =
  Random.nextIntBounded(10).delay(2.second)

val r1: ZIO[Any, Nothing, Option[Int]] =
  delayedNextInt.timeoutTo(None)(Some(_))(1.seconds)

val r2: ZIO[Any, Nothing, Either[String, Int]] =
  delayedNextInt.timeoutTo(Left("timeout"))(Right(_))(1.seconds)

val r3: ZIO[Any, Nothing, Int] =
  delayedNextInt.timeoutTo(-1)(identity)(1.seconds)
```

## `ZIO#timeoutFail`/`ZIO#timeoutFailCause`

In case of elapsing the timeout, we can produce a particular error message:

```scala

val r1: ZIO[Any, TimeoutException, Int] =
  delayedNextInt.timeoutFail(new TimeoutException)(1.second)

val r2: ZIO[Any, Nothing, Int] =
  delayedNextInt.timeoutFailCause(Cause.die(new Error("timeout")))(1.second)
```

---

## Sequential and Parallel Errors

A simple and regular ZIO application usually fails with one error, which is the first error encountered by the ZIO runtime:

```scala

object MainApp extends ZIOAppDefault {
  val fail = ZIO.fail("Oh uh!")
  val die = ZIO.dieMessage("Boom!")
  val interruption = ZIO.interrupt

  def run = (fail <*> die) *> interruption
}
```

This application will fail with the first error which is "Oh uh!":

```scala
timestamp=2022-03-09T09:50:22.067072131Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
	at <empty>.MainApp.fail(MainApp.scala:4)"
```

In some cases, we may run into multiple errors. When we perform parallel computations, the application may fail due to multiple errors:

```scala

object MainApp extends ZIOAppDefault {
  def run = ZIO.fail("Oh!") <&> ZIO.fail("Uh!")
}
```

If we run this application, we can see two exceptions in two different fibers that caused the failure (`zio-fiber-0` and `zio-fiber-14`):

```scala
timestamp=2022-03-09T08:05:48.703035927Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-13" java.lang.String: Oh!
	at <empty>.MainApp.run(MainApp.scala:4)
Exception in thread "zio-fiber-14" java.lang.String: Uh!
	at <empty>.MainApp.run(MainApp.scala:4)"
```

ZIO has a combinator called `ZIO#parallelErrors` that exposes all parallel failure errors in the error channel:

```scala

val result: ZIO[Any, ::[String], Nothing] =
  (ZIO.fail("Oh uh!") <&> ZIO.fail("Oh Error!")).parallelErrors
```

Note that this operator is only for failures, not defects or interruptions.

Also, when we work with resource-safety operators like `ZIO#ensuring` we can have multiple sequential errors. Why? because regardless of the original effect has any errors or not, the finalizer is uninterruptible. So the finalizer will be run. Unless the finalizer should be an unexceptional effect (`URIO`), it may die because of a defect. Therefore, it creates multiple sequential errors:

```scala

object MainApp extends ZIOAppDefault {
  def run = ZIO.fail("Oh uh!").ensuring(ZIO.dieMessage("Boom!"))
}
```

When we run this application, we can see that the original failure (`Oh uh!`) was suppressed by another defect (`Boom!`):

```scala
timestamp=2022-03-09T08:30:56.563179230Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
	at <empty>.MainApp.run(MainApp.scala:4)
	Suppressed: java.lang.RuntimeException: Boom!
		at <empty>.MainApp.run(MainApp.scala:4)"
```

---

## Typed Errors Guarantees

**Typed errors don't guarantee the absence of defects and interruptions.** Having an effect of type `ZIO[R, E, A]`, means it can fail because of some failure of type `E`, but it doesn't mean it can't die or be interrupted. So the error channel is only for `failure` errors.

In the following example, the type of the `validateNonNegativeNumber` function is `ZIO[Any, String, Int]` which denotes it is a typed exceptional effect. It can fail of type `String` but it still can die with the type of `NumberFormatException` defect:

```scala

def validateNonNegativeNumber(input: String): ZIO[Any, String, Int] =
  input.toIntOption match {
    case Some(value) if value >= 0 =>
      ZIO.succeed(value)
    case Some(other) =>
      ZIO.fail(s"the entered number is negative: $other")
    case None =>
      ZIO.die(
        new NumberFormatException(
          s"the entered input is not in the correct number format: $input"
        )
      )
  }
```

Also, its underlying fiber can be interrupted without affecting the type of the error channel:

```scala

val myApp: ZIO[Any, String, Int] =
  for {
    f <- validateNonNegativeNumber("5").fork
    _ <- f.interrupt
    r <- f.join
  } yield r
```

Therefore, if we run the `myApp` effect, it will be interrupted before it gets the chance to finish.

---

## Defects

By providing a `Throwable` value to the `ZIO.die` constructor, we can describe a dying effect:

```scala
object ZIO {
  def die(t: => Throwable): ZIO[Any, Nothing, Nothing]
}
```

Here is an example of such effect, which will die because of encountering _divide by zero_ defect:

```scala

val dyingEffect: ZIO[Any, Nothing, Nothing] =
  ZIO.die(new ArithmeticException("divide by zero"))
```

The result is the creation of a ZIO effect whose error channel and success channel are both `Nothing`. In other words, this effect cannot fail and does not produce anything. Instead, it is an effect describing a _defect_ or an _unexpected error_.

Let's see what happens if we run this effect:

```scala

object MainApp extends ZIOAppDefault {
  def run = ZIO.die(new ArithmeticException("divide by zero"))
}
```

If we run this effect, ZIO runtime will print the stack trace that belongs to this defect. So, here is the output:

```scala
timestamp=2022-02-16T13:02:44.057191215Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.ArithmeticException: divide by zero
	at MainApp$.$anonfun$run$1(MainApp.scala:4)
	at zio.ZIO$.$anonfun$die$1(ZIO.scala:3384)
	at zio.internal.FiberContext.runUntil(FiberContext.scala:255)
	at zio.internal.FiberContext.run(FiberContext.scala:115)
	at zio.internal.ZScheduler$$anon$1.run(ZScheduler.scala:151)
	at <empty>.MainApp.run(MainApp.scala:4)"
```

The `ZIO.die` constructor is used to manually describe a dying effect because of a defect inside the code.

For example, assume we want to write a `divide` function that takes two numbers and divides the first number by the second. We know that the `divide` function is not defined for zero dominators. Therefore, we should signal an error if division by zero occurs.

We have two choices to implement this function using the ZIO effect:

1. We can divide the first number by the second, and if the second number was zero, we can fail the effect using `ZIO.fail` with the `ArithmeticException` failure value:

```scala

def divide(a: Int, b: Int): ZIO[Any, ArithmeticException, Int] =
  if (b == 0)
    ZIO.fail(new ArithmeticException("divide by zero"))
  else
    ZIO.succeed(a / b)
```

2. We can divide the first number by the second. In the case of zero for the second number, we use `ZIO.die` to kill the effect by sending a signal of `ArithmeticException` as a defect:

```scala

def divide(a: Int, b: Int): ZIO[Any, Nothing, Int] =
  if (b == 0)
    ZIO.die(new ArithmeticException("divide by zero")) // Unexpected error
  else
    ZIO.succeed(a / b)
```

So what is the difference between these two approaches? Let's compare the function signature:

```scala
def divide(a: Int, b: Int): ZIO[Any, ArithmeticException, Int]   // using ZIO.fail
def divide(a: Int, b: Int): ZIO[Any, Nothing,             Int]   // using ZIO.die
```

1. The first approach, models the _divide by zero_ error by _failing_ the effect. We call these failures _expected errors_ or _typed error_.
2. While the second approach models the _divide by zero_ error by _dying_ the effect. We call these kinds of errors _unexpected errors_, _defects_ or _untyped errors_.

We use the first method when we are handling errors as we expect them, and thus we know how to handle them. In contrast, the second method is used when we aren't expecting those errors in our domain, and we don't know how to handle them. Therefore, we use the _let it crash_ philosophy.

In the second approach, we can see that the `divide` function indicates that it cannot fail because it's error channel is `Nothing`. However, it doesn't mean that this function hasn't any defects. ZIO defects are not typed, so they cannot be seen in type parameters.

Note that to create an effect that will die, we shouldn't throw an exception inside the `ZIO.die` constructor, although it works. Instead, the idiomatic way of creating a dying effect is to provide a `Throwable` value into the `ZIO.die` constructor:

```scala

val defect1 = ZIO.die(new ArithmeticException("divide by zero"))       // recommended
val defect2 = ZIO.die(throw new ArithmeticException("divide by zero")) // not recommended
```

Also, if we import a code that may throw an exception, all the exceptions will be translated to the ZIO defect:

```scala

val defect3 = ZIO.succeed(throw new Exception("boom!"))
```

Therefore, in the second approach of the `divide` function, we do not require to manually die the effect in case of the _dividing by zero_, because the JVM itself throws an `ArithmeticException` when the denominator is zero.

When we import any code into the `ZIO` effect, any exception is thrown inside that code will be translated to _ZIO defects_ by default. So the following program is the same as the previous example:

```scala

def divide(a: Int, b: Int): ZIO[Any, Nothing, Int] =
  ZIO.succeed(a / b)
```

Another important note is that if we `map`/`flatMap` a ZIO effect and then accidentally throw an exception inside the map operation, that exception will be translated to a ZIO defect:

```scala

val defect4 = ZIO.succeed(???).map(_ => throw new Exception("Boom!"))
val defect5 = ZIO.attempt(???).map(_ => throw new Exception("Boom!"))
```

---

## Failures

When writing ZIO application, we can model a failure, using the `ZIO.fail` constructor:

```scala
trait ZIO {
  def fail[E](error: => E): ZIO[Any, E, Nothing]
}
```

Let's try to model some failures using this constructor:

```scala

val f1: ZIO[Any, String, Nothing] = ZIO.fail("Oh uh!")
val f2: ZIO[Any, String, Int]     = ZIO.succeed(5) *> ZIO.fail("Oh uh!")
```

Now, let's try to run a failing effect and see what happens:

```scala

object MainApp extends ZIOAppDefault {
  def run = ZIO.succeed(5) *> ZIO.fail("Oh uh!")
}
```

This will crash the application and print the following stack trace:

```scala
timestamp=2022-03-08T17:55:50.002161369Z level=ERROR thread=#zio-fiber-0 message="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
	at <empty>.MainApp.run(MainApp.scala:4)"
```

We can also model a failure using `Exception`:

```
val f3: ZIO[Any, Exception, Nothing] =
  ZIO.fail(new Exception("Oh uh!"))
```

Or using user-defined failure types (domain errors):

```scala

case class NegativeNumberException(msg: String) extends Exception(msg)

def validateNonNegative(input: Int): ZIO[Any, NegativeNumberException, Int] =
  if (input < 0)
    ZIO.fail(NegativeNumberException(s"entered negative number: $input"))
  else
    ZIO.succeed(input)
```

In the above examples, we can see that the type of the `validateNonNegative` function is `ZIO[Any, NegativeNumberException, Int]`. It means this is an exceptional effect, which may fail with the type of `NegativeNumberException`.

The `ZIO.fail` constructor is somehow the moral equivalent of `throw` for pure codes. We will discuss this [further](../declarative.md).

---

## Fatal Errors

In ZIO on the JVM platform, the `VirtualMachineError` and all its subtypes are the only errors considered fatal by the ZIO runtime. So if during the running application, the JVM throws any of these errors like `StackOverflowError`, the ZIO runtime considers it as a catastrophic fatal error. So it will interrupt the whole application immediately without safe resource interruption. None of the `ZIO#catchAll` and `ZIO#catchAllDefects` can catch these fatal errors. At most, if we set the `Runtime.setReportFatal`, the application will log the stack trace before interrupting the entire application.

Here is an example of manually creating a fatal error. Although we are ignoring all expected and unexpected errors, the fatal error interrupts the whole application:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    ZIO
      .attempt(
        throw new StackOverflowError(
          "The call stack pointer exceeds the stack bound."
        )
      )
      .catchAll(_ => ZIO.unit)       // ignoring all expected errors
      .catchAllDefect(_ => ZIO.unit) // ignoring all unexpected errors
}
```

The output will be something like this:

```scala
java.lang.StackOverflowError: The call stack pointer exceeds the stack bound.
at zio.examples.MainApp$.$anonfun$run$1(MainApp.scala:10)
at zio.ZIO$.liftedTree1$1(ZIO.scala:2603)
at zio.ZIO$.$anonfun$attempt$1(ZIO.scala:2603)
at zio.ZIO$.$anonfun$isFatalWith$1(ZIO.scala:3571)
at zio.internal.FiberContext.runUntil(FiberContext.scala:410)
at zio.internal.FiberContext.run(FiberContext.scala:111)
at zio.Runtime.unsafeRunWithRefs(Runtime.scala:400)
  ...
**** WARNING ****
Catastrophic error encountered. Application not safely interrupted. Resources may be leaked. Check the logs for more details and consider overriding `Runtime.reportFatal` to capture context.
```

Note that we can change the default way to report fatal errors using `Runtime#reportFatal` or the `Runtime.setReportFatal` layer.

---

## Three Types of Errors

We should consider three types of errors when writing ZIO applications:

1. **[Failures](failures.md)** are expected errors. We use `ZIO.fail` to model failures. As they are expected, we know how to handle them. We should handle these errors and prevent them from propagating throughout the call stack.

2. **[Defects](defects.md)** are unexpected errors. We use `ZIO.die` to model a defect. As they are not expected, we need to propagate them through the application stack, until in the upper layers one of the following situations happens:

    - In one of the upper layers, it makes sense to expect these errors. So we will convert them to failure, and then they can be handled.
    - None of the upper layers will catch these errors, so it will finally crash the whole application.

3. **[Fatals](fatals.md)** are catastrophic unexpected errors. When they occur we should kill the application immediately without propagating the error furthermore. At most, we might need to log the error and print its call stack.

---

## Fiber

To perform an effect without blocking the current process, we can use fibers, which are a lightweight concurrency mechanism.

We can `fork` any `ZIO[R, E, A]` to immediately yield an `URIO[R, Fiber[E, A]]`. The provided `Fiber` can be used to `join` the fiber, which will resume on production of the fiber's value, or to `interrupt` the fiber, which immediately terminates the fiber and safely releases all resources acquired by the fiber.

```scala
val analyzed =
  for {
    fiber1   <- analyzeData(data).fork  // IO[E, Analysis]
    fiber2   <- validateData(data).fork // IO[E, Boolean]
    // Do other stuff
    valid    <- fiber2.join
    _        <- if (!valid) fiber1.interrupt
                else ZIO.unit
    analyzed <- fiber1.join
  } yield analyzed
```

## Lifetime of Child Fibers

When we fork fibers, depending on how we fork them we can have four different lifetime strategies for the child fibers:

1. **Fork With Automatic Supervision**— If we use the ordinary `ZIO#fork` operation, the child fiber will be automatically supervised by the parent fiber. The lifetime child fibers are tied to the lifetime of their parent fiber. This means that these fibers will be terminated either when they end naturally, or when their parent fiber is terminated.
3. **Fork in Global Scope (Daemon)**— Sometimes we want to run long-running background fibers that aren't tied to their parent fiber, and also we want to fork them in a global scope. Any fiber that is forked in global scope will become daemon fiber. This can be achieved by using the `ZIO#forkDaemon` operator. As these fibers have no parent, they are not supervised, and they will be terminated when they end naturally, or when our application is terminated.
4. **Fork in Local Scope**— Sometimes, we want to run a background fiber that isn't tied to its parent fiber, but we want to live that fiber in the local scope. We can fork fibers in the local scope by using `ZIO#forkScoped`. Such fibers can outlive their parent fiber (so they are not supervised by their parents), and they will be terminated when their life end or their local scope is closed.
5. **Fork in Specific Scope**— This is similar to the previous strategy, but we can have more fine-grained control over the lifetime of the child fiber by forking it in a specific scope. We can do this by using the `ZIO#forkIn` operator.

:::note
Forking with **automatic supervision** is the _default strategy_. When we use the `ZIO#fork` method, the lifetime of child fibers is tied to their parent fiber. However, sometimes we don't want this behavior. Instead, we use three other alternatives.
:::

:::note Managing Fiber Lifetime Using `Scope` Data Type
The second and third strategies are required to work with the `Scope` data type. A contextual data type that describes a resource's lifetime, in this case, the fiber's lifetime. To learn more about `Scope` we have a [separate section](../resource/scope.md) on it.
:::

### Fork with Automatic Supervision

ZIO uses a **structured concurrency** model where fiber lifetimes are cleanly nested. The lifetime of a fiber depends on the lifetime of its parent fiber.

To illustrate this, let's look at some examples:

1. In the following example, we have two fibers. The first fiber is responsible for running the first and last debug tasks. It is also responsible for creating the second fiber. The second fiber is a task that forked from the first fiber and will never produce anything:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      _ <- ZIO.debug(s"Application started!")
      _ <- ZIO.never.onInterrupt(_ => ZIO.debug(s"The child fiber interrupted!")).fork
      _ <- ZIO.debug(s"Application finished!")
    } yield ()
}
```

In this example, the child fiber will be interrupted when the parent fiber is finished (successfully or interrupted).

:::caution
The example above is just for educational purposes. In a real application, the `onInterrupt` callback is not guaranteed to be called just before the parent fiber finishes its execution. It is possible that the parent fiber will be finished just before the `onInterrupt` callback is called. So this example is the simplified version of the following example:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    ZIO.fiberIdWith { parent =>
      for {
        _     <- ZIO.debug(s"fiber-${parent.id} Application started!")
        latch <- Promise.make[Nothing, Unit]
        _ <- ZIO.fiberIdWith { child =>
               (latch.succeed(()) *> ZIO.never).onInterrupt(_ =>
                 ZIO.debug(s"fiber-${child.id} The child fiber interrupted!")
               )
             }.fork
        _ <- latch.await
        _ <- ZIO.debug(s"fiber-${parent.id} Application finished!")
      } yield ()
    }
}
```

In this version, other than making sure that the `onInterrupt` callback is called, we also added fiber ids to the debug messages. Here is the output of the above example:

```
fiber-6 Application started!
fiber-6 Application finished!
fiber-7 The child fiber interrupted!
```
:::

2. Here is another example. In this case, the `foo` fiber creates the `bar` fiber. The `bar` fiber has a long-running task that never finishes. ZIO guarantees that the `bar` fiber will not outlive the `foo` fiber:

```scala

object MainApp extends ZIOAppDefault {
  val barJob: ZIO[Any, Nothing, Long] =
    ZIO
      .debug("Bar: still running!")
      .repeat(Schedule.fixed(1.seconds))

  val fooJob: ZIO[Any, Nothing, Unit] =
    for {
      _ <- ZIO.debug("Foo: started!")
      _ <- barJob.fork
      _ <- ZIO.sleep(3.seconds)
      _ <- ZIO.debug("Foo: finished!")
    } yield ()

  def run =
    for {
      f <- fooJob.fork
      _ <- f.join
    } yield ()
}
```

The output of the above program is:

```
Foo: started!
Bar: still running!
Bar: still running!
Bar: still running!
Foo: finished!
```

This pattern can be applied to any nested level of fibers. 

### Fork in Global Scope (Daemon)

Using `ZIO#forkDaemon` we can create a daemon fiber from a `ZIO` effect. Its lifetime is tied to the global scope. So if the parent fiber terminates, the daemon fiber will not be terminated. It will only will be terminated when the global scope is closed, or its life end naturaly.

```scala

object MainApp extends ZIOAppDefault {
  val barJob: ZIO[Any, Nothing, Long] =
    ZIO
      .debug("Bar: still running!")
      .repeat(Schedule.fixed(1.seconds))

  val fooJob: ZIO[Any, Nothing, Unit] =
    for {
      _ <- ZIO.debug("Foo: started!")
      _ <- barJob.forkDaemon
      _ <- ZIO.sleep(3.seconds)
      _ <- ZIO.debug("Foo: finished!")
    } yield ()

  def run =
    for {
      f <- fooJob.fork
      _ <- ZIO.sleep(5.seconds)
    } yield ()
}
```

If we run the above program, we will see the following output which shows that while the lifetime of the `foo` fiber ends after 3 seconds, the daemon fiber (`bar`) is still running until the global scope is closed:

```
Foo: started!
Bar: still running!
Bar: still running!
Bar: still running!
Foo: finished!
Bar: still running!
Bar: still running!
```

Even if we interrupt the `foo` fiber, the daemon fiber (`foo`) will not be interrupted:

```scala

object MainApp2 extends ZIOAppDefault {
  val barJob: ZIO[Any, Nothing, Long] =
    ZIO
      .debug("Bar: still running!")
      .repeat(Schedule.fixed(1.seconds))

  val fooJob: ZIO[Any, Nothing, Unit] =
    (for {
      _ <- ZIO.debug("Foo: started!")
      _ <- barJob.forkDaemon
      _ <- ZIO.sleep(3.seconds)
      _ <- ZIO.debug("Foo: finished!")
    } yield ()).onInterrupt(_ => ZIO.debug("Foo: interrupted!"))

  def run =
    for {
      f <- fooJob.fork
      _ <- ZIO.sleep(2.seconds)
      _ <- f.interrupt
      _ <- ZIO.sleep(3.seconds)
    } yield ()
}
```

The output:

```
Foo: started!
Bar: still running!
Bar: still running!
Foo: interrupted!
Bar: still running!
Bar: still running!
Bar: still running!
Bar: still running!
```

### Fork in Local Scope

Sometimes we want to attach fiber to a local scope. In such cases, we can use the `ZIO#forkScoped` operator. By using this operator, the lifetime of the forked fiber can be outlived the lifetime of its parent fiber, and it will be terminated when the local scope is closed:

```scala

object MainApp extends ZIOAppDefault {
  val barJob: ZIO[Any, Nothing, Long] =
    ZIO
      .debug("Bar: still running!")
      .repeat(Schedule.fixed(1.seconds))

  val fooJob: ZIO[Scope, Nothing, Unit] =
    (for {
      _ <- ZIO.debug("Foo: started!")
      _ <- barJob.forkScoped
      _ <- ZIO.sleep(2.seconds)
      _ <- ZIO.debug("Foo: finished!")
    } yield ()).onInterrupt(_ => ZIO.debug("Foo: interrupted!"))

  def run =
    for {
      _ <- ZIO.scoped {
        for {
          _ <- ZIO.debug("Local scope started!")
          _ <- fooJob.fork
          _ <- ZIO.sleep(5.seconds)
          _ <- ZIO.debug("Leaving the local scope!")
        } yield ()
      }
      _ <- ZIO.debug("Do something else and sleep for 10 seconds")
      _ <- ZIO.sleep(10.seconds)
      _ <- ZIO.debug("Application exited!")
    } yield ()
}
```

In the above example, the `bar` fiber forked in the local scope has bigger lifetime than its parent fiber (`foo`). So, when its parent fiber (`foo`) is terminated, the `bar` fiber still running in the local scope until the local scope is closed. Let's see the output:

```
Local scope started!
Foo: started!
Bar: still running!
Bar: still running!
Foo: finished!
Bar: still running!
Bar: still running!
Bar: still running!
Leaving the local scope!
Do something else and sleep for 10 seconds
Application exited!
```

### Fork in Specific Scope

There are some cases where we need more fine-grained control, so we want to fork a fiber in a specific scope. We can use the `ZIO#forkIn` operator which takes the target scope as an argument:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    ZIO.scoped {
      for {
        scope <- ZIO.scope
        _     <-
          ZIO.scoped {
            for {
              _ <- ZIO
                     .debug("Still running ...")
                     .repeat(Schedule.fixed(1.second))
                     .forkIn(scope)
              _ <- ZIO.sleep(3.seconds)
              _ <- ZIO.debug("The innermost scope is about to be closed.")
            } yield ()
          }
        _     <- ZIO.sleep(5.seconds)
        _     <- ZIO.debug("The outer scope is about to be closed.")
      } yield ()
    }
}
```

The output:

```
Still running ...
Still running ...
Still running ...
The innermost scope is about to be closed.
Still running ...
Still running ...
Still running ...
Still running ...
Still running ...
Still running ...
The outer scope is about to be closed.
```

## Background Processes and Layers

Sometimes, we want to create a layer that has a background process attached to the global scope. For example, think of a Kafka layer that has a background process that is constantly reading/writing messages from/to Kafka topics. In such situations, we can create a layer from such a background process.

In the following example, even though we have provided the `layer` locally to one part of our application, the background process inside the layer is still running in the global scope:

```scala

object MainApp extends ZIOAppDefault {
  val layer: ZLayer[Scope, Nothing, Int] =
    ZLayer.fromZIO {
      ZIO
        .debug("Still running ...")
        .repeat(Schedule.fixed(1.second))
        .forkDaemon
        .as(42)
    }

  def run =
    for {
      _ <- ZIO.service[Int].provideLayer(layer) *> ZIO.debug("Int layer provided")
      _ <- ZIO.sleep(5.seconds)
    } yield ()
}
```

Output:

```
Still running ...
int layer provided
Still running ...
Still running ...
Still running ...
Still running ...
Still running ...
```

## Operations

### fork and join
Whenever we need to start a fiber, we have to `fork` an effect to get a new fiber. This is similar to the `start` method on Java thread or submitting a new thread to the thread pool in Java, it is the same idea. Also, joining is a way of waiting for that fiber to compute its value. We are going to wait until it's done and receive its result.

:::note
Fibers (including those created with `forkDaemon`) **inherit the interruptibility status of their parent**. In other words, if the parent effect is currently running uninterruptibly then the child fiber will also be uninterruptible even calling `child.interrupt` will have no effect. To ensure a forked fiber is interruptible while preserving the parent’s uninterruptibility, use `ZIO.uninterruptibleMask`.

For example, this code hangs because the child inherited uninterruptibility using `fib.interrupt`:

```scala

val parent = ZIO.uninterruptible {
  for {
    _   <- ZIO.logInfo("Parent is uninterruptible")
    fib <- ZIO.never.fork
    _ <- ZIO.logInfo("Attempting to interrupt the child in 5 seconds...")
    _ <- ZIO.sleep(5.seconds)
    _ <- fib.interrupt *> ZIO.logInfo("Interrupt invoked!") // <— this will hang: child inherited uninterruptibility
  } yield ()
}
```
Using `ZIO.uninterruptibleMask` at the top level keeps the parent uninterruptible. Inside the mask, calling `restore(ZIO.never)` runs that effect as interruptible so the forked fiber becomes interruptible even though its parent is not. For example:

```scala
val parentInterruptibleChild = ZIO.uninterruptibleMask { restore =>
  for {
    _   <- ZIO.logInfo("Parent is uninterruptible")
    fib <- restore(ZIO.never).fork   // <— child is now interruptible
    _ <- ZIO.logInfo("Attempting to interrupt the child in 5 seconds...")
    _ <- ZIO.sleep(5.seconds)
    _ <- fib.interrupt *> ZIO.logInfo("Interrupt invoked!")
  } yield ()
}
```
:::

In the following example, we create a separate fiber to output a delayed print message and then wait for that fiber to succeed with a value:

```scala

for {
  fiber <- (ZIO.sleep(3.seconds) *>
    printLine("Hello, after 3 second") *>
    ZIO.succeed(10)).fork
  _ <- printLine(s"Hello, World!")
  res <- fiber.join
  _ <- printLine(s"Our fiber succeeded with $res")
} yield ()
```

### interrupt
Whenever we want to get rid of our fiber, we can simply call `interrupt` on that. The interrupt operation does not resume until the fiber has completed or has been interrupted and all its finalizers have been run. These precise semantics allow construction of programs that do not leak resources.

### await
To inspect whether our fiber succeeded or failed, we can call `await` on that fiber. This call will wait for that fiber to terminate, and it will give us back the fiber's value as an `Exit`. That exit value could be failure or success:

```scala

for {
  b <- Random.nextBoolean
  fiber <- (if (b) ZIO.succeed(10) else ZIO.fail("The boolean was not true")).fork
  exitValue <- fiber.await
  _ <- exitValue match {
    case Exit.Success(value) => printLine(s"Fiber succeeded with $value")
    case Exit.Failure(cause) => printLine(s"Fiber failed")
  }
} yield ()
```

`await` is similar to `join`, but they react differently to errors and interruption: `await` always succeeds with `Exit` information, even if the fiber fails or is interrupted. In contrast to that, `join` on a fiber that fails will itself fail with the same error as the fiber, and `join` on a fiber that is interrupted will itself become interrupted.

### Parallelism
To execute actions in parallel, the `zipPar` method can be used:

```scala
def bigCompute(m1: Matrix, m2: Matrix, v: Matrix): UIO[Matrix] =
  for {
    t <- computeInverse(m1).zipPar(computeInverse(m2))
    (i1, i2) = t
    r <- applyMatrices(i1, i2, v)
  } yield r
```

The `zipPar` combinator has resource-safe semantics. If one computation fails, the other computation will be interrupted, to prevent wasting resources.

### Racing

Two actions can be *raced*, which means they will be executed in parallel, and the value of the first action that completes successfully will be returned.

```scala
fib(100) race fib(200)
```

The `race` combinator is resource-safe, which means that if one of the two actions returns a value, the other one will be interrupted, to prevent wasting resources.

The `race` and even `zipPar` combinators are a specialization of a much-more powerful combinator called `raceWith`, which allows executing user-defined logic when the first of two actions succeeds.

On the JVM, fibers will use threads, but will not consume *unlimited* threads. Instead, fibers yield cooperatively during periods of high-contention.

```scala
def fib(n: Int): UIO[Int] =
  if (n <= 1) {
    ZIO.succeed(1)
  } else {
    for {
      fiber1 <- fib(n - 2).fork
      fiber2 <- fib(n - 1).fork
      v2     <- fiber2.join
      v1     <- fiber1.join
    } yield v1 + v2
  }
```

## Error Model
The `ZIO` error model is simple, consistent, permits both typed errors and termination, and does not violate any laws in the `Functor` hierarchy.

A `ZIO[R, E, A]` value may only raise errors of type `E`. These errors are recoverable by using the `either` method.  The resulting effect cannot fail, because the failure case has been exposed as part of the `Either` success case.  

```scala
val error: Task[String] = ZIO.fail(new RuntimeException("Some Error"))
val errorEither: ZIO[Any, Nothing, Either[Throwable, String]] = error.either
```

Separately from errors of type `E`, a fiber may be terminated for the following reasons:

* **The fiber self-terminated or was interrupted by another fiber**. The "main" fiber cannot be interrupted because it was not forked from any other fiber.

* **The fiber failed to handle some error of type `E`**. This can happen only when an `ZIO.fail` is not handled. For values of type `UIO[A]`, this type of failure is impossible.

* **The fiber has a defect that leads to a non-recoverable error**. There are only two ways this can happen:

     1. A partial function is passed to a higher-order function such as `map` or `flatMap`. For example, `io.map(_ => throw e)`, or `io.flatMap(a => throw e)`. The solution to this problem is to not to pass impure functions to purely functional libraries like ZIO, because doing so leads to violations of laws and destruction of equational reasoning.
     2. Error-throwing code was embedded into some value via `ZIO.succeed`, etc. For importing partial effects into `ZIO`, the proper solution is to use a method such as `ZIO.attempt`, which safely translates exceptions into values.

When a fiber is terminated, the reason for the termination, expressed as a `Throwable`, is passed to the fiber's supervisor, which may choose to log, print the stack trace, restart the fiber, or perform some other action appropriate to the context.

A fiber cannot stop its own interruption. However, all finalizers will be run during termination, even when some finalizers throw non-recoverable errors. Errors thrown by finalizers are passed to the fiber's supervisor.

There are no circumstances in which any errors will be "lost", which makes the `ZIO` error model more diagnostic-friendly than the `try`/`catch`/`finally` construct that is baked into both Scala and Java, which can easily lose errors.

## Fiber Interruption

In Java, a thread can be interrupted via `Thread#interrupt` from another thread, but it may refuse the interruption request and continue processing. Unlike Java, in ZIO when a fiber interrupts another fiber, we know that the interruption occurs, and it always works.

When working with ZIO fibers, we should consider these notes about fiber interruptions:

### Interruptible/Uninterruptible Regions

All fibers are interruptible by default. To make an effect uninterruptible we can use `Fiber#uninterruptible`, `ZIO#uninterruptible` or `ZIO.uninterruptible`. ZIO provides also the reverse direction of these methods to make an uninterruptible effect interruptible.

```scala
for {
  fiber <- Clock.currentDateTime
    .flatMap(time => printLine(time))
    .schedule(Schedule.fixed(1.seconds))
    .uninterruptible
    .fork
  _     <- fiber.interrupt // Runtime stuck here and does not go further
} yield ()
```

Note that there is no way to stop interruption. We can only delay it, by making an effect uninterruptible.

### Fiber Finalization on Interruption

When a fiber has done its work or has been interrupted, the finalizer of that fiber is guaranteed to be executed:

```scala
for {
  fiber <- printLine("Working on the first job")
    .schedule(Schedule.fixed(1.seconds))
    .ensuring {
      (printLine(
        "Finalizing or releasing a resource that is time-consuming"
      ) *> ZIO.sleep(7.seconds)).orDie
    }
    .fork
  _     <- fiber.interrupt.delay(4.seconds)
  _     <- printLine(
          "Starting another task when the interruption of the previous task finished"
        )
} yield ()
```

In the above example, the release action takes some time for freeing up resources. So it slows down the call to `interrupt` the fiber.

### Fast Interruption

As we saw in the previous section, the ZIO runtime gets stuck on interruption until the fiber's finalizer finishes its job. We can prevent this behavior by using `ZIO#disconnect` or `Fiber#interruptFork` which perform fiber's interruption in the background or in separate daemon fiber:

Let's try the `Fiber#interruptFork`:

```scala
for {
  fiber <- printLine("Working on the first job")
    .schedule(Schedule.fixed(1.seconds))
    .ensuring {
      (printLine(
        "Finalizing or releasing a resource that is time-consuming"
      ) *> ZIO.sleep(7.seconds)).orDie
    }
    .fork
  _ <- fiber.interruptFork.delay(4.seconds) // fast interruption
  _ <- printLine(
    "Starting another task while interruption of the previous fiber happening in the background"
  )
} yield ()
```

### Interrupting Blocking Operations

The `ZIO#attemptBlocking` is interruptible by default, but its interruption will not translate to JVM thread interruption. Instead, we can use `ZIO#attemptBlockingInterrupt` to translate the ZIO interruption of that effect into JVM thread interruption. For details and examples on interrupting blocking operations see [here](../core/zio/zio.md#blocking-synchronous-side-effects).

### Automatic Interruption

If we never _cancel_ a running effect explicitly, ZIO performs **automatic interruption** for several reasons:

1. **Structured Concurrency** — If a parent fiber terminates, then by default, all child fibers are interrupted, and they cannot outlive their parent. We can prevent this behavior by using `ZIO#forkDaemon` or `ZIO#forkIn` instead of `ZIO#fork`.

2. **Parallelism** — If one effect fails during the execution of many effects in parallel, the others will be canceled. Examples include `foreachPar`, `zipPar`, and all other parallel operators.

3. **Timeouts** — If a running effect with a `timeout` has not been completed in the specified amount of time, then the execution is canceled.

4. **Racing** — The loser of a `race`, if still running, is canceled.

### Joining an Interrupted Fiber

We can join an interrupted fiber, which will cause our fiber to become interrupted. This process preserves the proper finalization of the caller. So, **ZIO's concurrency model respects brackets even when we _join_ an interrupted fiber**:

```scala
val myApp =
  (
    for {
      fiber <- printLine("Running a job").delay(1.seconds).forever.fork
      _     <- fiber.interrupt.delay(3.seconds)
      _     <- fiber.join // Joining an interrupted fiber
    } yield ()
  ).ensuring(
    printLine(
      "This finalizer will be executed without occurring any deadlock"
    ).orDie
  )
```

A fiber that is interrupted because of joining another interrupted fiber will properly finalize; this is a distinction between ZIO and the other effect systems, which _deadlock_ the joining fiber.

## Thread Shifting - JVM
By default, fibers give no guarantees as to which thread they execute on. They may shift between threads, especially as they execute for long periods of time.

Fibers only ever shift onto the thread pool of the runtime system, which means that by default, fibers running for a sufficiently long time will always return to the runtime system's thread pool, even when their (asynchronous) resumptions were initiated from other threads.

For performance reasons, fibers will attempt to execute on the same thread for a (configurable) minimum period, before yielding to other fibers. Fibers that resume from asynchronous callbacks will resume on the initiating thread, and continue for some time before yielding and resuming on the runtime thread pool.

These defaults help to guarantee stack safety and cooperative multitasking. They can be changed in `Runtime` if automatic thread shifting is not desired.

## Types of Workloads
Let's discuss the types of workloads that a fiber can handle. There are three types of them:
1. **CPU Work/Pure CPU Bound** is a workload that exploits the processing power of a CPU for pure computations of a huge chunk of work, e.g. complex numerical computations.
4. **Blocking I/O** is a workload that goes beyond pure computation by doing communication in a blocking fashion. For example, waiting for a certain amount of time to elapse or waiting for an external event to happen are blocking I/O operations.
5. **Asynchronous I/O** is a workload that goes beyond pure computation by doing communication asynchronously, e.g. registering a callback for a specific event.

### CPU Work
What we refer to as CPU Work is pure computational firepower without involving any interaction and communication with the outside world. It doesn't involve any I/O operation. It's a pure computation. By I/O, we mean anything that involves reading from and writing to an external resource such as a file or a socket or web API, or anything that would be characterized as I/O. 

Fibers are designed to be **cooperative** which means that **they will yield to each other as required to preserve some level of fairness**. If we have a fiber that is doing CPU Work which passes through one or more ZIO operations such as `flatMap` or `map`, as long as there exists a touchpoint where the ZIO runtime system can sort of keep a tab on that ongoing CPU Work then that fiber will yield to other fibers. These touchpoints allow many fibers who are doing CPU Work to end up sharing the same thread.

What if though, we have a CPU Work operation that takes a really long time to run? Let's say 30 seconds it does pure CPU Work very computationally intensive? What happens if we take that single gigantic function and put that into a `ZIO#attempt`? In that case there is no way for the ZIO Runtime to force that fiber to yield to other fibers. In this situation, the ZIO Runtime cannot preserve some level of fairness, and that single big CPU operation monopolizes the underlying thread. It is not a good practice to monopolize the underlying thread.

ZIO has a special thread pool that can be used to do these computations. That's the **blocking thread pool**. The `ZIO#blocking` operator and its variants (see [here](../core/zio/zio.md#blocking-synchronous-side-effects)) can be used to run a big CPU Work on a dedicated thread. So, it doesn't interfere with all the other work that is going on simultaneously in the ZIO Runtime system.

If a CPU Work doesn't yield quickly, then that is going to monopolize a thread. So how can we determine that our CPU Work can yield quickly or not? 
- If that overall CPU Work composes many ZIO operations, then due to the composition of ZIO operations, it has a chance to yield quickly to other fibers and doesn't monopolize a thread.
- If that CPU work doesn't compose any ZIO operations, or we lift that from a legacy library, then the ZIO Runtime doesn't have any chance of yielding quickly to other fibers. So this fiber is going to monopolize the underlying thread.

The best practice is to run those huge CPU Work on a dedicated thread pool, by lifting them with `ZIO#blocking`.

:::note

So as a rule of thumb, when we have a huge CPU Work that is not chunked with built-in ZIO operations, and thus going to monopolize the underlying thread, we should run that on a dedicated thread pool that is designed to perform CPU-driven tasks.
:::

### Blocking I/O
Inside Java, there are many methods that will put our thread to sleep. For example, if we call `read` on a socket and there is nothing to read right now because not enough bytes have been read from the other side over the TCP/IP protocol, then that will put our thread to sleep. 

Most of the I/O operations and certainly all the classic I/O operations like `InputStream` and `OutputStream` are utilizing a locking mechanism that will `park` a thread. When we `write` to `OutputStream`, this method will `park` the thread and `wait` until the data has actually been written to file. It is the same way for `read` and similar blocking operations. Anytime we use a `lock`, anything in `java.util.concurrent.locks`, all those locks use this mechanism. All these operations are called blocking because they `park` the thread that is doing the work, and the thread that's doing the work goes to sleep.

:::note
What we refer to as blocking I/O is not necessarily just an I/O operation. Remember every time we use a `lock` we are also `park`ing a thread. It goes to `sleep`, and it has to be woken up again. We refer to this entire class of operations as **blocking I/O**.
:::

There are multiple types of overhead associated with parking a thread:

1. When we `park` a thread then that thread is still consuming resources, it's still obviously consuming stack resources, the heap, and all metadata associated with the underlying thread in the JVM.

2. Since every JVM thread corresponds to an operating system level thread, there's a large amount of overhead even inside the operating system. Every thread has a pre-allocated stack size so that memory is reserved even if that thread is not doing any work. That memory is sort of reserved for the thread, and we cannot touch it. 

3. Besides, the actual process of putting a thread to sleep and then later waking it up again is computationally intensive. It slows down our computations.

This is why it has become a sort of best practice and part of the architectural pattern of reactive applications to design what is called **non-blocking application**. Non-blocking is synonymous with asynchronous. Non-blocking and asynchronous and to some degree even reactive, they're all trying to get at something which is what we want: scalable applications.

Scalable applications cannot afford to have thousands of threads just sitting around doing nothing and just consuming work and taking a long time to wake up again. We cannot do blocking I/O in scalable applications. It is considered an anti-pattern because it is not efficient. That is not a way to build scalable applications, but nonetheless, we have to support that use case.

Today, we have lots of common Java libraries that use blocking constructs, like `InputStream` and `OutputStream`, and `Reader` and `Writer`. Also, the JDBC is entirely blocking. The only way of doing database I/O in Java is blocking. So obviously, we have to do blocking I/O. We can do blocking I/O from a fiber. So is it best practice? No, it should be avoided whenever possible, but the reality is we have to do blocking I/O. 

Whenever we lift a blocking I/O operation into ZIO, the ZIO Runtime is executing a fiber that is doing blocking I/O. The underlying thread will be parked, and it has to be woken up later. It doesn't have any ability to avoid this. It can't stop an underlying thread from being parked. That's just the way these APIs are designed. So, we have to block. There's no way around that. That fiber will be monopolizing the underneath thread and therefore that thread is not available for performing all the work of the other fibers in the system. So that can be a bottleneck point in our application. 

And again, the solution to this problem is the same as the solution to the first class of problems, the CPU Work. The solution is to run this action **using the blocking thread pool** in ZIO which will ensure that this blocking code executes on its dedicated thread pool. **So it doesn't interfere or compete with the threads that are used for doing the bulk of work in your application**. So basically ZIO's philosophy is that if we _have_ to do CPU Work or blocking synchronous work that's ok we can do that. Just we need to do it in the right place. So it doesn't interfere with our primary thread pool.

ZIO has one primary built-in fixed thread pool. This sort of workhorse thread pool is designed to be used for the majority of our application requirements. It has a certain number of threads in it and that stays constant over the lifetime of our application.

Why is that the case? Well because for the majority of workloads in our applications, it does not actually help things to create more threads than the number of CPU cores. If we have eight cores, it does not accelerate any sort of processing to create more than eight threads. Because at the end of the day our hardware is only capable of running eight things at the same time. 

If we create a thousand threads on a system that can only run eight of them in parallel at a time, then what does the operating system do? As we have not enough CPU cores, the operating system starts giving a little slice of the eight cores to all these threads by switching between them over and over again.

The overhead for context switching between threads is significant. The CPU has to load in new registers, refill all its caches, it has to go through all these crazy complex processes that interfere with its main job to get stuff done. There's significant overhead associated with that. As a result, it's not going to be very efficient. We are going to waste a lot of our time and resources just switching back and forth between all these threads, that would kill our application.

So for that reason, ZIO's default thread pool is fixed with a number of threads equal to the number of CPU cores. That is the best practice. That means that no matter how much work we create if we create a hundred thousand fibers, they will still run on a fixed number of threads.

Let's say we do blocking I/O on the main ZIO thread pool, so we have got eight threads all sitting and parked on a socket read. What happens to all the other 100000 fibers in our system? They line up in a queue waiting for their chance to run. That's not ideal. That's why we should take these effects that either do blocking I/O, or they do big CPU Work that's not chunked and run them using ZIO's blocking thread pool which will give us a dedicated thread. 

That dedicated thread is not efficient, but again, sometimes we have to interact with legacy code and legacy code is full of blocking code. We just need to be able to handle that gracefully and ZIO does that using the blocking thread pool.

### Asynchronous I/O
The third category is asynchronous I/O, and we refer to it as Async Work. Async Work is code that whenever it runs into something that it needs to wait on, instead of blocking and parking the thread, it registers a callback, and returns immediately.

 It allows us to register a callback and when that result is available then our callback will be invoked. Callbacks are the fundamental way by which all async code on the JVM works. There is no mechanism in the JVM right now to support async code natively, but once that would happen in the future, probably in the Loom project, it will simplify a lot of things.

But for now, in current days, sort of all published JVM versions have no such thing. The only way we can get a non-blocking async code is to have this callback registering mechanism.

Callbacks have the pro that they don't wait for CPU. Instead of waiting to read the next chunk from a socket or instead of waiting to acquire a lock, all we have to do is call it and give it a callback. It doesn't need to do anything else. It can return control to the thread pool and then later on when that data has been read from the socket or when that lock has been acquired or when that amount of time has elapsed if we're sleeping for a certain amount of time, then our callback can be invoked. 

It has the potential to be extraordinarily efficient. The drawback of callbacks is they are not so pretty and fun to work with. They don't compose well with `try` / `finally` constructs. Error handling is really terrible, we have to do error propagation on our own. So that is what gave rise to data types like `Future` which eliminates the need for callbacks. By using `Future`s we can wrap callback-based APIs and get the benefits of async but without the callbacks. Also, it supports for-comprehension, so we can structure our code as a nice linear sequence. 

Similarly, in ZIO we never see a callback with ZIO, but fundamentally everything boils down to asynchronous operations on the JVM in a callback fashion. Callback base code is obscenely efficient, but it is extraordinarily painful to deal with directly. Data types like `Future` and `ZIO` allow us to avoid even seeing a callback in our code. 

With ZIO, we do not have to think about callbacks, unless sometimes, when we need to integrate with legacy code. ZIO has an appropriate constructor to turn that ugly callback-based API into a ZIO effect. It is the `async` constructor.

Most of the ZIO operations that one would expect to be blocking do actually not block the underlying thread, but they offer blocking semantics managed by ZIO. For example, every time we see something like `ZIO.sleep` or when we take something from a queue (`queue.take`) or offer something to a queue (`queue.offer`) or if we acquire a permit from a semaphore (`semaphore.withPermit`) and so forth, we are just blocking semantically without actually blocking an underlying thread. If we use the corresponding methods in Java, like `Thread.sleep` or any of its `lock` machinery, then those methods are going to block a thread. So this is why we say that ZIO is 100% non-blocking, while Java threads are not.

All of the pieces of machinery that ZIO gives us are 100% asynchronous and non-blocking. As they don't block and monopolize the thread, all of the async work is executed on the primary thread pool in ZIO.

---

## FiberId

`FiberId` is the identity of a [Fiber](fiber.md), described by a globally unique sequence number and the time when it began life:
  * `id` — unique monotonically increasing sequence number `0,1,2,...`, derived from an atomic counter
  * `startTimeSeconds` — UTC time seconds, derived from `java.lang.System.currentTimeMillis / 1000`

---

## Fiber.Status

`Fiber.Status` describes the current status of a [Fiber](fiber.md).

Each fiber can be in one of the following status:

- Done
- Running
- Suspended

In the following example, we are going to `await` on a never-ending fiber and determine the id of that fiber, which we are blocking on:

```scala

for {
  f1 <- ZIO.never.fork
  f2 <- f1.await.fork
  blockingOn <- f2.status
    .collect(()) { case Fiber.Status.Suspended(_, _, blockingOn) =>
      blockingOn
    }
    .eventually
} yield (assert(blockingOn == f1.id))
```

---

## Introduction to ZIO Fibers

A Fiber can be thought of as a virtual thread. A Fiber is the analog of a Java thread (`java.lang.Thread`), but it performs much better. Fibers are implemented in such a fashion that a single JVM thread will execute many fibers. We can think of fibers as unbounded JVM threads.

> **Warning** if you are not an experienced ZIO programmer:
>
> You should avoid using fibers manually. ZIO gives you many concurrent primitives like `raceWith`, `zipPar`, `foreachPar`, and so forth, which utilize fibers under the hood without any manual effort required.
>
> Fibers, just like threads, are low-level constructs. It's not generally recommended to deal with them in your code directly. It is very easy to make lots of mistakes or to introduce performance problems by manually using them.

## Why Fibers?

There are some limitations with JVM threads:

1. **Threads are scarce** — Threads on the JVM map to the operating system level threads which imposes an upper bound on the number of threads that we can have inside our application.
2. **Expensive on creation** — The creation of threads is expensive in terms of time and memory complexity.
3. **Much Overhead on Context Switching** — Switching between the execution of one thread to another thread is not cheap, it takes a lot of time.
4. **Lack of Composability** — Threads are not typed. They don't have a meaningful return type. In Java, when we create a thread, we have to provide a `run` function that returns void. So threads cannot finish with any specific value. Due to this limitation, we cannot compose threads. Also, a thread has no type parameter for error. It is expected to throw any exception of type `Throwable` to signal errors.

In the following sections, we are going to discuss the key features of fibers, and how fibers overcome the drawbacks of Java threads.

### Unbounded Size

So whereas the mapping from JVM threads to operating system threads is one-to-one, **the mapping of fibers to threads is many-to-one**. Each JVM thread will end up executing anywhere from hundreds to thousands or even tens of thousands of fibers concurrently, by hopping back and forth between them as necessary. This gives us virtual threads that have the benefits of threads, but the scalability way beyond threads. In other words, fibers offer us massive concurrent **lightweight green threading** on the JVM.

As a rough rule of thumb, we can have an application with a thousand real threads. No problem, modern servers can support applications with a thousand threads. However, we cannot have an application with a hundred thousand threads, that application will die. That just won't make any progress. The JVM nor our operating system can physically support a hundred thousand threads. However, it is no problem to have a Scala application with a hundred thousand fibers. Such an application can still perform in a very high-performance fashion, and the miracle that enables that to happen is fiber.

### Lightweight

**JVM threads are expensive to create in terms of time and memory complexity.** Also it takes a lot of time to switch between one thread of execution to another. In contrast to that, fibers are virtual, and as they use **green threading**, they are considered to be **lightweight cooperative threads**. This means that fibers always _yield_ their executions to each other without the overhead of preemptive scheduling.

### Asynchronous

Fibers are asynchronous, while threads are always synchronous. That is why fibers scale better than threads.

### Typed and Composable

**Fibers have typed error and success values**. A fiber has two type parameters, `E` and `A`:

- The `E` corresponds to the error channel. It indicates the error type with which the fiber can fail.

- The `A` corresponds to the success value of the computation. That is the type with which the fiber can succeed.

The fact that fibers are typed allows us to write more type-safe programs. Also, it increases the compositional properties of our programs because we can wait on a fiber to finish and then expect to receive a value of type `A`.

### Interrupt Safe

Threads in Java can be terminated via the stop method, but this is not a safe operation. The stop operation has been [deprecated](https://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html). So this is not a safe way to force kill a thread. Instead, we should try to request an interruption of the thread, but in this case, **the thread may not respond to our request, and it may just go forever**.

**Fiber has a safe version of this functionality that works very well**. Just like we can interrupt a thread, we can interrupt a fiber too, but interruption of fibers is much more reliable. It will always work, and **it probably works very fast**. We don't need to wait around, we can just try to interrupt them, and they will be gone very soon.

### Structured Concurrency

With fibers, we can have hundreds of thousands and even millions of fibers that are started and working together. So we can reach a very massive concurrency with fibers. Now how can we manage all these fibers? Some of them are top-level fibers and some others are forked and become children of their parents. How can we manage their scopes, how to keep track of all fibers, and prevent them to leak? What happens to the execution of a child fiber if its parent execution is interrupted? The child fibers should be scoped to their parent fibers. We need a way to manage these scopes automatically. This is where structured concurrency shines.

> _**Important**:_
>
> It's worth mentioning that in the ZIO model, all code runs on fibers. There is no such thing as code that is executed outside of fibers. When we create a main function in ZIO that returns an effect, then even if we don't explicitly fork a fiber, the effect will be executed on what is called the main fiber. It's a top-level fiber.
>
>It's just like if we have a main function in Java then that main function will execute on the main thread. There is no code in Java that does not execute on a thread. All code executes on a thread even if you didn't create a thread.

ZIO provides structured concurrency. The way ZIO's structured concurrency works is that **the child fibers are scoped to their parent fibers** which means that **when the parent effect finishes execution, then all childs' effects will be automatically interrupted**. So when we fork, and we get back a fiber, the fiber's lifetime is bound to the parent fiber that forked it. It is almost impossible to leak fibers because child fibers are guaranteed to complete before their parents.

The structured concurrency gives us a way to reason about fiber lifespans. We can statically reason about the lifetimes of children fibers just by looking at our code. We don't need to insert complicated logic to keep track of all the child fibers and manually shut them down.

#### Global Lifetime

Sometimes we want a child fiber to outlive the scope of the parent. What can we do in that case? Well, ZIO offers an operator called `forkDaemon` which forks the fiber as a daemon fiber. Daemon fibers can outlive their parents. They can live forever. They run in the background doing their work until they end with failure or success. This gives us a way to spawn background jobs that should just keep on going regardless of what happens to the parent.

#### Fine-grained Scope

If we need a very flexible fine-grained control over the lifetime of a fiber there is another operator called `forkin`. We can fork a fiber inside a specific scope, and when that scope is closed then the fiber will be terminated.

## Fiber Data Types

ZIO fiber contains a few data types that can help us to solve complex problems:

- **[Fiber](fiber.md)** — A fiber value models an `IO` value that has started running, and is the moral equivalent of a green thread.
- **[Fiber.Status](fiberstatus.md)** — `Fiber.Status` describes the current status of a Fiber.
- **[FiberId](fiberid.md)** — `FiberId` describes the unique identity of a Fiber.

---

## Introduction

ZIO contains a few data types that can help you solve complex problems in asynchronous and concurrent programming. ZIO data types categorize into these sections:

1. [Core Data Types](#core-data-types)
2. [Contextual Data Types](#contextual-data-types)
3. [State Management](#state-management)
4. [Concurrency](#concurrency)
    - [Fiber Primitives](#fiber-primitives)
    - [Concurrency Primitives](#concurrency-primitives)
    - [Synchronization Aids](#synchronization-aids)
    - [STM](#stm)
5. [Resource Management](#resource-management)
6. [Streaming](#streaming)
7. [Metrics](#metrics)
8. [Testing](#testing)
9. [Miscellaneous](#miscellaneous)

## Core Data Types
- **[ZIO](core/zio/zio.md)** — `ZIO` is a value that models an effectful program, which might fail or succeed.
    + **[UIO](core/zio/uio.md)** — `UIO[A]` is a type alias for `ZIO[Any, Nothing, A]`.
    + **[URIO](core/zio/urio.md)** — `URIO[R, A]` is a type alias for `ZIO[R, Nothing, A]`.
    + **[Task](core/zio/task.md)** — `Task[A]` is a type alias for `ZIO[Any, Throwable, A]`.
    + **[RIO](core/zio/rio.md)** — `RIO[R, A]` is a type alias for `ZIO[R, Throwable, A]`.
    + **[IO](core/zio/io.md)** — `IO[E, A]` is a type alias for `ZIO[Any, E, A]`.
- **[ZIOApp](core/zioapp.md)** — `ZIOApp` and the `ZIOAppDefault` are entry points for ZIO applications.
- **[Runtime](core/runtime.md)** — `Runtime[R]` is capable of executing tasks within an environment `R`.
- **[Exit](core/exit.md)** — `Exit[E, A]` describes the result of executing an `IO` value.
- **[Cause](core/cause.md)** — `Cause[E]` is a description of a full story of a fiber failure.

## Contextual Data Types

- **[ZEnvironment](contextual/zenvironment.md)** — `ZEnvironment[R]` is a built-in type-level map for the `ZIO` data type which is responsible for maintaining the environment of a `ZIO` effect.
- **[ZLayer](contextual/zlayer.md)** — `ZLayer[-RIn, +E, +ROut]` is a recipe to build an environment of type `ROut`, starting from a value `RIn`, and possibly producing an error `E` during creation.
    + **[RLayer](contextual/rlayer.md)** — `RLayer[-RIn, +ROut]` is a type alias for `ZLayer[RIn, Throwable, ROut]`, which represents a layer that requires `RIn` as its input, it may fail with `Throwable` value, or returns `ROut` as its output.
    + **[ULayer](contextual/ulayer.md)** — `ULayer[+ROut]` is a type alias for `ZLayer[Any, Nothing, ROut]`, which represents a layer that doesn't require any services as its input, it can't fail, and returns `ROut` as its output.
    + **[Layer](contextual/layer.md)** — `Layer[+E, +ROut]` is a type alias for `ZLayer[Any, E, ROut]`, which represents a layer that doesn't require any services, it may fail with an error type of `E`, and returns `ROut` as its output.
    + **[URLayer](contextual/urlayer.md)** — `URLayer[-RIn, +ROut]` is a type alias for `ZLayer[RIn, Nothing, ROut]`, which represents a layer that requires `RIn` as its input, it can't fail, and returns `ROut` as its output.
    + **[TaskLayer](contextual/task-layer.md)** — `TaskLayer[+ROut]` is a type alias for `ZLayer[Any, Throwable, ROut]`, which represents a layer that doesn't require any services as its input, it may fail with `Throwable` value, and returns `ROut` as its output.

## State Management

- **[ZState](state-management/zstate.md)**— It models a state that can be read from and written to during the execution of an effect.
- **[Ref](state-management/global-shared-state.md)**— `Ref[A]` models a mutable reference to a value of type `A`.
- **[FiberRef](state-management/fiberref.md)**— `FiberRef[A]` models a mutable reference to a value of type `A`. As opposed to `Ref[A]`, a value is bound to an executing `Fiber` only.  You can think of it as Java's `ThreadLocal` on steroids.

## Concurrency

### Fiber Primitives

- **[Fiber](fiber/fiber.md)** — A fiber value models an `IO` value that has started running, and is the moral equivalent of a green thread.
- **[Fiber.Status](fiber/fiberstatus.md)** — `Fiber.Status` describe the current status of a Fiber.
- **[FiberId](fiber/fiberid.md)** — `FiberId` describe the unique identity of a Fiber.

### Concurrency Primitives

- **[Hub](concurrency/hub.md)** — A `Hub` is an asynchronous message hub that allows publishers to efficiently broadcast values to many subscribers.
- **[Promise](concurrency/promise.md)** — A `Promise` is a model of a variable that may be set a single time, and awaited on by many fibers.
- **[Semaphore](concurrency/semaphore.md)** — A `Semaphore` is an asynchronous (non-blocking) semaphore that plays well with ZIO's interruption.
- **[Ref](concurrency/ref.md)** — `Ref[A]` models a mutable reference to a value of type `A`. The two basic operations are `set`, which fills the `Ref` with a new value, and `get`, which retrieves its current content. All operations on a `Ref` are atomic and thread-safe, providing a reliable foundation for synchronizing concurrent programs.
- **[Ref.Synchronized](concurrency/refsynchronized.md)** — `Ref.Synchronized[A]` models a **mutable reference** to a value of type `A` in which we can store **immutable** data, and update it atomically **and** effectfully.
- **[Queue](concurrency/queue.md)** — A `Queue` is an asynchronous queue that never blocks, which is safe for multiple concurrent producers and consumers.

### Synchronization Aids

- **[ReentrantLock](sync/reentrantlock.md)**— The `ReentrantLock` is a synchronization tool that is useful for synchronizing blocks of code.
- **[CountdownLatch](sync/countdownlatch.md)** — A synchronization aid that allows one or more fibers to wait until a set of operations being performed in other fibers completes.
- **[CyclicBarrier](sync/cyclicbarrier.md)** — A synchronization aid that allows a set of fibers to all wait for each other to reach a common barrier point.
- **[ConcurrentMap](sync/concurrentmap.md)** — A Map wrapper over `java.util.concurrent.ConcurrentHashMap`
- **[ConcurrentSet](sync/concurrentset.md)** — A Set implementation over `java.util.concurrent.ConcurrentHashMap`

### STM

- **[STM](stm/stm.md)** — An `STM` represents an effect that can be performed transactionally resulting in a failure or success.
- **[TArray](stm/tarray.md)** — A `TArray` is an array of mutable references that can participate in transactions.
- **[TSet](stm/tset.md)** — A `TSet` is a mutable set that can participate in transactions.
- **[TMap](stm/tmap.md)** — A `TMap` is a mutable map that can participate in transactions.
- **[TRef](stm/tref.md)** — A `TRef` is a mutable reference to an immutable value that can participate in transactions.
- **[TPriorityQueue](stm/tpriorityqueue.md)** — A `TPriorityQueue` is a mutable priority queue that can participate in transactions.
- **[TPromise](stm/tpromise.md)** — A `TPromise` is a mutable reference that can be set exactly once and can participate in transactions.
- **[TQueue](stm/tqueue.md)** — A `TQueue` is a mutable queue that can participate in transactions.
- **[TReentrantLock](stm/treentrantlock.md)** — A `TReentrantLock` is a reentrant read / write lock that can be composed.
- **[TSemaphore](stm/tsemaphore.md)** — A `TSemaphore` is a semaphore that can participate in transactions.

## Resource Management

- **[Scope](resource/scope.md)** — A scope in which resources can safely be used.
- **[ZPool](resource/zpool.md)** — An asynchronous and concurrent generalized pool of reusable resources.

## Streaming

- **[ZStream](stream/zstream/index.md)** — `ZStream` is a lazy, concurrent, asynchronous source of values.
    + **Stream** — `Stream[E, A]` is a type alias for `ZStream[Any, E, A]`, which represents a ZIO stream that does not require any services, and may fail with an `E`, or produce elements with an `A`.
- **[ZSink](stream/zsink/index.md)** — `ZSink` is a consumer of values from a `ZStream`, which may produce a value when it has consumed enough.
    + **[Sink](stream/zsink/index.md)** — `Sink[InErr, A, OutErr, L, B]` is a type alias for `ZSink[Any, InErr, A, OutErr, L, B]`.
- **[ZPipeline](stream/zpipeline.md)** — `ZPipeline` is a polymorphic stream transformer.
- **[SubscriptionRef](stream/subscriptionref.md)** — `SubscriptionRef[A]` contains a current value of type `A` and a stream that can be consumed to observe all changes to that value.

## Metrics

IO supports 5 types of Metrics:

- **[Counter](observability/metrics/counter.md)** — The Counter is used for any value that increases over time like _request counts_.
- **[Gauge](observability/metrics/gauge.md)** — The gauge is a single numerical value that can arbitrary goes up or down over time like _memory usage_.
- **[Histogram](observability/metrics/histogram.md)** — The Histogram is used to track the distribution of a set of observed values across a set of buckets like _request latencies_.
- **[Summary](observability/metrics/summary.md)** — The Summary represents a sliding window of a time series along with metrics for certain percentiles of the time series, referred to as quantiles like _request latencies_.
- **[Frequency](observability/metrics/frequency.md)** — The Frequency is a metric that counts the number of occurrences of distinct string values.

## Testing

- **[Spec](test/spec.md)**— A `Spec[R, E]` is the backbone of ZIO Test. All specs require an environment of type `R` and may potentially fail with an error of type `E`.
- **[Assertion](test/assertions/index.md)**— An `Assertion[A]` is a test assertion that can be used to assert the predicate of type `A => Boolean`.
- **[TestAspect](test/aspects/index.md)**— A `TestAspect` is an aspect that can be weaved into specs. We can think of an aspect as a polymorphic function, capable of transforming one test into another.
- **[Gen](test/property-testing/built-in-generators.md)**— A `Gen[R, A]` represents a generator of values of type `A`, which requires an environment `R`.
- **Test Service**— ZIO Test has the following out-of-the-box test services:
    - **[TestConsole](test/services/console.md)**— It allows testing of applications that interact with the console.
    - **[TestClock](test/services/clock.md)**— We can deterministically and efficiently test effects involving the passage of time without actually having to wait for the full amount of time to pass.
    - **[TestRandom](test/services/random.md)**— This service allows us having fully deterministic testing of code that deals with Randomness.
    - **[TestSystem](test/services/system.md)**— It supports deterministic testing of effects involving system properties.
    - **[Live](test/services/live.md)**— It provides access to the live environment from within the test environment for effects.
    - **[TestConfig](test/services/test-config.md)**— It provides access to default configuration settings used by ZIO Test.
    - **[Sized](test/services/sized.md)**— It enables _Sized Generators_ to access the size from the ZIO Test environment.

## Miscellaneous

- **[Chunk](stream/chunk.md)**— `Chunk` is a fast, pure alternative to Arrays.
- **[Supervisor](observability/supervisor.md)**— `Supervisor[A]` is allowed to supervise the launching and termination of fibers, producing some visible value of type `A` from the supervision.

To learn more about these data types, please explore the pages above, or check out the Scaladoc documentation.

---

## Introduction to ZIO's Interruption Model

While developing concurrent applications, there are several cases that we need to _interrupt_ the execution of other fibers, for example:

1. A parent fiber might start some child fibers to perform a task, and later the parent might decide that it doesn't need the result of some or all of the child fibers.
2. Two or more fibers start a race with each other. The fiber whose result is computed first wins and all other fibers are no longer needed so they should be interrupted.
3. In interactive applications, a user may want to stop some already running tasks, such as clicking on the "stop" button to prevent downloading more files.
4. Computations that run longer than expected should be aborted by using timeout operations.
5. When we have an application that perform compute-intensive tasks based on the user inputs, if the user changes the input we should cancel the current task and perform another one.

## Polling vs. Asynchronous Interruption

A simple and naive way to implement fiber interruption is to provide a mechanism for one fiber to _kill/terminate_ another fiber. This is not a correct solution because if the target fiber is in the middle of changing a shared state it leads to an inconsistent state. So this solution doesn't guarantee to leave the shared mutable state internally consistent.

Other than the very simple kill solution, there are two popular valid solutions to this problem:

  1. **Semi-asynchronous Interruption (Polling for Interruption)**— Imperative languages such as Java often use polling to implement a semi-asynchronous signaling mechanism. In this model, a fiber sends a request for interruption of other fiber. The target fiber keeps polling the interrupt status, and based on the interrupt status will find out that whether there is an interruption request from other fibers. If so, it should terminate itself as soon as possible.

  Using this solution, the fiber itself takes care of critical sections. So while a fiber is in the middle of a critical section, if it receives an interruption request it should ignore the interruption and postpone the delivery of interruption during the critical section.

  The drawback of this solution is that, if the programmer forgets to poll regularly enough, then the target fiber becomes unresponsive and causes deadlocks. Another problem is that polling a global flag is not a functional operation and doesn't fit with ZIO's paradigm.

  2. **Asynchronous Interruption**— In asynchronous interruption, a fiber is allowed to terminate another fiber. So the target fiber is not responsible for polling the status, instead in critical sections the target fiber disables the interruptibility of these regions. This is a purely-functional solution and doesn't require polling a global state. ZIO uses this solution for its interruption model. It is a fully asynchronous signalling mechanism.

  This mechanism doesn't have the drawback of forgetting to poll regularly and also it's fully compatible with the functional paradigm because in a purely-functional computation, we can abort the computation at any point, except for critical sections.

## When Does a Fiber Get Interrupted?

There are several ways and situations that fibers can be interrupted. In this section we will introduce each one with an example of how to reproduce these situations:

### Calling `Fiber#interrupt` Operator

A fiber can be interrupted by calling `Fiber#interrupt` on that fiber.

Let's try to make a fiber and then interrupt it:

```scala

object MainApp extends ZIOAppDefault {
  def task = {
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _ <- ZIO.debug(s"$fn starts a long running task")
      _ <- ZIO.sleep(1.minute)
      _ <- ZIO.debug("done!")
    } yield ()
  }

  def run =
    for {
      f <-
        task.onInterrupt(
          ZIO.debug(s"Task interrupted while running")
        ).fork
      _ <- f.interrupt
    } yield ()
}
```

Here is the output of running this piece of code, which denotes that the task was interrupted:

```
Task interrupted while running
```

### Interruption of Parallel Effects

When composing multiple parallel effects, when one of them is interrupted the other fibers will be interrupted also. So if we have two parallel tasks, if one of them fails or gets interrupted, the other will be interrupted:

```scala

object MainApp extends ZIOAppDefault {

  def debugInterruption(taskName: String) = (fibers: Set[FiberId]) =>
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _ <- ZIO.debug(
        s"The $fn fiber which is the underlying fiber of the $taskName task " +
          s"interrupted by ${fibers.map(_.threadName).mkString(", ")}"
      )
    } yield ()

  def task[R, E, A](name: String)(zio: ZIO[R, E, A]): ZIO[R, E, A] =
    zio.onInterrupt(debugInterruption(name))

  def debugMainFiber =
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _ <- ZIO.debug(s"Main fiber ($fn) starts executing the whole application.")
    } yield ()

  def run = {
    // self interrupting fiber 
    val first = task("first")(ZIO.interrupt)

    // never ending fiber
    val second = task("second")(ZIO.never)

    debugMainFiber *> {
      // uncomment each line and run the code to see the result

      // first fiber will be interrupted 
      first *> second

      // never ending application
      // second *> first

      // first fiber will be interrupted
      // first <*> second

      // never ending application
      // second <*> first

      // first and second will be interrupted
      // first <&> second

      // first and second will be interrupted 
      // second <&> first
    }
  }

}
```

In the above code the `first <&> second` is a parallel composition of the `first` and `second` tasks. When we run them together, the `zipWithPar`/`<&>` operator will run these two tasks in two parallel fibers. If either side of this operator fails or is interrupted the other side will be interrupted.

### Child Fibers Are Scoped to Their Parents

1. If a child fiber does not complete its job or does not join its parent before the parent has completed its job, the child fiber will be interrupted:

```scala

object MainApp extends ZIOAppDefault {
  def debugInterruption(taskName: String) = (fibers: Set[FiberId]) =>
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _  <- ZIO.debug(
              s"the $fn fiber which is the underlying fiber of the $taskName task " +
              s"interrupted by ${fibers.map(_.threadName).mkString(", ")}"
            )
    } yield ()

  def run =
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _  <- ZIO.debug(s"$fn starts working.")
      child =
        for {
          cfn <- ZIO.fiberId.map(_.threadName)
          _   <- ZIO.debug(s"$cfn starts working by forking from its parent ($fn)")
          _   <- ZIO.never
        } yield ()
      _  <- child.onInterrupt(debugInterruption("child")).fork
      _  <- ZIO.sleep(1.second)
      _  <- ZIO.debug(s"$fn finishes its job and is going go exit.")
    } yield ()
    
}
```

Here is the result of one of the executions of this sample code:

```
zio-fiber-2 starts working.
zio-fiber-7 starts working by forking from its parent (zio-fiber-2)
zio-fiber-2 finishes its job and is going to exit.
the zio-fiber-7 fiber which is the underlying fiber of the child task interrupted by zio-fiber-2
```

2. If a parent fiber is interrupted, all its children will be interrupted:

```scala

object MainApp extends ZIOAppDefault {

  def debugInterruption(taskName: String) = (fibers: Set[FiberId]) =>
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _ <- ZIO.debug(
        s"The $fn fiber which is the underlying fiber of the $taskName task " +
          s"interrupted by ${fibers.map(_.threadName).mkString(", ")}"
      )
    } yield ()

  def task =
    for {
      fn <- ZIO.fiberId.map(_.threadName)
      _ <- ZIO.debug(s"$fn starts running that will print random numbers and booleans")
      f1 <- Random.nextIntBounded(100)
        .debug("random number ")
        .schedule(Schedule.spaced(1.second).forever)
        .onInterrupt(debugInterruption("random number"))
        .fork
      f2 <- Random.nextBoolean
        .debug("random boolean ")
        .schedule(Schedule.spaced(2.second).forever)
        .onInterrupt(debugInterruption("random boolean"))
        .fork
        _ <- f1.join
        _ <- f2.join
    } yield ()

  def run =
    for {
      f <- task.fork
      _ <- ZIO.sleep(5.second)
      _ <- f.interrupt
    } yield ()
}
```

Here is one sample output for this program:

```
zio-fiber-7 starts running that will print random numbers and booleans
random number : 65
random boolean : true
random number : 51
random number : 46
random boolean : true
random number : 30
The zio-fiber-9 fiber which is the underlying fiber of the random boolean task interrupted by zio-fiber-7
The zio-fiber-8 fiber which is the underlying fiber of the random number task interrupted by zio-fiber-7
```

## Blocking Operations

### Interruption of Blocking Operations

By default, when we convert a blocking operation into a ZIO effect using `attemptBlocking`, there is no guarantee that if that effect is interrupted the underlying effect will be interrupted.

Let's create a blocking effect from an endless loop:

```scala

for {
  _ <- Console.printLine("Starting a blocking operation")
  fiber <- ZIO.attemptBlocking {
    while (true) {
      Thread.sleep(1000)
      println("Doing some blocking operation")
    }
  }.ensuring(
    Console.printLine("End of a blocking operation").orDie
  ).fork
  _ <- fiber.interrupt.schedule(
    Schedule.delayed(
      Schedule.duration(1.seconds)
    )
  )
} yield ()
```

When we interrupt this loop after one second it will still not stop. It will only stop when the entire JVM stops. The `attemptBlocking` doesn't translate the ZIO interruption into thread interruption (`Thread.interrupt`).

Instead, we should use `attemptBlockingInterrupt` to create interruptible blocking effects:

```scala

for {
  _ <- Console.printLine("Starting a blocking operation")
  fiber <- ZIO.attemptBlockingInterrupt {
    while(true) {
      Thread.sleep(1000)
      println("Doing some blocking operation")
    }
  }.ensuring(
     Console.printLine("End of the blocking operation").orDie
   ).fork
  _ <- fiber.interrupt.schedule(
    Schedule.delayed(
      Schedule.duration(3.seconds)
    )
  )
} yield ()
```

Notes:

1. If we are converting a blocking I/O to a ZIO effect, it would be better to use `attemptBlockingIO` which refines the error type to `java.io.IOException`.

2. The `attemptBlockingInterrupt` method adds significant overhead. So for performance-sensitive applications, it is better to handle interruptions manually using `attemptBlockingCancelable`.

### Cancellation of Blocking Operation

Some blocking operations do not respect `Thread#interrupt` by swallowing `InterruptedException`. So they will not be interrupted via `attemptBlockingInterrupt`. Instead, they may provide us an API to signal them to _cancel_ their operation.

The following `BlockingService` will not be interrupted in case of a `Thread#interrupt` call, but it checks the `released` flag constantly. If this flag becomes true, the blocking service will finish its job:

```scala

final case class BlockingService() {
  private val released = new AtomicReference(false)

  def start(): Unit = {
    while (!released.get()) {
      println("Doing some blocking operation")
      try Thread.sleep(1000)
      catch {
        case _: InterruptedException => () // Swallowing InterruptedException
      }
    }
    println("Blocking operation closed.")
  }

  def close(): Unit = {
    println("Releasing resources and ready to be closed.")
    released.getAndSet(true)
  }
}
```

So to translate ZIO interruption into cancellation of these types of blocking operations we should use `attemptBlockingCancelable`. This method takes a `cancel` effect which is responsible for signalling the blocking code to close itself when ZIO interruption occurs:

```scala

val myApp =
  for {
    service <- ZIO.attempt(BlockingService())
    fiber   <- ZIO.attemptBlockingCancelable(
      effect = service.start()
    )(
      cancel = ZIO.succeed(service.close())
    ).fork
    _       <- fiber.interrupt.schedule(
      Schedule.delayed(
        Schedule.duration(3.seconds)
      )
    )
  } yield ()
```

Here is another example of the cancellation of a blocking operation. When we `accept` a server socket, this blocking operation will never be interrupted until we close it using the `ServerSocket#close` method:

```scala

def accept(ss: ServerSocket): Task[Socket] =
  ZIO.attemptBlockingCancelable(ss.accept())(ZIO.succeed(ss.close()))
```

## Disabling Interruption of Fibers

As we discussed earlier, it is dangerous for fibers to interrupt others. The danger with such an interruption is that:

- If the interruption occurs during the execution of an operation that must be _finalized_, the finalization will not be executed.

- If this interruption occurs in the middle of a _critical section_, it will cause an application state to become inconsistent.

- It is also a threat to _resource safety_. If the fiber is in the middle of acquiring a resource and is interrupted, the application will leak resources.

ZIO introduces the `uninterruptible` and `uninterruptibleMask` operations for this purpose. The former creates a region of code uninterruptible and the latter has the same functionality but gives us a `restore` function that can be applied to any region of code to restore the interruptibility of that region.

These operators are advanced and very low-level so we don't use them in regularly in application development unless we know what we are doing as library designers. If you find yourself using these operators, think again about refactoring your code using high-level operators like `ZIO#onInterrupt`, `ZIO#onDone`, `ZIO#ensuring`, `ZIO.acquireRelease*` and many other concurrent operators like `race`, `foreachPar`, etc.

---

## Introduction to Logging in ZIO

ZIO supports a lightweight built-in logging facade that standardizes the interface to logging functionality. So it doesn't replace existing logging libraries, but also we can plug it into one of the existing logging backends.

We can easily log using the `ZIO.log` function:

```scala

val app =
  for {
    _ <- ZIO.log("Application started!")
    name <- Console.readLine("Please enter your name: ")
    _ <- ZIO.log("User entered its name: $name")
    _ <- Console.printLine("Hello, $name")
  } yield ()
```

## Logging Levels

To log with a specific log-level, we can use the `ZIO.logLevel` combinator:

```scala
ZIO.logLevel(LogLevel.Warning) {
  ZIO.log("The response time exceeded its threshold!")
}
```

Or we can use the following functions directly:

* `ZIO.logDebug`
* `ZIO.logError`
* `ZIO.logFatal`
* `ZIO.logInfo`
* `ZIO.logWarning`

For example, for log with the error level, we can use `ZIO.logError` like this:

```scala
ZIO.logError("File does not exist: ~/var/www/favicon.ico")
```

## Spans

It also supports spans:

```scala
ZIO.logSpan("myspan") {
  ZIO.sleep(1.second) *> ZIO.log("The job is finished!")
}
```

ZIO Logging calculates the running duration of that span and includes that in the logging data corresponding to its span
label.

## Log Annotations

ZIO by default adds some contextual information to the log messages, like the timestamp, log level, fiber ID, and source location. Sometimes these default contextual information are not sufficient to understand the circumstances under which they were generated. In such cases, we need to add custom contextual information to the log messages. We can do this using log annotations.

### ZIO's Built-in Log Annotation

For example, in microservice environments, we might have several services that are communicating with each other. In such cases, we might want to correlate the logs generated by different services. We can do this by adding a log annotation called `correlation_id`. This log annotation can be very simple, just a string, that is passed along with requests and responses. So, when we log a message, we know which request or response it is related to. ZIO has a built-in log annotation API that allows us to add such custom contextual information to the log messages:

```scala

object MainApp extends ZIOAppDefault {
  def randomDelay = Random.nextIntBounded(1000).flatMap(t => ZIO.sleep(t.millis))

  def run =
    ZIO.foreachParDiscard(Chunk("UserA", "UserB", "UserC")) { user =>
      ZIO.logAnnotate("correlation_id", user) {
        for {
          _ <- ZIO.log("fetching user from database") *> randomDelay
          _ <- ZIO.log("downloading user's profile picture") *> randomDelay
        } yield ()
      }
    }
}
```

Here is an example of the log messages generated by the above code, each log message contains the `correlation_id` log annotation:

```
timestamp=2024-05-14T15:44:50.734129Z level=INFO thread=#zio-fiber-851563977 message="fetching user from database" location=zio.examples.MainApp.run file=MainApp.scala line=12 correlation_id=UserC
timestamp=2024-05-14T15:44:50.734127Z level=INFO thread=#zio-fiber-41969365 message="fetching user from database" location=zio.examples.MainApp.run file=MainApp.scala line=12 correlation_id=UserA
timestamp=2024-05-14T15:44:50.734123Z level=INFO thread=#zio-fiber-1775966732 message="fetching user from database" location=zio.examples.MainApp.run file=MainApp.scala line=12 correlation_id=UserB
timestamp=2024-05-14T15:44:50.928248Z level=INFO thread=#zio-fiber-851563977 message="downloading user's profile picture" location=zio.examples.MainApp.run file=MainApp.scala line=13 correlation_id=UserC
timestamp=2024-05-14T15:44:51.054287Z level=INFO thread=#zio-fiber-41969365 message="downloading user's profile picture" location=zio.examples.MainApp.run file=MainApp.scala line=13 correlation_id=UserA
timestamp=2024-05-14T15:44:51.534263Z level=INFO thread=#zio-fiber-1775966732 message="downloading user's profile picture" location=zio.examples.MainApp.run file=MainApp.scala line=13 correlation_id=UserB
```

### Typed Log Annotations

In more complex scenarios, we might want to add more structured information to the log messages. For example, we might want to add the user information to the log messages. In such cases, we need a typed log annotation that supports structured information, e.g. a `User` case class that contains the user's id, name, email, etc. 

Using [ZIO Logging](https://zio.dev/zio-logging), we can define typed log annotations using the `LogAnnotation` class. So let's add required dependencies to the `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-logging" % "4.0.2",
  "dev.zio" %% "zio-json"    % "0.7.38"
)
```

Now, let's assume we have a `User` case class:

```scala
case class User(firstName: String, lastName: String)
```

We can define a typed log annotation for the `User` case class like this:

```scala

object TypedLogAnnotationExample extends ZIOAppDefault {

  case class User(firstName: String, lastName: String)

  object User {
    implicit val encoder = DeriveJsonEncoder.gen[User]
  }

  private val userLogAnnotation = LogAnnotation[User]("user", (_, u) => u, _.toJson)
  
  private val logConfig = ConsoleLoggerConfig.default.copy(
    format = LogFormat.default + LogFormat.annotation(LogAnnotation.TraceId) + LogFormat.annotation(userLogAnnotation)
  )

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleJsonLogger(logConfig)

  def run =
    for {
      _ <- ZIO.foreachPar(List(User("John", "Doe"), User("Jane", "Doe"))) { user =>
        {
          ZIO.logInfo("Starting operation") *>
            ZIO.sleep(500.millis) *>
            ZIO.logInfo("Stopping operation")
        } @@ userLogAnnotation(user)
      }
      _ <- ZIO.logInfo("Done")
    } yield ()

}
```

The log messages generated by the above code will contain the `user` log annotation:

```
{"timestamp":"2024-05-14T20:37:33.744171+04:30","level":"INFO","thread":"zio-fiber-6","message":"Starting operation","user":{"firstName":"Jane","lastName":"Doe"}}
{"timestamp":"2024-05-14T20:37:33.744166+04:30","level":"INFO","thread":"zio-fiber-5","message":"Starting operation","user":{"firstName":"John","lastName":"Doe"}}
{"timestamp":"2024-05-14T20:37:34.334837+04:30","level":"INFO","thread":"zio-fiber-5","message":"Stopping operation","user":{"firstName":"John","lastName":"Doe"}}
{"timestamp":"2024-05-14T20:37:34.334848+04:30","level":"INFO","thread":"zio-fiber-6","message":"Stopping operation","user":{"firstName":"Jane","lastName":"Doe"}}
{"timestamp":"2024-05-14T20:37:34.337953+04:30","level":"INFO","thread":"zio-fiber-4","message":"Done"}
```

## Further Reading

* [ZIO Logging](https://zio.dev/zio-logging)
* [How to Enable Logging in a ZIO Application](../../guides/tutorials/enable-logging-in-a-zio-application.md)
* [How to Create a Custom Logger for a ZIO Application?](../../guides/tutorials/create-custom-logger-for-a-zio-application.md)

---

## Counter

A `Counter` is a metric representing a single numerical value that may be incremented over time. A typical use of this metric would be to track the number of a certain type of request received.

With a counter, the quantity of interest is the cumulative value over time, as opposed to a [gauge](gauge.md) where the quantity of interest is the value as of a specific point in time.

## API

With one of the following constructors, we can create a counter of `Long`, `Double` or `Int` type:

```scala
object Metric {
  def counter(name: String): Counter[Long] = ???
  def counterDouble(name: String): Counter[Double] = ???
  def counterInt(name: String): Counter[Int] = ???
}
```

## Use Cases

We use the counter metric type for any value that increases, such as request counts. Note that we should never use the counter for a value that can decrease.

So when we should use counters?

- When we want to track a value over time, that only goes up
- When we want to measure the increasing rate of something, how fast something is growing, such as request rates.

Here are some of the use cases:

- Request Counts
- Completed Tasks
- Error Counts

## Examples

Create a counter named `countAll` which is incremented by `1` every time it is invoked:

```scala

val countAll = Metric.counter("countAll").fromConst(1)
```

Now the counter can be applied to any effect. Note, that the same aspect can be applied to more than one effect. In the example we would count the sum of executions of both effects in the for comprehension:

```scala
val myApp = for {
  _ <- ZIO.unit @@ countAll
  _ <- ZIO.unit @@ countAll
} yield ()
```

Or we can apply them in recurrence situations:

```scala
(zio.Random.nextLongBounded(10) @@ Metric.counter("request_counts")).repeatUntil(_ == 7)
```

Create a counter named `countBytes` that can be applied to effects having the output type `Double`:

```scala
val countBytes = Metric.counter("countBytes")
```

Now we can apply it to effects producing `Double` (in a real application the value might be the number of bytes read from a stream or something similar):

```scala
val myApp = Random.nextLongBetween(0, 100) @@ countBytes
```

---

## Frequency

A `Frequency` represents the number of occurrences of specified values. We can think of a `Frequency` as a set of counters associated with each value except that new counters will automatically be created when new values are observed.

Essentially, a `Frequency` is a set of related counters sharing the same name and tags. The counters are set apart from each other by an additional configurable tag. The values of the tag represent the observed distinct values.

## API
```scala
object Metric {
  def frequency(name: String): Frequency[String] = ???
}
```
## Use Cases

Sets are used to count the occurrences of distinct string values:
- Tracking number of invocations for each service, for an application that uses logical names for its services.
- Tracking frequency of different types of failures.

## Examples

Create a `Frequency` to observe the occurrences of unique `Strings`. It can be applied to effects yielding a `String`:

```scala

val freq = Metric.frequency("MySet")
```

Now we can generate some keys within an effect and start counting the occurrences for each value:

```scala

(Random.nextIntBounded(10).map(v => s"MyKey-$v") @@ freq).repeatN(100)
```

---

## Gauge

A `Gauge` is a metric representing a single numerical value that may be _set_ or _adjusted_. A typical use of this metric would be to track the current memory usage.

With a gauge, the quantity of interest is the current value, as opposed to a counter where the quantity of interest is the cumulative values over time.

A gauge is a named variable of type _Double_ that can change over time. It can either be set to an absolute value or relative to the current value.

## API

```scala
object Metric {
  def gauge(name: String): Gauge[Double] = ???
}
```

## Use Case

The gauge metric type is the best choice for things that their values can go down as well as up, such as queue size, and we don't want to query their rates. Thus, they are used to measuring things that have a particular value at a certain point in time:

- Memory Usage
- Queue Size
- In-Progress Request Counts
- Temperature

## Examples

Create a gauge that can be set to absolute values, it can be applied to effects yielding a `Double`:

```scala

val absoluteGauge = Metric.gauge("setGauge")
```

Now we can apply these gauges to effects having an output type `Double`. Note that we can instrument an effect with any number of aspects if the type constraints are satisfied:

```scala
for {
  _ <- Random.nextDoubleBetween(0.0d, 100.0d) @@ absoluteGauge @@ countAll
} yield ()
```

---

## Histogram

A `Histogram` is a metric representing a collection of numerical with the distribution of the cumulative values over time. They organize a range of measurements into distinct intervals, known as buckets, and record the frequency of measurements falling within each bucket. 

Histograms allow representing not only the value of the quantity being measured but its distribution. They are representation of the distribution of a dataset, which organizes the data into buckets and display the frequency or count of data points within each bucket. 

## Internals

In a histogram, we assign the incoming samples to pre-defined buckets. So each data point increases the count for the bucket that it falls into, and then the individual samples are discarded. As histograms are bucketed, we can aggregate data across multiple instances. Histograms are a typical way to measure percentiles. We can look at bucket counts to estimate a specific percentile.

A histogram observes _Double_ values and counts the observed values in buckets. Each bucket is defined by an upper boundary, and the count for a bucket with the upper boundary `b` increases by `1` if an observed value `v` is less or
equal to `b`.

As a consequence, all buckets that have a boundary `b1` with `b1 > b` will increase by `1` after observing `v`.

A histogram also keeps track of the overall count of observed values, and the sum of all observed values.

By definition, the last bucket is always defined as `Double.MaxValue`, so that the count of observed values in the last bucket is always equal to the overall count of observed values within the histogram.

The mental model for histogram is inspired from [Prometheus](https://prometheus.io/docs/concepts/metric_types/#histogram).

## API

```scala
object Metric {
  def histogram(
      name: String,
      boundaries: Histogram.Boundaries
    ): Histogram[Double] = ???
  
  def timer(
      name: String,
      description: String,
      chronoUnit: ChronoUnit
    ): Metric[MetricKeyType.Histogram, Duration, MetricState.Histogram] = ???
  
  def timer(
      name: String,
      chronoUnit: ChronoUnit,
      boundaries: Chunk[Double]
    ): Metric[MetricKeyType.Histogram, Duration, MetricState.Histogram] = ???
}
```

## Use Cases

Histograms are widely used in software metrics for various purposes. They are useful in analyzing the performance of software systems. They can represent metrics such as **response times**, **latencies**, or **throughput**. By visualizing the distribution of these metrics in a histogram, developers can identify **performance bottlenecks**, **outliers**, or **variations**. This information aids in optimizing code, infrastructure, and system configurations to improve overall performance.

Histogram measures the frequency of value observations that fall into specific _pre-defined buckets_. For example, we can measure the request duration of an HTTP request using histograms. Rather than storing every duration for every request, the histogram will make an approximation by storing the frequency of requests that fall into pre-defined particular buckets.

Thus, histograms are the best choice in these situations:

- When we want to observe many values and then later want to calculate the percentile of observed values
- When we can estimate the range of values upfront, as the histogram put the observations into pre-defined buckets
- When accuracy is not so important, and we don't want the exact values because of the lossy nature of bucketing data in histograms
- When we need to aggregate histograms across multiple instances

## Examples

### Histogram With Linear Buckets

Create a histogram with 12 buckets: `0..100` in steps of `10` and `Double.MaxValue`. It can be applied to effects yielding a `Double`:

```scala

val histogram =
  Metric.histogram("histogram", MetricKeyType.Histogram.Boundaries.linear(0, 10, 11))
```

Now we can apply the histogram to effects producing `Double`:

```scala

Random.nextDoubleBetween(0.0d, 120.0d) @@ histogram
```

### Timer Metric

Here is an example of adding timer metric to track workflow durations:

```scala

object Example extends ZIOAppDefault {

  def workflow = ZIO.succeed(42)

  def randomDelay =
    for {
      i <- Random.nextLongBetween(1L, 10)
      _ <- ZIO.sleep(Duration.fromMillis(i))
    } yield ()
    
  val timer =
    Metric.timer(
      name = "timer",
      chronoUnit = ChronoUnit.MILLIS,
      boundaries = Chunk.iterate(1.0, 10)(_ + 1.0)
    )

  val run = ((workflow <* randomDelay) @@ timer.trackDuration).repeatN(99)
}
``` 

If we add prometheus layer, we expose the metrics which is something like this:

```csv
# TYPE timer histogram
# HELP timer
timer_bucket{time_unit="millis",le="1.0",} 6.0 1686581577320
timer_bucket{time_unit="millis",le="2.0",} 16.0 1686581577320
timer_bucket{time_unit="millis",le="3.0",} 27.0 1686581577320
timer_bucket{time_unit="millis",le="4.0",} 41.0 1686581577320
timer_bucket{time_unit="millis",le="5.0",} 49.0 1686581577320
timer_bucket{time_unit="millis",le="6.0",} 60.0 1686581577320
timer_bucket{time_unit="millis",le="7.0",} 70.0 1686581577320
timer_bucket{time_unit="millis",le="8.0",} 85.0 1686581577320
timer_bucket{time_unit="millis",le="9.0",} 99.0 1686581577320
timer_bucket{time_unit="millis",le="10.0",} 99.0 1686581577320
timer_bucket{time_unit="millis",le="+Inf",} 100.0 1686581577320

timer_sum{time_unit="millis",} 603.0 1686581577320
timer_count{time_unit="millis",} 100.0 1686581577320
timer_min{time_unit="millis",} 1.0 1686581577320
timer_max{time_unit="millis",} 66.0 1686581577320⏎
```

This Prometheus result represents a histogram metric called "timer" with a time unit of milliseconds. The histogram provides information about the distribution of workflow durations.

The histogram is divided into multiple buckets, each representing a range of workflow durations. The "le" label indicates the upper bound of each bucket. The values next to each bucket indicate the count or frequency of measurements falling within that bucket. 

For instance, `"timer_bucket{time_unit="millis",le="5.0",} 49.0"` means there are 49 measurements with duration less than or equal to 5.0 millisecond.

---

## Introduction to ZIO Metrics

In highly concurrent applications, things are interconnected, so maintaining such a setup to run smoothly and without application downtimes is very challenging. 

Imagine we have a complex infrastructure with lots of services. Services are replicated and distributed across our servers. So we have no insight into what happening across our services at the level of errors, response latency, service uptime, etc. With ZIO Metrics, we can capture these different metrics and provide them to a _metric service_ for later investigation.

ZIO supports 5 types of Metrics:

* **[Counter](counter.md)** — The Counter is used for any value that increases over time like _request counts_.
* **[Gauge](gauge.md)** — The gauge is a single numerical value that can go up or down over time like _memory usage_.
* **[Histogram](histogram.md)** — The Histogram is used to track the distribution of a set of observed values across a set of buckets like _request latencies_.
* **[Summary](summary.md)** — The Summary represents a sliding window of a time series along with metrics for certain percentiles of the time series, referred to as quantiles like _request latencies_.
* **[Frequency](frequency.md)** — The Frequency is a metric that counts the number of occurrences of distinct string values.

All ZIO Metrics are defined in the form of ZIO Aspects that can be applied to effects without changing the signature of the effect it is applied to:

```scala

def memoryUsage: ZIO[Any, Nothing, Double] = {

  ZIO
    .succeed(getRuntime.totalMemory() - getRuntime.freeMemory())
    .map(_ / (1024.0 * 1024.0)) @@ Metric.gauge("memory_usage")
}
```

After adding metrics into our application, whenever we want we can capture snapshots of all metrics recorded by our application, by any of the metric backends supported by [ZIO Metrics Connectors](https://github.com/zio/zio-metrics-connectors) project.

Here is an example of adding a Prometheus connector to our application:

```scala title="examples/jvm/src/main/scala/zio/examples/metrics/MetricAppExample.scala" 
package zio.examples.metrics

object MetricAppExample extends ZIOAppDefault {
  def memoryUsage: ZIO[Any, Nothing, Double] = {

    ZIO
      .succeed(getRuntime.totalMemory() - getRuntime.freeMemory())
      .map(_ / (1024.0 * 1024.0)) @@ Metric.gauge("memory_usage")
  }

  private val httpApp =
    Routes(
      Method.GET / "metrics" ->
        handler(ZIO.serviceWithZIO[PrometheusPublisher](_.get.map(Response.text))),
      Method.GET / "foo" -> handler {
        for {
          _    <- memoryUsage
          time <- Clock.currentDateTime
        } yield Response.text(s"$time\t/foo API called")
      }
    )

  override def run = Server
    .serve(httpApp)
    .provide(
      // ZIO Http default server layer, default port: 8080
      Server.default,
      // The prometheus reporting layer
      prometheus.prometheusLayer,
      prometheus.publisherLayer,
      // Interval for polling metrics
      ZLayer.succeed(MetricsConfig(5.seconds))
    )
}
```

To run this example we need to add following lines to our `build.sbt` file:

```scala

```

ZIO Metrics Connectors currently supports the following backends:

- [Prometheus](https://prometheus.io/)
- [Datadog](https://www.datadoghq.com/)
- [New Relic](https://newrelic.com/)
- [StatsD](https://github.com/statsd/statsd)

---

## JVM Metrics

ZIO has built-in support for collecting JVM Metrics. These metrics are a direct port of the JVM metrics provided by the [Prometheus Java Hotspot library](https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot) and compatible with that library.

There are five categories of JVM metrics. Let's look at them one by one:

- Buffer Pools
    - `jvm_buffer_pool_used_bytes` — Used bytes of a given JVM buffer pool.
    - `jvm_buffer_pool_capacity_bytes` — Bytes capacity of a given JVM buffer pool.
    - `jvm_buffer_pool_used_buffers` — Used buffers of a given JVM buffer pool.
- Class Loading
    - `jvm_classes_loaded` — The number of classes that are currently loaded in the JVM
    - `jvm_classes_loaded_total` — The total number of classes that have been loaded since the JVM has started execution
    - `jvm_classes_unloaded_total` — The total number of classes that have been unloaded since the JVM has started
      execution
- Garbage Collector
    - `jvm_gc_collection_seconds_sum` — Time spent in a given JVM garbage collector in seconds.
    - `jvm_gc_collection_seconds_count`
- Memory Allocation
    - `jvm_memory_pool_allocated_bytes_total` — Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
- Memory Pools
    - `jvm_memory_bytes_used` — Used bytes of a given JVM memory area.
    - `jvm_memory_bytes_committed` — Committed (bytes) of a given JVM memory area.
    - `jvm_memory_bytes_max` — Max (bytes) of a given JVM memory area.
    - `jvm_memory_bytes_init` — Initial bytes of a given JVM memory area.
    - `jvm_memory_pool_bytes_used` — Used bytes of a given JVM memory pool.
    - `jvm_memory_pool_bytes_committed` — Committed bytes of a given JVM memory pool.
    - `jvm_memory_pool_bytes_max` — Max bytes of a given JVM memory pool.
    - `jvm_memory_pool_bytes_init` — Initial bytes of a given JVM memory pool.
- Standard
    - `process_cpu_seconds_total` — Total user and system CPU time spent in seconds.
    - `process_start_time_seconds` — Start time of the process since unix epoch in seconds.
    - `process_open_fds` — Number of open file descriptors.
    - `process_max_fds` — Maximum number of open file descriptors.
    - `process_virtual_memory_bytes` — Virtual memory size in bytes.
    - `process_resident_memory_bytes` — Resident memory size in bytes.
- Thread
    - `jvm_threads_current` — Current thread count of a JVM.
    - `jvm_threads_daemon` — Daemon thread count of a JVM.
    - `jvm_threads_peak` — Peak thread count of a JVM.
    - `jvm_threads_started_total` — Started thread count of a JVM.
    - `jvm_threads_deadlocked` — Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers.
    - `jvm_threads_deadlocked_monitor` — Cycles of JVM-threads that are in deadlock waiting to acquire object monitors.
    - `jvm_threads_state` — Current count of threads by state.
- Version Info
    - `jvm_info`
        - `version` — java.runtime.version 
        - `vendor` — java.vm.vendor
        - `runtime` — java.runtime.name

## Collecting Metrics

### Collecting Inside a ZIO Application

JVM Metrics are collection of the following ZIO services:
- BufferPools
- ClassLoading
- GarbageCollector
- MemoryAllocation
- MemoryPools
- Standard
- Thread
- VersionInfo

All of these services are available in the `zio.metrics.jvm` package. Each service has a `live` implementation that can be used to collect metrics, or we can use all of them at once with by providing `DefaultJvmMetrics.live` layer to our application.

### Collecting as a Sidecar to a ZIO Application

ZIO JVM metrics have built-in applications that collect the JVM metrics. They can be composed with other ZIO applications as a _sidecar_. By doing so, we are able to collect JVM metrics without modifying our main ZIO application. They will be executed as a daemon alongside the main app:

```scala title="examples/jvm/src/main/scala/zio/examples/metrics/JvmMetricAppExample.scala" 
package zio.examples.metrics

object JvmMetricAppExample extends ZIOAppDefault {
  private val httpApp =
    Routes(
      Method.GET / "metrics" ->
        handler(ZIO.serviceWithZIO[PrometheusPublisher](_.get.map(Response.text)))
    )

  override def run = Server
    .serve(httpApp)
    .provide(
      // ZIO Http default server layer, default port: 8080
      Server.default,

      // The prometheus reporting layer
      prometheus.prometheusLayer,
      prometheus.publisherLayer,
      // Interval for polling metrics
      ZLayer.succeed(MetricsConfig(5.seconds)),

      // Default JVM Metrics
      DefaultJvmMetrics.liveV2.unit
    )
}
```

---

## MetricLabel

A `MetricLabel` metadata represents a key-value pair that allows analyzing metrics at an additional level of granularity. For example, if a metric tracks the response time of a service, labels could be used to create separate versions that track response times for different clients. 

ZIO Metrics has a _label based dimensional_ data model where we have a metric name and just a list of key-value pairs attached to that metric name. So labels are the first-class citizen in ZIO Metrics. In monitoring dashboards, we can find or filter metrics using these labels.

For example, we can append following labels (dimensions) to our metric aspects:
- Endpoint (/api/users, /api/documents)
- HTTP Method (POST, GET)
- Deployment Environment (Staging, Production)
- HTTP Response Code
- Error code (404, 503)
- Datacenter Zone (us-east, eu-west)

```scala

val counter = Metric.counter("http_requests")
  .tagged(
    MetricLabel("env", "production"),
    MetricLabel("method", "GET"),
    MetricLabel("endpoint", "/api/users"),
    MetricLabel("zone", "ap-northeast"),
  )
```

By labeling metrics, we can query in a more granular way in monitoring dashboards, such as
- How many requests have been sent to each endpoint in total?
- In which zone are we about to violate SLAs?
- Which endpoint has the most latency?

---

## Summary(Metrics)

A `Summary` represents a sliding window of a time series along with metrics for certain percentiles of the time series, referred to as quantiles.

Quantiles describe specified percentiles of the sliding window that are of interest. For example, if we were using a summary to track the response time for requests over the last hour then we might be interested in the 50th percentile, 90th percentile, 95th percentile, and 99th percentile for response times.

## Internals

Similar to a [histogram](histogram.md) a summary also observes _Double_ values. While a histogram directly modifies the bucket counters and does not keep the individual samples, the summary keeps the observed samples in its internal state. To avoid the set of samples grow uncontrolled, the summary needs to be configured with a maximum age `t` and a maximum size `n`. To calculate the statistics, maximal `n` samples will be used, all of which are not older than `t`.

Essentially, the set of samples is a _sliding window_ over the last observed samples matching the conditions above.

A summary is used to calculate a set of quantiles over the current set of samples. A quantile is defined by a _Double_ value `q` with `0 <= q <= 1` and resolves to a `Double` as well.

The value of a given quantile `q` is the maximum value `v` out of the current sample buffer with size `n` where at most `q * n` values out of the sample buffer are less or equal to `v`.

Typical quantiles for observation are `0.5` (the median) and the `0.95`. Quantiles are very good for monitoring _Service Level Agreements_.

The ZIO Metrics API also allows summaries to be configured with an error margin `e`. The error margin is applied to the count of values, so that a quantile `q` for a set of size `s` resolves to value `v` if the number `n` of values less or equal to `v` is `(1 -e)q * s <= n <= (1+e)q`.

## API

```scala
object Metric {
  def summary(
    name: String,
    maxAge: Duration,
    maxSize: Int,
    error: Double,
    quantiles: Chunk[Double]
  ): Summary[Double] = ???
             
  def summaryInstant(
    name: String,
    maxAge: Duration,
    maxSize: Int,
    error: Double,
    quantiles: Chunk[Double]
  ): Summary[(Double, java.time.Instant)] =
}
```

## Use Cases

Like [histograms](histogram.md), summaries are used for _monitoring latencies_, but they don't require us to define buckets. So, summaries are the best choice in these situations:
- When histograms are not proper for us, in terms of accuracy
- When we can't use histograms, as we don't have a good estimation of the range of values
- When we don't need to perform aggregation or averages across multiple instances, as the calculations are done on the application side

## Examples

Create a summary that can hold `100` samples, the max age of the samples is `1 day` and the error margin is `3%`. The summary should report the `10%`, `50%` and `90%` Quantile. It can be applied to effects yielding an `Int`:

```scala

val summary: Summary[Double] =
  Metric.summary(
    name = "mySummary", 
    maxAge = 1.day,
    maxSize = 100,
    error = 0.03d, 
    quantiles = Chunk(0.1, 0.5, 0.9)
  )
``` 

Now we can apply this aspect to an effect producing an `Int`:

```scala
Random.nextDoubleBetween(100, 500) @@ summary
```

---

## Supervisor

A `Supervisor[A]` is allowed to supervise the launching and termination of fibers, producing some visible value of type `A` from the supervision.

## Creation

### track
The `track` creates a new supervisor that tracks children in a set. It takes a boolean `weak` parameter as input, which indicates whether track children in a `Weakset` or not.

```scala
val supervisor = Supervisor.track(true)
// supervisor: zio.package.UIO[Supervisor[zio.Chunk[zio.Fiber.Runtime[Any, Any]]]] = Sync(
//   trace = "repl.MdocSession.MdocApp.supervisor(supervisor.md:14)",
//   eval = zio.Supervisor$$$Lambda$20126/0x00007f02a7055b18@319d8c00
// )
```

We can periodically, report the status of the fibers of our program with the help of the Supervisor.

### fibersIn
The `fibersIn` creates a new supervisor with an initial sorted set of fibers.

In the following example we are creating a new supervisor from an initial set of fibers:

```scala
def fiberListSupervisor = for { 
  ref <- ZIO.succeed(new AtomicReference(SortedSet.from(fibers)))
  s <- Supervisor.fibersIn(ref)
} yield (s)
```

## Supervising

Whenever we need to supervise a ZIO effect, we can call `ZIO#supervised` function, `supervised` takes a supervisor and return another effect. The behavior of children fibers is reported to the provided supervisor:

```scala
val supervised = supervisor.flatMap(s => fib(20).supervised(s))
```

Now we can access all information of children fibers through the supervisor.

## Example

In the following example we are going to periodically monitor the number of fibers throughout our application life cycle:

```scala

object SupervisorExample extends ZIOAppDefault {

  def run = for {
    supervisor <- Supervisor.track(true)
    fiber <- fib(20).supervised(supervisor).fork
    policy = Schedule
      .spaced(500.milliseconds)
      .whileInputZIO[Any, Unit](_ => fiber.status.map(_ != Status.Done))
    logger <- monitorFibers(supervisor)
      .repeat(policy).fork
    _ <- logger.join
    result <- fiber.join
    _ <- Console.printLine(s"fibonacci result: $result")
  } yield ()

  def monitorFibers(supervisor: Supervisor[Chunk[Fiber.Runtime[Any, Any]]]) = for {
    length <- supervisor.value.map(_.length)
    _ <- Console.printLine(s"number of fibers: $length")
  } yield ()

  def fib(n: Int): ZIO[Any, Nothing, Int] =
    if (n <= 1) {
      ZIO.succeed(1)
    } else {
      for {
        _ <- ZIO.sleep(500.milliseconds)
        fiber1 <- fib(n - 2).fork
        fiber2 <- fib(n - 1).fork
        v2 <- fiber2.join
        v1 <- fiber1.join
      } yield v1 + v2
    }

}
```

---

## Introduction to Tracing in ZIO

Although logs and metrics are useful for understanding the behavior of individual services, they are not enough to provide a complete overview of the lifetime of a request in a distributed system.

In a distributed system, a request can span multiple services and each service can make multiple requests to other services to fulfill the request. In such a scenario, we need to have a way to track the lifetime of a request across multiple services to diagnose what services are the bottlenecks and where the request is spending most of its time.

ZIO Telemetry supports tracing through the OpenTelemetry API. To learn more about tracing, please refer to the [ZIO Telemetry documentation](https://zio.dev/zio-telemetry/opentracing).

---

## Introduction to Resource Management in ZIO

When we are writing a long-lived application, resource management is very important. Proper resource management is vital to any large-scale application. We need to make sure that our application is resource-safe, and it doesn't leak any resource.

Leaking socket connections, database connections or file descriptors is not acceptable in a web application. ZIO provides some good construct to make sure about this concern.

To write a resource-safe application, we need to make sure whenever we are opening a resource, we have a mechanism to close that resource whether we use that resource completely or not, for example, an exception occurred during resource usage.

## Try / Finally
Before we dive into the ZIO solution, it's better to review the `try` / `finally` which is the standard approach in the Scala language to manage resources.

Scala has a `try` / `finally` construct which helps us to make sure we don't leak resources because no matter what happens in the try, the `finally` block will be executed. So we can open files in the try block, and then we can close them in the `finally` block, and that gives us the guarantee that we will not leak resources.

Assume we want to read a file and return the number of its lines:

```scala
def lines(file: String): Task[Long] = ZIO.attempt {
  def countLines(br: BufferedReader): Long = br.lines().count()
  val bufferedReader = new BufferedReader(
    new InputStreamReader(new FileInputStream("file.txt")),
    2048
  )
  val count = countLines(bufferedReader)
  bufferedReader.close()
  count
}
```

What happens if after opening the file and before closing the file, an exception occurs? So, the `bufferedReader.close()` line, doesn't have a chance to close the resource. This creates a resource leakage. The Scala language has `try...finally` construct, which helps up to prevent these situations.

Let's rewrite the above example with `try..finally`:

```scala
def lines(file: String): Task[Long] = ZIO.attempt {
  def countLines(br: BufferedReader): Long = br.lines().count()
  val bufferedReader = new BufferedReader(
    new InputStreamReader(new FileInputStream("file.txt")),
    2048
  )
  try countLines(bufferedReader)
  finally bufferedReader.close()
}
```

Now, we are sure that if our program is interrupted during the process of a file, the `finally` block will be executed.

The `try` / `finally` solve simple problems, but it has some drawbacks:

1. It's not composable; We can't compose multiple resources together.

2. When we have multiple resources, we end up with messy and ugly code, hard to reason about, and refactoring.
3. We don't have any control over the order of resource clean-up
4. It only helps us to handle resources sequentially. It can't compose multiple resources, concurrently.
5. It doesn't support asynchronous workflows.
6. It's a manual way of resource management, not automatic. To have a resource-safe application we need to manually check that all resources are managed correctly. This way of resource management is error-prone in case of forgetting to manage resources, correctly.

## ZIO Solution

ZIO's resource management features work across synchronous, asynchronous, concurrent, and other effect types, and provide strong guarantees even in the presence of failure, interruption, or defects in the application.

ZIO has two major mechanisms to manage resources.

### Acquire Release

ZIO generalized the pattern of `try` / `finally` and encoded it in `ZIO.acquireRelease` or `ZIO#acquireRelease` operations. 

Every acquire release requires three actions:
1. **Acquiring Resource**— An effect describing the acquisition of resource. For example, opening a file.
2. **Using Resource**— An effect describing the actual process to produce a result. For example, counting the number of lines in a file.
3. **Releasing Resource**— An effect describing the final step of releasing or cleaning up the resource. For example, closing a file.

```scala
def use(resource: Resource): Task[Any] = ZIO.attempt(???)
def release(resource: Resource): UIO[Unit] = ZIO.succeed(???)
def acquire: Task[Resource]                = ZIO.attempt(???)

val result: Task[Any] = ZIO.acquireReleaseWith(acquire)(release)(use)
```

The acquire release guarantees us that the `acquiring` and `releasing` of a resource will not be interrupted. These two guarantees ensure us that the resource will always be released.

Let's try a real example. We are going to write a function which count line number of given file. As we are working with file resource, we should separate our logic into three part. At the first part, we create a `BufferedReader`. At the second, we count the file lines with given `BufferedReader` resource, and at the end we close that resource:

```scala
def lines(file: String): Task[Long] = {
  def countLines(reader: BufferedReader): Task[Long]    = ZIO.attempt(reader.lines().count())
  def releaseReader(reader: BufferedReader): UIO[Unit]  = ZIO.succeed(reader.close())
  def acquireReader(file: String): Task[BufferedReader] = ZIO.attempt(new BufferedReader(new FileReader(file), 2048))

  ZIO.acquireReleaseWith(acquireReader(file))(releaseReader)(countLines)
}
```

Let's write another function which copy a file from source to destination file. We can do that by nesting two acquire releases one for the `FileInputStream` and the other for `FileOutputStream`:

```scala
def is(file: String): Task[FileInputStream]  = ZIO.attempt(???)
def os(file: String): Task[FileOutputStream] = ZIO.attempt(???)

def close(resource: Closeable): UIO[Unit] = ZIO.succeed(???)
def copy(from: FileInputStream, to: FileOutputStream): Task[Unit] = ???

def transfer(src: String, dst: String): ZIO[Any, Throwable, Unit] = {
  ZIO.acquireReleaseWith(is(src))(close) { in =>
    ZIO.acquireReleaseWith(os(dst))(close) { out =>
      copy(in, out)
    }
  }
}
```

As there isn't any dependency between our two resources (`is` and `os`), it doesn't make sense to use nested acquire releases, so let's `zip` these two acquisition into one effect, and then use one acquire release on them:

```scala
def transfer(src: String, dst: String): ZIO[Any, Throwable, Unit] =
  ZIO.acquireReleaseWith {
    is(src).zipPar(os(dst))
  } { case (in, out) =>
    ZIO.succeed(in.close()).zipPar(ZIO.succeed(out.close()))
  } { case (in, out) =>
    copy(in, out)
  }
```

While using acquire release is a nice and simple way of managing resources, but there are some cases where an acquire release is not the best choice:

1. Acquire release is not composable— When we have multiple resources, composing them with an acquire release is not straightforward.

2. Messy nested acquire releases — In the case of multiple resources, nested acquire releases remind us of callback hell awkwardness. The acquire release is designed with nested resource acquisition. In the case of multiple resources, we encounter inefficient nested acquire release calls, and it causes refactoring a complicated process.

Using acquire releases is simple and straightforward, but in the case of multiple resources, it isn't a good player. This is where we need another abstraction to cover these issues.

### Scope 

`Scope` is a composable data type for resource management, which wraps the acquisition and release action of a resource. We can think of `Scope` as a handle with built-in acquisition and release logic.

To create a scoped resource, we need to provide `acquire` and `release` action of that resource to the `acquireRelease` constructor:

```scala
val scoped = ZIO.acquireRelease(acquire)(release)
```

We can use scoped resources by calling `scoped` on that. A scoped resource is meant to be used only inside of the `scoped` block. So that resource is not available outside of the `scoped` block. 

Let's try to rewrite a `transfer` example with `Scope`:

```scala
def transfer(from: String, to: String): ZIO[Any, Throwable, Unit] = {
  val resource = for {
    from <- ZIO.acquireRelease(is(from))(close)
    to   <- ZIO.acquireRelease(os(to))(close)
  } yield (from, to)

  ZIO.scoped {
    resource.flatMap { case (in, out) =>
      copy(in, out)
    }
  }
}
```

---

## Scope

The `Scope` data type is the foundation of safe and composable resources handling in ZIO.

Conceptually, a scope represents the lifetime of one or more resources. The resources can be used in the scope and are guaranteed to be released when the scope is closed.

The `Scope` data type takes this idea and represents it as a first class value.

```scala

trait Scope {
  def addFinalizerExit(finalizer: Exit[Any, Any] => UIO[Any]): UIO[Unit]
  def close(exit: => Exit[Any, Any]): UIO[Unit]
}

object Scope {
  def make: UIO[Scope] = ???
}
```

The `addFinalizerExit` operator lets us add a finalizer to the `Scope`. Based on the `Exit` value that the `Scope` is closed with, the finalizer will be run. The finalizer is guaranteed to be run when the scope is closed. The `close` operator closes the scope, running all the finalizers that have been added to the scope. It takes an `Exit` value and runs the finalizers based on that value.

In the following example, we create a `Scope`, add a finalizer to it, and then close the scope:

```scala

for {
  scope <- Scope.make
  _ <- ZIO.debug("Scope is created!")
  _ <- scope.addFinalizer(
    for {
      _ <- ZIO.debug("The finalizer is started!")
      _ <- ZIO.sleep(5.seconds)
      _ <- ZIO.debug("The finalizer is done!")
    } yield ()
  )
  _ <- ZIO.debug("Leaving scope!")
  _ <- scope.close(Exit.succeed(()))
  _ <- ZIO.debug("Scope is closed!")
} yield ()
```

The output of this program will be:

```
Scope is created!
Leaving scope!
The finalizer is started!
The finalizer is done!
Scope is closed!
```

We can see that the finalizer is run after we called `close` on the scope. So the finalizer is guaranteed to be run when the scope is closed.

The `Scope#extend` operator, takes a `ZIO` effect that requires a `Scope` and provides it with a `Scope` without closing it afterwards. This allows us to extend the lifetime of a scoped resource to the lifetime of a scope.

## Scopes and The ZIO Environment

In combination with the ZIO environment, `Scope` gives us an extremely powerful way to manage resources.

We can define a resource using operators such as `ZIO.acquireRelease`, which lets us construct a scoped value from an `acquire` and `release` workflow. For example, here is how we might define a simple resource:

```scala

def acquire(name: => String): ZIO[Any, IOException, Source] =
  ZIO.attemptBlockingIO(Source.fromFile(name))

def release(source: => Source): ZIO[Any, Nothing, Unit] =
  ZIO.succeedBlocking(source.close())

def source(name: => String): ZIO[Scope, IOException, Source] =
  ZIO.acquireRelease(acquire(name))(release(_))
```

Notice that the `acquireRelease` operator added a `Scope` to the environment required by the workflow. This indicates that this workflow needs a `Scope` to be run and will add a finalizer that will close the resource when the scope is closed.

We can continue working with the resource as long as we want by using `flatMap` or other `ZIO` operators. For example, here is how we might read the contents of a file:

```scala
source("cool.txt").flatMap { source =>
  ZIO.attemptBlockingIO(source.getLines())
}
// res2: ZIO[Scope, IOException, Iterator[String]] = FlatMap(
//   trace = "repl.MdocSession.MdocApp.res2(scope.md:85)",
//   first = DynamicNoBox(
//     trace = "repl.MdocSession.MdocApp.source(scope.md:79)",
//     update = 1L,
//     f = zio.ZIO$$$Lambda$20094/0x00007f02a70292e8@3bc37587
//   ),
//   successK = <function1>
// )
```

Once we are finished working with the file, we can close the scope using the `ZIO.scoped` operator. This function creates a new `Scope`, provides it to the workflow, and closes the `Scope` once the workflow is complete:

```scala
object ZIO {
  def scoped[R, E, A](zio: ZIO[Scope with R, E, A]): ZIO[R, E, A] = ???
}
```

The `scoped` operator removes the `Scope` from the environment, indicating that there are no longer any resources used by this workflow that require a scope. We now have a workflow that is ready to run:

```scala
def contents(name: => String): ZIO[Any, IOException, Chunk[String]] =
  ZIO.scoped {
    source(name).flatMap { source =>
      ZIO.attemptBlockingIO(Chunk.fromIterator(source.getLines()))
    }
  }
```

In some cases ZIO applications may provide a `Scope` for us for resources that we don't specify a scope for. For example `ZIOApp` provides a `Scope` for our entire application and ZIO Test provides a `Scope` for each test.

:::note
Please note that like any other services that we can obtain from the ZIO environment, we can do the same with `Scope`. By calling `ZIO.service[Scope]` we can obtain the `Scope` service and then use it to manage resources by adding finalizers to it:

```scala

val resourcefulApp: ZIO[Scope, Nothing, Unit] =
  for {
    scope <- ZIO.service[Scope]
    _     <- ZIO.debug("Entering the scope!")
    _ <- scope.addFinalizer(
      for {
        _ <- ZIO.debug("The finalizer is started!")
        _ <- ZIO.sleep(5.seconds)
        _ <- ZIO.debug("The finalizer is done!")
      } yield ()
    )
    _ <- ZIO.debug("Leaving scope!")
  } yield ()
```

Then we can run the `app` workflow by providing the `Scope` service to it:

```scala
val finalApp: ZIO[Any, Nothing, Unit] =
  Scope.make.flatMap(scope => resourcefulApp.provide(ZLayer.succeed(scope)).onExit(scope.close(_)))
```

Here is the output of the program:

```
Entering the scope!
Leaving scope!
The finalizer is started!
The finalizer is done!
```

So we can think of `Scope` as a service that helps us manage resources effectfully. However, the way we utilized it in the previous example is not as per the best practices, and it was only for educational purposes.

In real-world applications, we can easily manage resources by utilizing high-level operators such as `ZIO.acquireRelease` and `ZIO.scoped`.
:::

## Scopes are Dynamic

One important thing to note about `Scope` is that they are dynamic. This means that if we have an effect that requires a `Scope` we can `flatMap` over that effect and use its value to create a new effect. The new effect extends the lifetime of the original scope. So as we don't close the scope (by calling `ZIO.scoped`) the resources will not be released, and they can become bigger and bigger until we close them:

```scala
ZIO.scoped {
  file("path/to/file.txt").flatMap(getLines).flatMap(processLines) 
}
```

## Defining Resources

We have already seen the `acquireRelease` operator, which is one of the most fundamental operators for creating scoped resources.

```scala
object ZIO {
  def acquireRelease[R, E, A](acquire: => ZIO[R, E, A])(release: A => ZIO[R, Nothing, Any]): ZIO[R with Scope, E, A] =
    ???
}
```

The `acquireRelease` operator performs the `acquire` workflow uninterruptibly. This is important because if we allowed interruption during resource acquisition we could be interrupted when the resource was partially acquired.

The guarantee of the `acquireRelease` operator is that if the `acquire` workflow successfully completes execution then the `release` workflow is guaranteed to be run when the `Scope` is closed.

In addition to the `acquireRelease` operator, there is a more powerful variant called `acquireReleaseExit` that lets the finalizer depend on the `Exit` value that the `Scope` is closed with. This can be useful if we want to run a different finalizer depending on whether the `Scope` was closed with a success or a failure.

```scala
object ZIO {
  def acquireReleaseExit[R, E, A](acquire: => ZIO[R, E, A])(release: (A, Exit[Any, Any]) => ZIO[R, Nothing, Any]): ZIO[R with Scope, E, A] =
    ???
}
```

There is also another family of operators to be aware of that allow the `acquire` workflow to be interrupted.

```scala
object ZIO {
  def acquireReleaseInterruptible[R, E, A](acquire: => ZIO[R, E, A])(release: ZIO[R, Nothing, Any]): ZIO[R with Scope, E, A] =
    ???
  def acquireReleaseInterruptibleExit[R, E, A](acquire: => ZIO[R, E, A])(release: Exit[Any, Any] => ZIO[R, Nothing, Any]): ZIO[R with Scope, E, A] =
    ???
}
```

In this case the `release` workflow is not allowed to depend on the resource, since the `acquire` workflow might be interrupted after partially acquiring the resource. The `release` workflow is responsible for independently determining what finalization is required, for example by inspecting in-memory state.

This is a more advanced variant so we should generally use the standard `acquireRelease` operator. However, the `acquireReleaseInterruptible` operator can be very useful to describe more advanced resource acquisition scenarios where part of the acquisition can be interruptible.

## Converting Resources Into Other ZIO Data Types

We will commonly want to convert scoped resources into other ZIO data types, particularly `ZLayer` for dependency injection and `ZStream`, `ZSink`, and `ZChannel` for streaming.

We can easily do this using the `scoped` constructor on each of these data types. For example, here is how we might convert the `source` resource above into a `ZStream` of the contents:

```scala

def lines(name: => String): ZStream[Any, IOException, String] =
  ZStream.scoped(source(name)).flatMap { source =>
    ZStream.fromIteratorSucceed(source.getLines())
  }
```

Just like the `scoped` operator on `ZIO`, the `scoped` operator on `ZStream` removes the `Scope` from the environment, indicating that there are no longer any resources used by this workflow which require a scope.

The lifetime of these resources will now be governed by the lifetime of the stream, which generally means that the resources will be released as soon as we are done pulling from the stream. This lets the lifetime of these resources be managed by various stream operators to release those resources as efficiently as possible, for example releasing resources associated with each stream as soon as we are done with that stream when we merge two streams.

Similarly, we can convert a scoped resource into a `ZLayer` by using the `scoped` constructor on `ZLayer`:

```scala
def sourceLayer(name: => String): ZLayer[Any, IOException, Source] =
  ZLayer.scoped(source(name))
```

Again, the `Scope` has been removed from the environment, indicating that the lifetime of this resource will no longer be governed by the `Scope` but by the lifetime of the layer. In this case, that means the resource will be released as soon as the workflow that the layer is provided to completes execution, whether by success, failure, or interruption.

We should generally use the `scoped` operators on other ZIO data types to convert a scoped resource into a value of that data type. Having the lifetime of resources governed by the lifetime of those data types makes our code simpler and easier to reason about.

## Controlling Finalizer Ordering

By default, when a `Scope` is closed all finalizers added to that `Scope` will be closed in the reverse of the order in which those finalizers were added to the `Scope`.

Releasing resources in the reverse order in which they were acquired makes sense because a resource that was acquired first may be necessary for a later acquired resource to be closed.

For example, if we open a network connection and then open a file on a remote server we need to close the file before closing the network connection. Otherwise we would no longer be able to interact with the remote server to close the file!

Therefore, in most cases we don't have to do anything with regard to order of finalizers. However, in some cases we may want to run finalizers in parallel instead of sequentially, for example when the resources were also acquired in parallel.

For this we can use the `ZIO.parallelFinalizers` operator to indicate that finalizers should be run in parallel instead of sequentially when a scope is closed. Here is how we could use it to implement an operator that acquires and releases two resources in parallel.

```scala
def zipScoped[R <: Scope, E, A, B](
  left: ZIO[R, E, A],
  right: ZIO[R, E, B]
): ZIO[R, E, (A, B)] =
  ZIO.parallelFinalizers(left.zipPar(right))
```

The `zipPar` operator on `ZIO` takes care of acquiring the resources in parallel and the `parallelFinalizers` operator handles releasing them in parallel. This makes it easy for us to do parallel resource acquisition by leveraging the powerful concurrency operators that already exist on `ZIO`.

## Advanced Scope Operators

So far we have seen that while `Scope` is the foundation of safe and composable resource handling in ZIO, we don't actually need to work with the `Scope` data type directly other than being able to inpect the type signature to see if a workflow is scoped.

In most cases we just use the `acquireRelease` constructor or one of its variants to construct our resource and either work with the resource and close its scope using `ZIO.scoped` or convert the resource into another ZIO data type using an operator such as `ZStream.scoped` or `ZLayer.scoped`. However, for more advanced use cases we may need to work with scopes directly and `Scope` has several useful operators for helping us do so.

### Using a Scope

First, we can `use` a `Scope` by providing it to a workflow that needs a `Scope` and closing the `Scope` immediately after. This is analogous to the `ZIO.scoped` operator:

```scala
trait Closeable extends Scope {
  def use[R, E, A](zio: => ZIO[R with Scope, E, A]): ZIO[R, E, A]
}

object ZIO {
  def scoped[R, E, A](zio: => ZIO[R with Scope, E, A]): ZIO[R, E, A] = ???
}
```

In the following example, we obtained a `Scope` and added a finalizer to it, and then extended its lifetime to the lifetime of the `resource1` and `resource2`:

```scala

object ExtendingScopesExample extends ZIOAppDefault {
  val resource1: ZIO[Scope, Nothing, Unit] =
    ZIO.acquireRelease(ZIO.debug("Acquiring the resource 1"))(_ =>
      ZIO.debug("Releasing the resource one") *> ZIO.sleep(5.seconds)
    )
  val resource2: ZIO[Scope, Nothing, Unit] =
    ZIO.acquireRelease(ZIO.debug("Acquiring the resource 2"))(_ =>
      ZIO.debug("Releasing the resource two") *> ZIO.sleep(3.seconds)
    )

  def run =
    ZIO.scoped(
      for {
        scope <- ZIO.scope
        _     <- ZIO.debug("Entering the main scope!")
        _     <- scope.addFinalizer(ZIO.debug("Releasing the main resource!") *> ZIO.sleep(2.seconds))
        _     <- scope.extend(resource1)
        _     <- scope.extend(resource2)
        _     <- ZIO.debug("Leaving scope!")
      } yield ()
    )

}
```

output:

```
Entering the main scope!
Acquiring the resource 1
Acquiring the resource 2
Leaving scope!
Releasing the resource two
Releasing the resource one
Releasing the main resource!
```

### Extending a Scope

Second, we can use the `extend` operator on `Scope` to provide a workflow with a scope without closing it afterwards. This allows us to extend the lifetime of a scoped resource to the lifetime of a scope, effectively allowing us to "extend" the lifetime of that resource:

```scala
trait Scope {
  def extend[R, E, A](zio: => ZIO[Scope with R, E, A]): ZIO[R, E, A]
}
```

### Closing a Scope

Third, we can `close` a `Scope`. One thing to note here is that by default only the creator of a `Scope` can close it:

```scala
trait Closeable extends Scope {
  def close(exit: => Exit[Any, Any]): UIO[Unit]
}
```

Creating a new `Scope` returns a `Scope.Closeable` which can be closed. Normally users of a `Scope` will only be provided with a `Scope` which does not expose a `close` operator.

This way the creator of a `Scope` can be sure that someone else will not "pull the rug out from under them" by closing the scope prematurely.

---

## ScopedRef: Mutable Reference For Resources

`ScopedRef` is a resourceful version of `Ref` data type. So it is a `Ref` for resourceful effects.

## Operations

There are two basic operations: get and set:

- `ScopedRef#get` returns the current value of the scoped ref.

- `ScopedRef#set` sets the scoped ref to a new value by acquiring the new resource to create a new value of the scoped ref. Setting a new value releases the old resource automatically.

## Construction

The `ScopedRef` has two constructors:

```scala
object ScopedRef {
  def make[A](a: => A): ZIO[Scope, Nothing, ScopedRef[A]] = ???
  def fromAcquire[R, E, A](acquire: ZIO[R, E, A]): ZIO[R with Scope, E, ScopedRef[A]] = ???
}
```

So we have two options to create a `ScopedRef`:

- `ScopedRef.make` creates a scoped ref from an ordinary value. We can use this constructor when we don't need to acquire a resource to create a value of the scoped ref, for example, when we have a constant value.

- `ScopedRef.fromAcquire` creates a scoped ref from an effect that resourcefully produces a value.

:::note
`ScopedRef` is resourceful, so its lifetimes is scoped. Whenever we don't need it anymore, we can release it by using `ZIO#scoped` combinator.
:::

## Example

Let's see how changing the value of a `ScopedRef` automatically releases the old resource:

```scala

object MainApp extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.unit
    r1 = ZIO.acquireRelease(
           ZIO
             .debug("acquiring the first resource")
             .as(5)
         )(_ => ZIO.debug("releasing the first resource"))
    r2 = ZIO.acquireRelease(
           ZIO
             .debug("acquiring the second resource")
             .as(10)
         )(_ => ZIO.debug("releasing the second resource"))
    sref <- ScopedRef.fromAcquire(r1)
    _    <- sref.get.debug
    _    <- sref.set(r2)
    _    <- sref.get.debug
  } yield ()
}
```

The output:

```
acquiring the first resource
5
acquiring the second resource
releasing the first resource
10
releasing the second resource
```

## Interruption

The `set` operation runs in an uninterruptible block. This means that if you want to manage a `Fiber` using a `ScopedRef` by using `forkScoped`, this fiber must be explicitly made interruptible.

---

## ZKeyedPool

The `ZKeyedPool[+Err, -Key, Item]` is a pool of items of type `Item` that are associated with a key of type `Key`. An attempt to get an item from a pool may fail with an error of type `Err`.

The interface is similar to [`ZPool`](zpool.md), but it allows associating items with keys:

```scala
trait ZKeyedPool[+Err, -Key, Item] {
  def get(key: Key): ZIO[Scope, Err, Item]
  def invalidate(item: Item): UIO[Unit]
}
```

The two fundamental operators on a `ZPool` is `get` and `invalidate`:
- The `get` operator retrieves an item associated with the given key from the pool in a scoped effect.
- The `invalidate` operator invalidates the specified item. This will cause the pool to eventually reallocate the item.

There  couple of ways to create a `ZKeyedPool`:

Generally there are two ways to create a `ZKeyedPool`:
1. Fixed-size Pools
2. Dynamic-size Pools

### Fixed-size Pools

1. We can create a pool that has a fixed number of items for each key:

```scala
object ZKeyedPool {
  def make[Key, Env: EnvironmentTag, Err, Item](
       get: Key => ZIO[Env, Err, Item],
       size: => Int
    ): ZIO[Env with Scope, Nothing, ZKeyedPool[Err, Key, Item]] = ???
}
```

For example The `ZKeyedPool.make(key => resource(key), 3)` creates a pool of resources where each key has a pool of size 3:

```scala

object ZKeyedPoolExample extends ZIOAppDefault {
  def resource(key: String): ZIO[Scope, Nothing, String] = ZIO.acquireRelease(
    ZIO.random
      .flatMap(_.nextUUID.map(_.toString))
      .flatMap(uuid => ZIO.debug(s"Acquiring the resource with the $key key and the $uuid id").as(uuid))
  )(uuid => ZIO.debug(s"Releasing the resource with the $key key and the $uuid id!"))

  def run =
    for {
      pool <- ZKeyedPool.make(resource, 3)
      _    <- pool.get("foo")
      item <- pool.get("bar")
      _    <- ZIO.debug(s"Item: $item")
    } yield ()
}
```

Here is an example output of the above code:

```
Acquiring the resource with the foo key and the 82ee3cab-7f4c-47f1-b3e6-0cd49035925d id!
Acquiring the resource with the foo key and the f9cd881f-fa2e-421c-a6ae-c8d16f6b4500 id!
Acquiring the resource with the foo key and the 09a8f4c9-24ee-411c-b1d0-958479266cb0 id!
Acquiring the resource with the bar key and the 4d6f9c95-8d72-4560-bc20-0965b547cfb7 id!
Acquiring the resource with the bar key and the 44bf6641-bb0f-4088-989b-95fb442d93ab id!
Acquiring the resource with the bar key and the fc2780a7-1717-4027-b201-65441168bfce id!
Item: 4d6f9c95-8d72-4560-bc20-0965b547cfb7
Releasing the resource with the bar key and the fc2780a7-1717-4027-b201-65441168bfce id!
Releasing the resource with the bar key and the 44bf6641-bb0f-4088-989b-95fb442d93ab id!
Releasing the resource with the bar key and the 4d6f9c95-8d72-4560-bc20-0965b547cfb7 id!
Releasing the resource with the foo key and the 09a8f4c9-24ee-411c-b1d0-958479266cb0 id!
Releasing the resource with the foo key and the f9cd881f-fa2e-421c-a6ae-c8d16f6b4500 id!
Releasing the resource with the foo key and the 82ee3cab-7f4c-47f1-b3e6-0cd49035925d id!
```

2. We can create a pool that has a fixed number of items but with different pool size for each key:

```scala
object ZKeyedPool {
  def make[Key, Env: EnvironmentTag, Err, Item](
    get: Key => ZIO[Env, Err, Item],
    size: Key => Int
  ): ZIO[Env with Scope, Nothing, ZKeyedPool[Err, Key, Item]] = ???
}
```

In the following example, we have created a pool of resources where based on the key, the pool size for that key is different, the pool size for keys starting with "foo" is 2, and for keys starting with "bar" is 3, and for all other keys, the pool size is 1:

```scala
for {
  pool <- ZKeyedPool.make(resource, (key: String) => key match {
    case k if k.startsWith("foo") => 2
    case k if k.startsWith("bar") => 3
    case _                        => 1
  })
  _    <- pool.get("foo1")
  item <- pool.get("bar1")
  _    <- ZIO.debug(s"Item: $item")
} yield ()
```

Here is an example output of the above code:

```
Acquiring the resource with foo1 key and 052778eb-31c2-4eac-806b-46651813b457 id
Acquiring the resource with foo1 key and bd39dbe4-8f43-4376-a209-5af8ca118af2 id
Acquiring the resource with bar1 key and ecfc80da-c8b2-4726-813c-259748a98c3e id
Acquiring the resource with bar1 key and 0ddfd051-7bf8-4596-a7b9-4011ceeb0976 id
Acquiring the resource with bar1 key and 67239ac8-5def-45ac-962f-b05fb82bf0c3 id
Item: ecfc80da-c8b2-4726-813c-259748a98c3e
Releasing the resource with bar1 key and 67239ac8-5def-45ac-962f-b05fb82bf0c3 id!
Releasing the resource with bar1 key and 0ddfd051-7bf8-4596-a7b9-4011ceeb0976 id!
Releasing the resource with bar1 key and ecfc80da-c8b2-4726-813c-259748a98c3e id!
Releasing the resource with foo1 key and bd39dbe4-8f43-4376-a209-5af8ca118af2 id!
Releasing the resource with foo1 key and 052778eb-31c2-4eac-806b-46651813b457 id!
```

### Dynamic-size Pools

1. We can create a pool with the specified minimum and maximum sized and time to live before a pool whose excess items are not being used will be shrunk down to the minimum size:

```scala
object ZKeyedPool {
  def make[Key, Env: EnvironmentTag, Err, Item](
    get: Key => ZIO[Env, Err, Item],
    range: Key => Range,
    timeToLive: Duration
  ): ZIO[Env with Scope, Nothing, ZKeyedPool[Err, Key, Item]] = ???
}
```

2. Similarly, we can create a pool of resources where the minimum and maximum size of the pool is different for each key. Also, the time to live for each key can be different:

```scala
  def make[Key, Env: EnvironmentTag, Err, Item](
    get: Key => ZIO[Env, Err, Item],
    range: Key => Range,
    timeToLive: Key => Duration
  ): ZIO[Env with Scope, Nothing, ZKeyedPool[Err, Key, Item]] = ???
```

---

## ZPool

A `ZPool[E, A]` is a pool of items of type `A`, each of which may be associated with the acquisition and release of resources. An attempt to get an item `A` from a pool may fail with an error of type `E`.

## Motivation

Acquiring some resources is expensive to create and time-consuming. This includes network connections (sockets, databases, remote services), threads, and so on.

There are some cases that
- We require a **fast and predictable** way of accessing resources.
- We need a solution to **scale across the number of resources**.
- On the other hand, each **resource consumption doesn't take a long time**.

If we create a new resource for every resource acquisition, consequently we will find ourselves in a constant repetition of acquisition and release of resources. This might end up with thousands of resources (e.g. connection to a database) created within a short time, which will reduce the performance of our application.

To address these issues, we can create a pool of pre-initialized resources:
- Whenever we need a new resource, we acquire that from the existing resources of the pool. So the resource acquisition will be predictable, and it will avoid the overhead of acquisition.
- When the resource is no longer needed, we release that back to the resource pool. So the released resources will be recyclable, and it will avoid the overhead of re-acquisition.

`ZPool` is an implementation of such an idea with some excellent properties that we will cover on this page.

## Introduction

`ZPool` is an asynchronous and concurrent generalized pool of reusable resources, that is used to create and manage a pool of objects.

```scala
trait ZPool[+Error, Item] {
  def get: ZIO[Scope, Error, Item]
  def invalidate(item: Item): UIO[Unit]
}
```

The two fundamental operators on a `ZPool` is `get` and `invalidate`:
- The `get` operator retrieves an item from the pool in a scoped effect.
- The `invalidate` operator invalidates the specified item. This will cause the pool to eventually reallocate the item.

Let's assume we have a resource that is expensive to create and release. We want to access this resource frequently, but we don't want to create a new resource every time we need it. We can use `ZPool` to create a pool of resources and acquire them when only needed.

```scala

object PoolExample extends ZIOAppDefault {
  def resource: ZIO[Scope, Nothing, UUID] = ZIO.acquireRelease(
    ZIO.random.flatMap(_.nextUUID).flatMap(uuid => ZIO.debug(s"Acquiring the resource: $uuid!").as(uuid))
  )(uuid => ZIO.debug(s"Releasing the resource $uuid!"))

  def run =
    for {
      pool <- ZPool.make(resource, 3)
      item <- pool.get
      _ <- ZIO.debug(s"Item: $item")
    } yield ()
}
```

In this example, we created a pool of 3 resources. When we call `pool.get`, it will return one of the resources from the pool. After calling the `pool.get` three times, if we call it again, it will block until one of the resources is released back to the pool using `ZPool#invalidate`.

## Constructing ZPools

### Fixed-size Pools

The `make` constructor is a common way to create a `ZPool`:

```scala
object ZPool {
  def make[E, A](get: ZIO[Scope, E, A], size: Int): ZIO[Scope, Nothing, ZPool[E, A]] = ???
}
```

It takes a scoped resource of type `A`, and the `size` of the pool. The return type will be a scoped `ZPool`.
- A fixed pool size will be used to pre-allocate pool entries, so all the entries of the pool will be acquired eagerly. As a client of the `ZPool` it is recommended to analyze requirements to find out the best suitable size for the resource pool. If we set up a pool with too many eagerly-acquired resources, that may reduce the performance due to the resource contention.
- As the return type of the constructor is `ZIO[Scope, Nothing, ZPool[E, A]]`, it will manage automatically the life cycle of the pool. So, as a client of `ZPool`, we do not require to shutdown the pool manually.

There is another constructor called `ZPool.fromIterable` that is suitable when no cleanup or release actions are required.

### Dynamically-sized Pools

The previous constructor creates a `ZPool` with a fixed size, so all of its entries are pre-allocated. There is another constructor that creates a pool with pre-allocated minimum entries (eagerly-acquired resources), plus it can increase its entries _on-demand_ (lazily-acquired resources) until reaches the specified maximum size:

```scala
def make[E, A](get: ZIO[Scope, E, A],
        range: Range, // minimum and maximum size of the pool
        timeToLive: Duration): ZIO[Scope, Nothing, ZPool[E, A]]
```

Having a lot of resources that are over our average requirement can waste space and degrade the performance. Therefore, this variant of `ZPool` has an _eviction policy_. By taking the `timeToLive` argument, it will evict excess items that have not been acquired for more than the `timeToLive` time, until it reaches the minimum size.

Here is an example of creating pool of database connections:

```scala
ZIO.scoped {
  ZPool.make(acquireDbConnection, 10 to 20, 60.seconds).flatMap { pool =>
    ZIO.scoped {
      pool.get.flatMap { conn => useConnection(conn) }
    }
  }
}
```

## Operators on ZPool

### Resource Acquisition

After creating a pool, we can retrieve a resource from the pool using `ZPool#get` operation:

```scala
trait ZPool[+Error, Item] {
  def get: ZIO[Scope, Error, Item]
}
```

Here is how it works behind the scenes:
- If there is any resource available in the pre-allocated entries, it will return one of those.
- If the demand exceeds the available resources in the pool, one new resource will be allocated on-demand and returned to the client.
- If the demand exceeds the maximum size of the pool, it will block while it waits for a resource to become available.
- If the acquisition of a resource fails, then the returned effect will fail for the same reason.

In case of failure, we can retry a failed acquisition. It will repeat the acquisition attempt:

```scala
ZPool.make(acquireDbConnection, 10).flatMap { pool =>
  pool.get.flatMap( conn => useConnection(conn)).eventually
}
```

### Resource Invalidation

When we are working with resources, especially the remote ones, they may become invalid or faulty. On the other hand, when the resource's life cycle ends, it will be automatically returned to the resource pool. So what happens if another client acquires a resource from the pool? The faulty resource can cause a problem for the next client. To prevent that, we can use the `ZPool#invalidate` to claim that the resource is invalid:

```scala
trait ZPool[+Error, Item] {
  def invalidate(item: Item): UIO[Unit]
}
```

In this case, the pool will eventually reallocate the item, but this will happen lazily rather than eagerly.

---

## Built-in Schedules

## succeed

Returns a schedule that repeats one time, producing the specified constant value:

```scala
val constant = Schedule.succeed(5)
```

## fromFunction

A schedule that always recurs, mapping input values through the specified function:

```scala
val inc = Schedule.fromFunction[Int, Int](_ + 1)
```

## stop

A schedule that does not recur, just stops and returns one `Unit` element:

```scala
val stop = Schedule.stop
```

## once

A schedule that recurs one time an returns one `Unit` element:

```scala
val once = Schedule.once
```

## forever

A schedule that always recurs and produces number of recurrence at each run:

```scala
val forever = Schedule.forever
```

## recurs

A schedule that only recurs the specified number of times:

```scala
val recurs = Schedule.recurs(5)
```

## spaced

A schedule that recurs continuously, each repetition spaced the specified duration from the last run:

```scala
val spaced = Schedule.spaced(10.milliseconds)
```

## fixed

A schedule that recurs on a fixed interval. Returns the number of repetitions of the schedule so far:

```scala
val fixed = Schedule.fixed(10.seconds)
```

## exponential

A schedule that recurs using exponential backoff:

```scala
val exponential = Schedule.exponential(10.milliseconds)
```

## fibonacci

A schedule that always recurs, increasing delays by summing the preceding two delays (similar to the fibonacci sequence). Returns the current duration between recurrences:

```scala
val fibonacci = Schedule.fibonacci(10.milliseconds)
```

## identity

A schedule that always decides to continue. It recurs forever, without any delay. `identity` schedule consumes input, and emit the same as output (`Schedule[Any, A, A]`):

```scala
val identity = Schedule.identity[Int]
```

## unfold

A schedule that repeats one time from the specified state and iterator:

```scala
val unfold = Schedule.unfold(0)(_ + 1)
```

---

## Schedule Combinators

Schedules define stateful, possibly effectful, recurring schedules of events, and compose in a variety of ways. Combinators allow us to take schedules and combine them together to get other schedules and if we have combinators with just the right properties. Then in theory we should be able to solve an infinite number of problems, with only a few combinators and few base schedules.

## Composition

Schedules compose in the following primary ways:

* **Union**. This performs the union of the intervals of two schedules.
* **Intersection**. This performs the intersection of the intervals of two schedules.
* **Sequencing**. This concatenates the intervals of one schedule onto another.

### Union

Combines two schedules through union, by recurring if either schedule wants to
recur, using the minimum of the two delays between recurrences.

|                     | `s1`                | `s2`                | `s1` &#124; &#124; `s2`   |
|---------------------|---------------------|---------------------|---------------------------|
| Type                | `Schedule[R, A, B]` | `Schedule[R, A, C]` | `Schedule[R, A, (B, C)]`  |
| Continue: `Boolean` | `b1`                | `b2`                | `b1` &#124; &#124; `b2`   |
| Delay: `Duration`   | `d1`                | `d2`                | `d1.min(d2)`              |
| Emit: `(A, B)`      | `a`                 | `b`                 | `(a, b)`                  |

We can combine two schedule through union with `||` operator:

```scala
val expCapped = Schedule.exponential(100.milliseconds) || Schedule.spaced(1.second)
```

### Intersection

Combines two schedules through the intersection, by recurring only if both schedules want to recur, using the maximum of the two delays between recurrences.

|                     | `s1`                | `s2`                | `s1 && s2`               |
|---------------------|---------------------|---------------------|--------------------------|
| Type                | `Schedule[R, A, B]` | `Schedule[R, A, C]` | `Schedule[R, A, (B, C)]` |
| Continue: `Boolean` | `b1`                | `b2`                | `b1 && b2`               |
| Delay: `Duration`   | `d1`                | `d2`                | `d1.max(d2)`             |
| Emit: `(A, B)`      | `a`                 | `b`                 | `(a, b)`                 |

We can intersect two schedule with `&&` operator:

```scala
val expUpTo10 = Schedule.exponential(1.second) && Schedule.recurs(10)
```

### Sequencing

Combines two schedules sequentially, by following the first policy until it ends, and then following the second policy.

|                   | `s1`                | `s2`                | `s1 andThen s2`     |
|-------------------|---------------------|---------------------|---------------------|
| Type              | `Schedule[R, A, B]` | `Schedule[R, A, C]` | `Schedule[R, A, C]` |
| Delay: `Duration` | `d1`                | `d2`                | `d1 + d2`           |
| Emit: `B`         | `a`                 | `b`                 | `b`                 |

We can sequence two schedule by using `andThen`:

```scala
val sequential = Schedule.recurs(10) andThen Schedule.spaced(1.second)
```

## Piping

Combine two schedules by piping the output of the first schedule to the input of the other. Effects described by the first schedule will always be executed before the effects described by the second schedule.

|                   | `s1`                | `s2`                | `s1 >>> s2`         |
|-------------------|---------------------|---------------------|---------------------|
| Type              | `Schedule[R, A, B]` | `Schedule[R, B, C]` | `Schedule[R, A, C]` |
| Delay: `Duration` | `d1`                | `d2`                | `d1 + d2`           |
| Emit: `B`         | `a`                 | `b`                 | `b`                 | 

We can pipe two schedule by using `>>>` operator:

```scala
val totalElapsed = Schedule.spaced(1.second) <* Schedule.recurs(5) >>> Schedule.elapsed
```

## Jittering

A `jittered` is a combinator that takes one schedule and returns another schedule of the same type except for the delay which is applied randomly:

| Function   | Input Type                 | Output Type                          |
|------------|----------------------------|--------------------------------------|
| `jittered` |                            | `Schedule[Env with Random, In, Out]` |
| `jittered` | `min: Double, max: Double` | `Schedule[Env with Random, In, Out]` |

We can jitter any schedule by calling `jittered` on it:

```scala
val jitteredExp = Schedule.exponential(10.milliseconds).jittered
```

When a resource is out of service due to overload or contention, retrying and backing off doesn't help us. If all failed API calls are backed off to the same point of time, they cause another overload or contention. Jitter adds some amount of randomness to the delay of the schedule. This helps us to avoid ending up accidentally synchronizing and taking the service down by accident.

The form with parameters `min` and `max` creates a new schedule where the new interval size is randomly distributed between `min * old interval` and `max * old interval`.

[Research](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) shows that `Schedule.jittered(0.0, 1.0)` is very suitable for retrying.

## Collecting

A `collectAll` is a combinator that when we call it on a schedule, produces a new schedule that collects the outputs of the first schedule into a chunk.

| Function     | Input Type               | Output Type                     |
|--------------|--------------------------|---------------------------------|
| `collectAll` | `Schedule[Env, In, Out]` | `Schedule[Env, In, Chunk[Out]]` |

In the following example, we are catching all recurrence of schedule into `Chunk`, so at the end, it would contain `Chunk(0, 1, 2, 3, 4)`:

```scala
val collect = Schedule.recurs(5).collectAll
```

## Filtering

We can filter inputs or outputs of a schedule with `whileInput` and `whileOutput`. Alse ZIO schedule has an effectful version of these two functions, `whileInputZIO` and `whileOutputZIO`.

| Function         | Input Type                   | Output Type                |
|------------------|------------------------------|----------------------------|
| `whileInput`     | `In1 => Boolean`             | `Schedule[Env, In1, Out]`  |
| `whileOutput`    | `Out => Boolean`             | `Schedule[Env, In, Out]`   |
| `whileInputZIO`  | `In1 => URIO[Env1, Boolean]` | `Schedule[Env1, In1, Out]` |
| `whileOutputZIO` | `Out => URIO[Env1, Boolean]` | `Schedule[Env1, In, Out]`  |

In following example we collect all emiting outputs before reaching the 5 output, so it would return `Chunk(0, 1, 2, 3, 4)`:

```scala
val res = Schedule.unfold(0)(_ + 1).whileOutput(_ < 5).collectAll
```

## Mapping

There are two versions for mapping schedules, `map` and its effectful version `mapZIO`.

| Function | Input Type                   | Output Type                |
|----------|------------------------------|----------------------------|
| `map`    | `f: Out => Out2`             | `Schedule[Env, In, Out2]`  |
| `mapZIO` | `f: Out => URIO[Env1, Out2]` | `Schedule[Env1, In, Out2]` |

## Left/Right Ap

Sometimes when we intersect two schedules with the `&&` operator, we just need to ignore the left or the right output.
- * `*>` ignore the left output
- * `<*` ignore the right output

## Modifying

Modifies the delay of a schedule:

```scala
val boosted = Schedule.spaced(1.second).delayed(_ => 100.milliseconds)
```

## Tapping

Whenever we need to effectfully process each schedule input/output, we can use `tapInput` and `tapOutput`.

We can use these two functions for logging purposes:

```scala
val tappedSchedule = Schedule.count.whileOutput(_ < 5).tapOutput(o => Console.printLine(s"retrying $o").orDie)
```

---

## Examples(Schedule)

Let's try some example of creating and combining schedules.

1. Stop retrying after a specified amount of time has elapsed:

```scala
val expMaxElapsed = (Schedule.exponential(10.milliseconds) >>> Schedule.elapsed).whileOutput(_ < 30.seconds)
```

2. Retry only when a specific exception occurs:

```scala

val whileTimeout = Schedule.exponential(10.milliseconds) && Schedule.recurWhile[Throwable] {
  case _: TimeoutException => true
  case _ => false
}
```

---

## Introduction to Scheduling ZIO Effects

A `Schedule[Env, In, Out]` is an **immutable value** that **describes** a recurring effectful schedule, which runs in some environment `Env`, after consuming values of type `In` (errors in the case of `retry`, or values in the case of `repeat`) produces values of type `Out`, and in every step based on input values and the internal state decides to halt or continue after some delay **d**.

Schedules are defined as a possibly infinite set of intervals spread out over time. Each interval defines a window in which recurrence is possible.

[Repetition](repetition.md) and [retrying](retrying.md) are two similar concepts in the domain of scheduling. It is the same concept and idea, only one of them looks for successes and the other one looks for failures. 

When schedules are used to repeat or retry effects, the starting boundary of each interval produced by a schedule is used as the moment when the effect will be executed again. 

Schedules allow us to define and compose flexible recurrence schedules, which can be used to **repeat** actions, or **retry** actions in the event of errors. We will discuss them on the following pages.

A variety of [combinators](combinators.md) exist for transforming and combining schedules, and the companion object for `Schedule` contains [all common types of schedules](built-in-schedules.md), both for performing retrying and repetition.

---

## Repetition

In the case of repetition, ZIO has a `ZIO#repeat` function, which takes a schedule as a repetition policy and returns another effect that describes an effect with repetition strategy according to that policy.

Repeat policies are used in the following functions:

* `ZIO#repeat` — Repeats an effect until the schedule is done.
* `ZIO#repeatOrElse` — Repeats an effect until the schedule is done, with a fallback for errors.

> _**Note:**_
>
> Scheduled recurrences are in addition to the first execution, so that `io.repeat(Schedule.once)` yields an effect that executes `io`, and then if that succeeds, executes `io` an additional time.

Let's see how we can create a repeated effect by using `ZIO#repeat` function:

```scala
val action:      ZIO[R, E, A] = ???
val policy: Schedule[R1, A, B] = ???

val repeated = action repeat policy
```

There is another version of `repeat` that helps us to have a fallback strategy in case of errors, if something goes wrong we can handle that by using the `ZIO#repeatOrElse` function, which helps up to add an `orElse` callback that will run in case of repetition failure:

```scala
val action:       ZIO[R, E, A] = ???
val policy: Schedule[R1, A, B] = ???

val orElse: (E, Option[B]) => ZIO[R1, E2, B] = ???

val repeated = action repeatOrElse (policy, orElse)
```

---

## Retrying(Schedule)

In the case of retrying, ZIO has a `ZIO#retry` function, which takes a schedule as a repetition policy and returns another effect that describes an effect with repetition strategy which will retry following the failure of the original effect.

Repeat policies are used in the following functions:

* `ZIO#retry` – Retries an effect until it succeeds.
* `ZIO#retryOrElse` — Retries an effect until it succeeds, with a fallback for errors.

Let's see how we can create a repeated effect by using `ZIO#retry` function:

```scala
val action:       ZIO[R, E, A] = ???
val policy: Schedule[R1, E, S] = ???

val repeated = action retry policy

```

There is another version of `retry` that helps us to have a fallback strategy in case of erros, if something goes wrong we can handle that by using the `ZIO#retryOrElse` function, which helps up to add an `orElse` callback that will run in case of failure of repetition failure:

```scala
val action:       ZIO[R, E, A] = ???
val policy: Schedule[R1, A, B] = ???

val orElse: (E, S) => ZIO[R1, E1, A1] = ???

val repeated = action retryOrElse (policy, orElse)
```

---

## Accessor Methods (deprecated)

Accessor methods are little helper methods that lookup a service from the environment, and then forward your call to
that service.

:::info
The [service pattern](service-pattern.md) provides a better way to structure programs, and it does not need accessor
methods. Therefore, accessor methods are now deprecated.
:::

## What is an Accessor Method?

Imagine a service defined as:

```scala

trait BlobStorage {
  def get(id: String): ZIO[Any, Throwable, Array[Byte]]

  def put(content: Array[Byte]): ZIO[Any, Throwable, String]
}
```

The accessor methods are then defined as:

```scala

object BlobStorage {
  // Accessor method for BlobStorage.get
  def get(id: String): ZIO[BlobStorage, Throwable, Array[Byte]] =
    ZIO.serviceWithZIO[BlobStorage](_.get(id))

  // Accessor method for BlobStorage.put
  def put(content: Array[Byte]): ZIO[BlobStorage, Throwable, String] =
    ZIO.serviceWithZIO[BlobStorage](_.put(content))
}
```

Each accessor method fetches the service from the environment, and then immediately forwards the method call.

The service can now be used as:

```scala
  BlobStorage.get("blob-id")  // returns a ZIO[BlobStorage, Throwable, Array[Byte]]
```

Notice how the `BlobStorage` trait is in the environment (the `R` channel) of the returned ZIO.

## Why are accessor methods deprecated?

Accessor methods have some drawbacks:

- You must write more code.
- The extra code must stay in sync with the service's trait.
- The service is looked up in the environment each time it is used. This incurs (a small) performance penalty.
- The ZIO environment permeates deeper into your code than strictly necessary. This problem is exacerbated when
  services start exposing the services they depend on in the `R` channel of their method's return types.

The recommended [service pattern](service-pattern.md) injects service dependencies directly, and therefore has none of
these problems.

## Monomorphic Services

We can use the `@accessible` macro to generate _service member accessors_:

```scala

@accessible
trait ServiceA {
  def method(input: Something): UIO[Unit]
}

// below will be autogenerated
object ServiceA {
  def method(input: Something) =
    ZIO.serviceWithZIO[ServiceA](_.method(input))
}
```

For normal values, a `ZIO` with `Nothing` on error channel is generated:

```scala

@accessible
trait ServiceB {
  def pureMethod(input: Something): SomethingElse
}

// below will be autogenerated
object ServiceB {
  def pureMethod(input: Something): ZIO[ServiceB, Nothing, SomethingElse] =
    ZIO.serviceWith[ServiceB](_.pureMethod(input))
}
```

The `@throwing` annotation will mark impure methods. Using this annotation will request ZIO to push the error on the
error channel:

```scala

@accessible
trait ServiceC {
  @throwing
  def impureMethod(input: Something): SomethingElse
}

// below will be autogenerated
object ServiceC {
  def impureMethod(input: Something): ZIO[ServiceC, Throwable, SomethingElse] =
    ZIO.serviceWithZIO[ServiceC](s => ZIO(s.impureMethod(input)))
}
```

Below is a fully working example:

```scala

@accessible
trait KeyValueStore {
  def set(key: String, value: Int): Task[Int]

  def get(key: String): Task[Int]
}

case class InmemoryKeyValueStore(map: Ref[Map[String, Int]])
  extends KeyValueStore {
  override def set(key: String, value: Int): Task[Int] =
    map.update(_.updated(key, value)).map(_ => value)

  override def get(key: String): Task[Int] =
    map.get.map(_.get(key)).someOrFailException
}

object InmemoryKeyValueStore {
  val layer: ULayer[KeyValueStore] =
    ZLayer {
      for {
        map <- Ref.make(Map[String, Int]())
      } yield InmemoryKeyValueStore(map)
    }
}

object MainApp extends ZIOAppDefault {
  val myApp =
    for {
      _ <- KeyValueStore.set("key", 5)
      key <- KeyValueStore.get("key")
    } yield key

  def run = myApp.provide(InmemoryKeyValueStore.layer).debug
}
```

## Writing Polymorphic Services

### With Proper Type Parameters

If the service is polymorphic for some proper types, we can use the `@accessible` macro like previous examples.

Assume we have a `KeyValueStore` like below, as we will see using `@accessible` will generate us the accessor methods:

```scala

@accessible
trait KeyValueStore[K, V] {
  def set(key: K, value: V): Task[V]

  def get(key: K): Task[V]
}

case class InmemoryKeyValueStore(map: Ref[Map[String, Int]])
  extends KeyValueStore[String, Int] {
  override def set(key: String, value: Int): Task[Int] =
    map.update(_.updated(key, value)).map(_ => value)

  override def get(key: String): Task[Int] =
    map.get.map(_.get(key)).someOrFailException
}

object InmemoryKeyValueStore {
  val layer: ULayer[KeyValueStore[String, Int]] =
    ZLayer {
      for {
        map <- Ref.make(Map[String, Int]())
      } yield InmemoryKeyValueStore(map)
    }
}

object MainApp extends ZIOAppDefault {
  val myApp =
    for {
      _ <- KeyValueStore.set("key", 5)
      key <- KeyValueStore.get[String, Int]("key")
    } yield key

  def run = myApp.provide(InmemoryKeyValueStore.layer).debug
}
```

### With Higher-Kinded Type Parameters (`F[_]`)

If a service has a higher-kinded type parameter like `F[_]` we should use the `accessibleM` macro. Here is an example of
such a service:

```scala

@accessibleM[Task]
trait KeyValueStore[K, V, F[_]] {
  def set(key: K, value: V): F[V]

  def get(key: K): F[V]
}

case class InmemoryKeyValueStore(map: Ref[Map[String, Int]])
  extends KeyValueStore[String, Int, Task] {
  override def set(key: String, value: Int): Task[Int] =
    map.update(_.updated(key, value)).map(_ => value)

  override def get(key: String): Task[Int] =
    map.get.map(_.get(key)).someOrFailException

}

object InmemoryKeyValueStore {
  val layer: ULayer[KeyValueStore[String, Int, Task]] =
    ZLayer {
      for {
        map <- Ref.make(Map[String, Int]())
      } yield InmemoryKeyValueStore(map)
    }
}

object MainApp extends ZIOAppDefault {
  val myApp =
    for {
      key <- KeyValueStore.set[String, Int]("key", 5)
      _ <- KeyValueStore.get[String, Int]("key")
    } yield key

  def run = myApp.provide(InmemoryKeyValueStore.layer).debug
}

```

### With Higher-Kinded Type Parameters (`F[_, _]`)

If the service has a higher-kinded type parameter like `F[_, _]` we should use the `accessibleMM` macro. Let's see an
example:

```scala

@accessibleMM[IO]
trait KeyValueStore[K, V, E, F[_, _]] {
  def set(key: K, value: V): F[E, V]

  def get(key: K): F[E, V]
}

case class InmemoryKeyValueStore(map: Ref[Map[String, Int]])
  extends KeyValueStore[String, Int, String, IO] {
  override def set(key: String, value: Int): IO[String, Int] =
    map.update(_.updated(key, value)).map(_ => value)

  override def get(key: String): IO[String, Int] =
    map.get.map(_.get(key)).someOrFail(s"key not found: $key")
}

object InmemoryKeyValueStore {
  val layer: ULayer[KeyValueStore[String, Int, String, IO]] =
    ZLayer {
      for {
        map <- Ref.make(Map[String, Int]())
      } yield InmemoryKeyValueStore(map)
    }
}

object MainApp extends ZIOAppDefault {
  val myApp =
    for {
      _ <- KeyValueStore.set[String, Int, String]("key", 5)
      key <- KeyValueStore.get[String, Int, String]("key")
    } yield key

  def run = myApp.provide(InmemoryKeyValueStore.layer).debug

}
```

---

## Defining Polymorphic Services in ZIO

As we discussed [here](../contextual/zenvironment.md), the `ZEnvironment`, which is the underlying data type used by `ZLayer`, is backed by a type-level mapping from types of services to implementations of those services. This functionality is backed by `izumi.reflect.Tag`, which captures a type as a value.

We just need to know what is the type of service when we put it in the `ZEnvironment` because `ZEnvironment` is essentially a map from _service types (interfaces)_ to _implementation of those interfaces_. To implement the map, the `ZEnvironment` needs a type tag for the new service, and also needs a way to remove the old service from the type level map. So we should have service type information at the runtime.

We can think of `Tag[A]` as like a `TypeTag[A]` or `ClassTag[A]` from the Scala standard library but available on a cross-version and cross-platform basis. Basically, it carries information about a certain type into runtime that was available at compile time. Methods that construct `ZEnvironment` values generally require a tag for the value being included in the “bundle of services”.

As a user, we should not normally interact with `Tag` except where we define polymorphic services. In general, a `Tag` should always be available whenever we have a concrete type. The only time we should have to use it is when we have a _polymorphic service_. If we are using polymorphic code, we need to provide implicit evidence that a tag exists for that type (`implicit tag: Tag[A]`) or as a context-bound for that type parameter: (`A: Tag`).

Let's try to write a polymorphic service. Assume we have the following service interface:

```scala
trait KeyValueStore[K, V, E, F[_, _]] {
  def get(key: K): F[E, V]

  def set(key: K, value: V): F[E, V]

  def remove(key: K): F[E, Unit]
}
```

In the next step, we are going to write its accessors (note: [accessor methods](accessor-methods.md) are deprecated). We might end up with the following snippet code:

```scala

object KeyValueStore {
  def get[K, V, E](key: K): ZIO[KeyValueStore[K, V, E, IO], E, V] =
    ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.get(key))

  def set[K, V, E](key: K, value: V): ZIO[KeyValueStore[K, V, E, IO], E, V] =
    ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.set(key, value))

  def remove[K, V, E](key: K): ZIO[KeyValueStore[K, V, E, IO], E, Unit] =
    ZIO.serviceWithZIO(_.remove(key))
}

// error: could not find implicit value for izumi.reflect.Tag[K]. Did you forget to put on a Tag, TagK or TagKK context bound on one of the parameters in K? e.g. def x[T: Tag, F[_]: TagK] = ...
// 
// 
// <trace>: 
//   deriving Tag for K, dealiased: K:
//   could not find implicit value for Tag[K]: K is a type parameter without an implicit Tag!
//     ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.get(key))
//                                                   ^
// error: could not find implicit value for izumi.reflect.Tag[K]. Did you forget to put on a Tag, TagK or TagKK context bound on one of the parameters in K? e.g. def x[T: Tag, F[_]: TagK] = ...
// 
// 
// <trace>: 
//   deriving Tag for K, dealiased: K:
//   could not find implicit value for Tag[K]: K is a type parameter without an implicit Tag!
//     ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.set(key, value))
//                                                   ^
// error: could not find implicit value for izumi.reflect.Tag[K]. Did you forget to put on a Tag, TagK or TagKK context bound on one of the parameters in K? e.g. def x[T: Tag, F[_]: TagK] = ...
// 
// 
// <trace>: 
//   deriving Tag for K, dealiased: K:
//   could not find implicit value for Tag[K]: K is a type parameter without an implicit Tag!
//     ZIO.serviceWithZIO(_.remove(key))
//                       ^
```

The compiler generates the following errors:

```
could not find implicit value for izumi.reflect.Tag[K]. Did you forget to put on a Tag, TagK or TagKK context bound on one of the parameters in K? e.g. def x[T: Tag, F[_]: TagK] = ...

<trace>: 
  deriving Tag for K, dealiased: K:
  could not find implicit value for Tag[K]: K is a type parameter without an implicit Tag!
    ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.get(key))
```

As the compiler says, we should put `Tag` as a context-bound for `K`, `V`, and `E` type parameters:

```scala

object KeyValueStore {
  def get[K: Tag, V: Tag, E: Tag](key: K): ZIO[KeyValueStore[K, V, E, IO], E, V] =
    ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.get(key))

  def set[K: Tag, V: Tag, E: Tag](key: K, value: V): ZIO[KeyValueStore[K, V, E, IO], E, V] =
    ZIO.serviceWithZIO[KeyValueStore[K, V, E, IO]](_.set(key, value))

  def remove[K: Tag, V: Tag, E: Tag](key: K): ZIO[KeyValueStore[K, V, E, IO], E, Unit] =
    ZIO.serviceWithZIO(_.remove(key))
}
```

Now, we can continue and implement the in-memory version of this key-value store:

```scala
case class InmemoryKeyValueStore(map: Ref[Map[String, Int]])
  extends KeyValueStore[String, Int, String, IO] {

  override def get(key: String): IO[String, Int] =
    map.get.map(_.get(key)).someOrFail(s"$key not found")

  override def set(key: String, value: Int): IO[String, Int] =
    map.update(_.updated(key, value)).map(_ => value)

  override def remove(key: String): IO[String, Unit] =
    map.update(_.removed(key))
}

object InmemoryKeyValueStore {
  def layer: ULayer[KeyValueStore[String, Int, String, IO]] =
    ZLayer {
      Ref.make(Map[String, Int]()).map(InmemoryKeyValueStore.apply)
    }
}
```

The last step is to use the service in a ZIO application:

```scala

object MainApp extends ZIOAppDefault {

  val myApp: ZIO[KeyValueStore[String, Int, String, IO], String, Unit] =
    for {
      _ <- KeyValueStore.set[String, Int, String]("key1", 3).debug
      _ <- KeyValueStore.get[String, Int, String]("key1").debug
      _ <- KeyValueStore.remove[String, Int, String]("key1")
      _ <- KeyValueStore.get[String, Int, String]("key1").either.debug
    } yield ()

  def run = myApp.provide(InmemoryKeyValueStore.layer)
  
}

// Output:
// 3
// 3
// not found
```

Note that in the above example, one might want to write accessors more polymorphic. So in this case we should add `TagKK` as a context-bound of the `F` type parameter:

```scala
object KeyValueStore {
  def get[K: Tag, V: Tag, E: Tag, F[_, _] : TagKK](key: K): ZIO[KeyValueStore[K, V, E, F], Nothing, F[E, V]] =
    ZIO.serviceWith[KeyValueStore[K, V, E, F]](_.get(key))

  def set[K: Tag, V: Tag, E: Tag, F[_, _] : TagKK](key: K, value: V): ZIO[KeyValueStore[K, V, E, F], Nothing, F[E, V]] =
    ZIO.serviceWith[KeyValueStore[K, V, E, F]](_.set(key, value))

  def remove[K: Tag, V: Tag, E: Tag, F[_, _] : TagKK](key: K): ZIO[KeyValueStore[K, V, E, F], E, Unit] =
    ZIO.serviceWith(_.remove(key))
}
```

---

## Introduction to Writing ZIO Services

Defining services in ZIO is not very different from object-oriented style, it has the same principle: coding to an interface, not an implementation. Therefore, ZIO encourages us to implement this principle by using _Service Pattern_, which is quite similar to the object-oriented style.

Before diving into writing services in ZIO style, let's review how we define them in an object-oriented fashion in the next section.

## Defining Services in OOP

Here are the steps we take to implement a service in object-oriented programming:

1. **Service Definition**— In object-oriented programming, we define services with traits. A service is a bundle of related functionality that is defined in a trait:

```scala
trait FooService {

}
```

2. **Service Implementation**— We implement these services by using classes:

```scala
class FooServiceImpl extends FooService {
    
}
```

3. **Defining Dependencies**— If the creation of a service depends on other services, we can define these dependencies by using constructors:

```scala
trait ServiceA {

}

trait ServiceB {

}

class FooServiceImpl(a: ServiceA, b: ServiceB) extends FooService {

}
```

In object-oriented programming, the best practice is to _program to an interface, not an implementation_. So in the previous example, `ServiceA` and `ServiceB` are interfaces, not concrete classes.

4. **Injecting Dependencies**— Now, the client of `FooServiceImpl` service can provide its own implementation of `ServiceA` and `ServiceB`, and inject them to the `FooServiceImpl` constructor:

```scala
class ServiceAImpl extends ServiceA
class ServiceBImpl extends ServiceB
val fooService = new FooServiceImpl(new ServiceAImpl, new ServiceBImpl)
```

Sometimes, as the number of dependent services grows and the dependency graph of our application becomes complicated, we need an automatic way of wiring and providing dependencies into the services of our application. In these situations, we might use a dependency injection framework to do all its magic machinery for us.

## Defining Services in ZIO

A service is a group of functions that deals with only one concern. Keeping the scope of each service limited to a single responsibility improves our ability to understand code, in that we need to focus only on one topic at a time without juggling too many concepts together in our head.

In functional Scala as well as in object-oriented programming the best practice is to _Program to an Interface, Not an Implementation_. This is the most important design principle in software development and helps us to write maintainable code by:

* Allowing the client to hold an **interface as a contract** and don't worry about the implementation. The interface signature determines all operations that should be done.

* Enabling a developer to **write more testable programs**. When we write a test for our business logic we don't have to run and interact with real services like databases which makes our test run very slow. If our code is correct our test code should always pass, there should be no hidden variables or depend on outside sources. We can't know that the database is always running correctly. We don't want to fail our tests because of the failure of external service.

* Providing the ability to **write more modular applications**. So we can plug in different implementations for different purposes without a major modification.

It is not mandatory, but ZIO encourages us to follow this principle by bundling related functionality as an interface by using the _Service Pattern_.

The core idea is that a layer depends upon the interfaces exposed by the layers immediately below itself, but is completely unaware of its dependencies' internal implementations.

In object-oriented programming:

- **Service Definition** is done by using _interfaces_ (Scala trait or Java Interface).
- **Service Implementation** is done by implementing interfaces using _classes_ or creating _new object_ of the interface.
- **Defining Dependencies** is done by using _constructors_. They allow us to build classes, given their dependencies. This is called constructor-based dependency injection.

We have a similar analogy in the Service Pattern, except instead of using _constructors_ we use **`ZLayer`** to define dependencies. So in ZIO fashion, we can think of `ZLayer` as a service constructor.

---

## Introduction to Reloadable Services

Reloadable services are a feature in ZIO that allow us to reload services when necessary. With ZIO When we reload a service, it will automatically deallocate any resources that the service was using. This includes any open files, network connections, or database connections. ZIO will then reallocate new resources for the new service. This process of deallocation and reallocation is handled automatically by ZIO, so you don't need to worry about it. 

Here are some examples of how we might use reloadable services:

- **Config changes**: If we make a change to the configuration of a service, we might want to reload the service so that it can pick up the new configuration.
- **Scheduled reloads**: We might want to reload a service on a regular interval, such as every n minutes. This can be useful for services that need to be refreshed on a regular basis.
- **Reload on change on Schema**: Assume we have a service that reads data from a database. If we make a change to the database schema, we might want to reload the service so that it can pick up the new schema. 

This article explores two methods for implementing reloadable services in ZIO:

1. The first method is a non-trivial method that uses the `Reloadable` service, which requires some boilerplate code.
2. The second method is a simpler method introduced by `zio-macros` that uses the `ServiceReloader` service.

Before going into further detail, through this article whenever we use `Counter` class, we refere to this source code:

```scala

trait Counter {
  def increment: UIO[Unit]
  def get: UIO[Int]
}

object Counter {
  val increment: ZIO[Counter, Nothing, Unit] =
    ZIO.serviceWithZIO[Counter](_.increment)

  val get: ZIO[Counter, Nothing, RuntimeFlags] =
    ZIO.serviceWithZIO[Counter](_.get)

  val live: ZLayer[Any, Nothing, Counter] = ZLayer.scoped {
    for {
      id <- Ref.make(UUID.randomUUID())
      ref <- Ref.make(0)
      service = CounterLive(id, ref)
      _ <- service.acquire
      _ <- ZIO.addFinalizer(service.release)
    } yield service
  }
}

final case class CounterLive(id: Ref[UUID], ref: Ref[Int]) extends Counter {
  def acquire: UIO[Unit] = {
    Random.nextUUID
      .flatMap(n => id.set(n) *> ZIO.debug(s"Acquired counter $n"))
  }

  def increment: UIO[Unit] =
    ref.update(_ + 1)

  def get: UIO[Int] =
    ref.get

  def release: UIO[Unit] =
    id.get.flatMap(id => ZIO.debug(s"Released counter $id"))
}
```

## 1. The `Reloadable` Service

In line with the principles of typical ZIO services, reloadable services are specifically crafted to operate seamlessly within the ZIO environment. The `Reloadable[Service]` data type serves as a wrapper around any reloadable service. This data type encompasses two fundamental methods: `get` and `reload`. The `get` method facilitates the retrieval of the underlying service managed by the `ScopedRef`, while the `reload` method enables the reloading of the service.

### Reloadable Operations

Before diving into further details, let's begin by examining the definition of the `Reloadable` class:

```scala
case class Reloadable[Service](scopedRef: ScopedRef[Service], reload: IO[Any, Unit]) {
  def get: UIO[Service] = scopedRef.get
  def reloadFork: UIO[Unit] = reload.ignoreLogged.forkDaemon.unit
}
```

The `Reloadable` service encapsulates a [scoped reference](../resource/scopedref.md) to a service and provides methods to retrieve the service value (`get`) and trigger service reloading (`reload` and `reloadFork`).

The two fundamental operations of `Reloadable` are as follows:

1. `Reloadable#get` - By calling the `get` method, we can retrieve the underlying service and interact with it directly.

    For example, let's consider that we have acquired the `Reloadable[Counter]` service from the ZIO environment using the `ZIO.service[Reloadable[Counter]]` accessor. We can then use the `get` method to obtain the Counter instance and directly perform operations on it:

```scala

val app: ZIO[Reloadable[Counter], Nothing, Unit] =
  for {
    reloadable <- ZIO.service[Reloadable[Counter]]
    counter    <- reloadable.get 
  } yield ()
```

2. **`Reloadable#reload`**— This operation involves acquiring a new service and releasing the old one, thereby enabling the reloading of the service:

```scala

val app: ZIO[Reloadable[Counter], Any, Unit] =
  for {
    reloadable <- ZIO.service[Reloadable[Counter]]
    counter    <- reloadable.get
    _          <- counter.increment
    _          <- counter.increment
    _          <- counter.increment
    _          <- counter.get.debug("Counter value is")
    _          <- reloadable.reload
    counter    <- reloadable.get
    _          <- counter.increment
    _          <- counter.increment
    _          <- counter.get.debug("Counter value is")
  } yield ()
```

### Creating Reloadable Services

Up to this point, we have explored the process of acquiring reloadable services from the ZIO environment and interacting with them. However, these workflows cannot be executed without fulfilling their requirements. For instance, in the previous example, the type of our workflow is `ZIO[Reloadable[Counter], Any, Unit]`. This implies that we need to provide a layer of type `Reloadable[Counter]`. It is necessary to create reloadable services and `provide` them as a `ZLayer`. In this section, we will delve into the creation of such services.

First, let's explore the definition of the two primary constructors for Reloadable:

```scala
object Reloadable {
  def manual[In, E, Out](
      layer: ZLayer[In, E, Out]
    ): ZLayer[In, E, Reloadable[Out]] = ???
    
  def auto[In, E, Out](
      layer: ZLayer[In, E, Out],
      schedule: Schedule[In, Any, Any]
    ): ZLayer[In, E, Reloadable[Out]] = ???
}
```

There are two fundamental approaches to creating reloadable services:

1. **`Reloadable.manual`** - Using the manual constructor, we can create a reloadable service and subsequently reload it whenever needed by explicitly invoking the `Reloadable#reload` method. This method accepts a layer of type `ZLayer[In, E, Out]` and returns a layer of type `ZLayer[In, E, Reloadable[Out]]`.

Continuing the previous example, assume we have written the `Counter.live` as follows:

```scala

object Counter {
  val live: ZLayer[Any, Nothing, Counter] = ZLayer.scoped {
    for {
      id <- Ref.make(UUID.randomUUID())
      ref <- Ref.make(0)
      service = CounterLive(id, ref)
      _ <- service.acquire
      _ <- ZIO.addFinalizer(service.release)
    } yield service
  }
}
```

We can easily convert that to reloadable layer using `Reloadable.manual`:

```scala
object Counter {
  val live: ZLayer[Any, Nothing, Counter] = ???
  
  val reloadable: ZLayer[Any, Nothing, Reloadable[Counter]] = 
    Reloadable.manual(live)
}
```

Alternatively, we can directly utilize the `ZLayer#reloadableManual` method:

```scala
object Counter {
  val live: ZLayer[Any, Nothing, Counter] = ???

  val reloadable: ZLayer[Any, Nothing, Reloadable[Counter]] =
    live.reloadableManual
}
```

Now we can `provide` the `Counter.reloadable` layer to the app workflow and execute the application:

```scala

object ReloadableExample extends ZIOAppDefault {
  val app: ZIO[Reloadable[Counter], Any, Unit] =
    for {
      reloadable <- ZIO.service[Reloadable[Counter]]
      counter <- reloadable.get
      _ <- counter.increment
      _ <- counter.increment
      _ <- counter.increment
      _ <- counter.get.debug("Counter value is")
      _ <- reloadable.reload *> ZIO.sleep(1.second)
      counter <- reloadable.get
      _ <- counter.increment
      _ <- counter.increment
      _ <- counter.get.debug("Counter value is")
    } yield ()

  def run = app.provide(Counter.reloadable)
}
```

This program defines an application that operates on a reloadable counter service (`Reloadable[Counter]`). It obtains the `Counter` service from the reloadable service, performs three `increment` operations, displays its current value, proceeds to `reload` the counter service, performs two additional increment operations, and finally displays its value once again. Therefore, the expected output would be as follows:

```bash
Acquired counter d04519a3-7332-43ca-bc86-f61fbaf2e3d6
Counter value: 3
Released counter d04519a3-7332-43ca-bc86-f61fbaf2e3d6
Acquired counter bc66ba00-0b50-4e6e-9f60-c38b6e140a82
Counter value: 2
Released counter bc66ba00-0b50-4e6e-9f60-c38b6e140a82
```

Observing the behavior, we notice that the service undergoes reloading, causing the counter to reset and begin incrementing from zero once more.

2. **`Reloadable.auto`** - By utilizing this constructor, we can provide a schedule alongside a layer of type `ZLayer[In, E, Out]`, resulting in a layer of reloadable service that will be automatically reloaded based on the specified schedule. Additionally, there is another constructor called `Reloadable.autoFromConfig` which can be used to extract the schedule from the ZIO environment.

Let's change the previous example to reload the Counter service automatically every 5 second. First we need to create auto reloadable service:

```scala

object Counter {
  val live: ZLayer[Any, Nothing, Counter] = ???
  
  val autoReloadable: ZLayer[Any, Nothing, Reloadable[Counter]] =
    Reloadable.auto(live, Schedule.fixed(5.seconds))
}
```

Or we can use `ZLayer#reloadableAuto` to convert a layer to auto reloadable service:

```scala

object Counter {
  val live: ZLayer[Any, Nothing, Counter] = ???
  
  val autoReloadable: ZLayer[Any, Nothing, Reloadable[Counter]] =
    live.reloadableAuto(Schedule.fixed(5.seconds))
}
```

Finally, we don't require to manually execute `Reloadable#reload` and the service will be reloaded every 5 second:

```scala

object AutoReloadableExample extends ZIOAppDefault {
  val app: ZIO[Reloadable[Counter], Any, Unit] =
    for {
      reloadable <- ZIO.service[Reloadable[Counter]]
      counter <- reloadable.get
      _ <- counter.increment
      _ <- counter.increment
      _ <- counter.increment
      _ <- counter.get.debug("Counter value is")
      _ <- ZIO.sleep(6.second)
      counter <- reloadable.get // getting reloadable service from environment
      _ <- counter.increment
      _ <- counter.increment
      _ <- counter.get.debug("Counter value is")
    } yield ()

  def run = app.provide(Counter.autoReloadable)
}
```

## 2. The `ServiceReloader` Service

Please note that in the previous example, there was no need for manual service reloading. However, we still had to manually retrieve the reloadable service from the environment using `ZIO.service[Reloadable[Counter]]` and then access the `Counter` service from the reloadable counter service. This approach involves some boilerplate code and indirection. We aim to adopt an approach that eliminates the necessity of `Reloadable[Service]` from the environment and instead directly requires the `Service` from the environment.

To address this issue, the `ServiceReloader` has been developed as a solution. It is part of the `zio-macros` package. To utilize it, we need to include the following line in our `build.sbt` file:

```scala
libraryDependencies ++= Seq("dev.zio" %% "zio-macros" % "<version>")
```

The `ServiceReloader` serves as a registry for services, enabling dynamic reloading of services. By applying the `reloadable` operator on `ZLayer`, we can create a reloadable version of our service. Subsequently, we can simply invoke `ServiceLoader.reload` to initiate the reloading of the service. Let's explore the definition of the ServiceLoader trait:

```scala
trait ServiceReloader {
  def register[A: Tag: IsReloadable](serviceLayer: ZLayer[Any, Any, A]): IO[ServiceReloader.Error, A]
  def reload[A: Tag]: IO[ServiceReloader.Error, Unit]
}
```

For example, if we require a reloadable `Counter` service, we can simply invoke `ServiceReloader.reload[Counter]`. The resulting type will be `ZIO[ServiceReloader, ServiceReloader.Error, Unit]`. Consequently, instead of providing a ZLayer of type `Reloadable[Counter]`, we now need to provide a layer of type `ServiceReloader`. 

With this approach, there is no longer a need to retrieve `Reloadable[Counter]` from the ZIO environment, eliminating the requirement to access `Counter` from an instance of the `Reloadable[Counter]` class. Instead, we can work with services in a manner consistent with the idiomatic approach used for regular services. Thus, rather than calling `ZIO.serviceWithZIO[Reloadable[Counter]](_.get)`, we can conveniently use `ZIO.service[Counter]` to obtain the `Counter` service directly from the ZIO environment.

Let's see how we can rewrite the `Reloadable.manual` example with this approach:

```scala

object ServiceReloaderExample extends ZIOAppDefault {

  def app: ZIO[Counter with ServiceReloader, ServiceReloader.Error, Unit] =
    for {
      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.get.debug("Counter value")

      _ <- ServiceReloader.reload[Counter]
      _ <- ZIO.sleep(1.seconds)

      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.get.debug("Counter value")
    } yield ()

  def run = app.provide(Counter.reloadable, ServiceReloader.live)
}
```

To create a reloadable layer, we need to import `zio.macros._`. Subsequently, by invoking the `ZLayer#reloadable` method, we can transform the `live` layer into a layer that depends on `ServiceReloader` and provides `Counter` services:

```scala

object Counter {

  val live: ZLayer[Any, Nothing, Counter] = ???

  val reloadable: ZLayer[ServiceReloader, ServiceReloader.Error, Counter] =
    live.reloadable
}
```

We can further enhance this application by decoupling the reload process from the application logic. In doing so, each time the service is reloaded, subsequent calls to the service will be served with the freshly reloaded services:

```scala

object ServiceReloaderParallelWorkflowExample extends ZIOAppDefault {
  def reloadWorkflow =
    ServiceReloader.reload[Counter].delay(5.seconds)

  def app: ZIO[Counter with ServiceReloader, ServiceReloader.Error, Unit] =
    for {
      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.get.debug("Counter value")

      _ <- ZIO.sleep(6.seconds)

      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.increment
      _ <- Counter.get.debug("Counter value")
    } yield ()

  def run = (app <&> reloadWorkflow).provide(Counter.reloadable, ServiceReloader.live)
}
```

## Conclusion

Int this article we introduced two methods for implementing reloadable services in ZIO. The first method involves using the `Reloadable` service, which requires some boilerplate code. With this approach, services can be manually reloaded using the reload method. The second method, introduced by `zio-macros`, simplifies the process by utilizing the `ServiceReloader` service. This approach eliminates the need for retrieving the reloadable service from the environment and allows direct access to the service.

Overall, reloadable services in ZIO offer a powerful tool for managing services that require reloading, enabling seamless integration within the ZIO environment and simplifying service management in complex applications.

All the source code associated with this article is available on the [ZIO Quickstart](https://github.com/zio/zio-quickstarts/tree/master/zio-quickstart-reloadable-services) on Github.

---

## The Four Elements of Service Pattern

Writing services in ZIO using the _Service Pattern_ is very similar to the object-oriented way of defining services. We use scala traits to define services, classes to implement services, and constructors to define service dependencies. Finally, we lift the class constructor into the `ZLayer`.

Let's start learning this service pattern by writing a `DocRepo` service:

## 1. Service Definition

Traits are how we define services. A service could be all the stuff that is related to one concept with singular responsibility. We define the service definition with a trait named `DocRepo`:

```scala

final case class Doc(
  title: String,
  description: String,
  language: String,
  format: String,
  content: Array[Byte]
)

trait DocRepo {
  def get(id: String): ZIO[Any, Throwable, Doc]

  def save(document: Doc): ZIO[Any, Throwable, String]

  def delete(id: String): ZIO[Any, Throwable, Unit]

  def findByTitle(title: String): ZIO[Any, Throwable, List[Doc]]
}
```

## 2. Service Implementation

It is the same as what we did in an object-oriented fashion. We implement the service with the Scala class:

```scala
final class DocRepoLive() extends DocRepo {
  override def get(id: String): ZIO[Any, Throwable, Doc] = ???

  override def save(document: Doc): ZIO[Any, Throwable, String] = ???

  override def delete(id: String): ZIO[Any, Throwable, Unit] = ???

  override def findByTitle(title: String): ZIO[Any, Throwable, List[Doc]] = ???
}
```

## 3. Service Dependencies

We might need `MetadataRepo` and `BlobStorage` services to implement the `DocRepo` service. Here, we put its dependencies into its constructor. All the dependencies are just interfaces, not implementation. Just like what we did in object-oriented style.

First, we need to define the interfaces for `MetadataRepo` and `BlobStorage` services:

```scala
final case class Metadata(
  title: String,
  description: String,
  language: String,
  format: String
)

trait MetadataRepo {
  def get(id: String): ZIO[Any, Throwable, Metadata]

  def put(id: String, metadata: Metadata): ZIO[Any, Throwable, Unit]

  def delete(id: String): ZIO[Any, Throwable, Unit]

  def findByTitle(title: String): ZIO[Any, Throwable, Map[String, Metadata]]
}

trait BlobStorage {
  def get(id: String): ZIO[Any, Throwable, Array[Byte]]

  def put(content: Array[Byte]): ZIO[Any, Throwable, String]

  def delete(id: String): ZIO[Any, Throwable, Unit]
}
```

Now, we can implement the `DocRepo` service:

```scala
final class DocRepoLive(
  metadataRepo: MetadataRepo,
  blobStorage: BlobStorage
) extends DocRepo {
  override def get(id: String): ZIO[Any, Throwable, Doc] =
    (metadataRepo.get(id) <&> blobStorage.get(id)).map {
      case (metadata, content) =>
        Doc(
          title = metadata.title,
          description = metadata.description,
          language = metadata.language,
          format = metadata.format,
          content = content
        )
    }
    
  override def save(document: Doc): ZIO[Any, Throwable, String] =
    for {
      id       <- blobStorage.put(document.content)
      metadata = Metadata(
        title = document.title,
        description = document.description,
        language = document.language,
        format = document.format
      )
      _        <- metadataRepo.put(id, metadata)
    } yield id

  override def delete(id: String): ZIO[Any, Throwable, Unit] = blobStorage.delete(id) &> metadataRepo.delete(id).unit

  override def findByTitle(title: String): ZIO[Any, Throwable, List[Doc]] =
    for {
      metadatas <- metadataRepo.findByTitle(title)
      content   <- ZIO.foreachPar(metadatas) { (id, metadata) =>
                     blobStorage
                       .get(id)
                       .map { content =>
                         val doc = Doc(
                           title = metadata.title,
                           description = metadata.description,
                           language = metadata.language,
                           format = metadata.format,
                           content = content
                         )
                    
                         id -> doc
                       }
                   }
    } yield content.values.toList
}
```

## 4. ZLayer (Constructor)

Now, we create a companion object for `DocRepoLive` data type and lift the service implementation into the `ZLayer`:

```scala
object DocRepo {
  /**
   * The "live" implementation of the `DocRepo` service.
   */
  val live: ZLayer[BlobStorage & MetadataRepo, Nothing, DocRepo] =
    ZLayer {
      for {
        metadataRepo <- ZIO.service[MetadataRepo]
        blobStorage  <- ZIO.service[BlobStorage]
      } yield new DocRepoLive(metadataRepo, blobStorage)
    }
}
```

And voila! We have implemented the `DocRepo` service using the _Service Pattern_.

## Assembling the application

Similarly, we need to implement the `BlobStorage` and `MetadataRepo` services:

```scala
object InmemoryBlobStorage {
  /**
   * An in-memory implementation of the `BlobStorage` service.
   */
  val layer = 
    ZLayer {
      ???
    } 
}

object InmemoryMetadataRepo {
  /**
   * An in-memory implementation of the `MetadataRepo` service.
   */
  val layer = 
    ZLayer {
      ???
    }
}
```

This is how ZIO services are created. Let's use the `DocRepo` service in our application. We should provide `DocRepo` layer to be able to run the application:

```scala

object MainApp extends ZIOAppDefault {
  val app =
    for {
      docRepo <- ZIO.service[DocRepo]
      id      <- docRepo.save(
                    Doc(
                      "title",
                      "description",
                      "en",
                      "text/plain",
                      "content".getBytes()
                    )
                 )
      doc     <- docRepo.get(id)
      _       <- Console.printLine(
                   s"""
                     |Downloaded the document with $id id:
                     |  title: ${doc.title}
                     |  description: ${doc.description}
                     |  language: ${doc.language}
                     |  format: ${doc.format}
                     |""".stripMargin
                 )  
      _       <- docRepo.delete(id)
      _       <- Console.printLine(s"Deleted the document with $id id")
    } yield ()

  def run =
    app.provide(
      DocRepo.live,
      InmemoryBlobStorage.layer,
      InmemoryMetadataRepo.layer
    )
}
```

During writing the application, we don't care which implementation version of the `BlobStorage` and `MetadataRepo` services will be injected into our `app`. Later at the end of the day, it will be provided by one of `ZIO#provide*` methods.

That's it! Very simple! ZIO encourages us to follow some of the best practices in object-oriented programming. So it doesn't require us to throw away all our object-oriented knowledge.

---

## The Three Laws of ZIO Environment

When we are working with the ZIO environment, one question might arise: "When should we use environment and when do we need to use constructors?".

Using ZIO environment follows three laws:

## 1. Service Interface (Trait)

**When we are defining service interfaces we should _never_ use the environment for dependencies of the service itself.**

For example, if the implementation of service `X` depends on service `Y` and `Z` then these should never be reflected in the trait that defines service `X`. It's leaking implementation details.

So the following service definition is wrong because the `BlobStorage` and `MetadataRepo` services are dependencies of the  `DocRepo` service's implementation, not the `DocRepo` interface itself:

```scala

trait DocRepo {
  def save(document: Doc): ZIO[BlobStorage & MetadataRepo, Throwable, String]
}
```

## 2. Service Implementation (Class)

**When implementing service interfaces, we should accept all dependencies in the class constructor.**

Again, let's see how `DocRepoImpl` accepts `BlobStorage` and `MetadataRepo` dependencies from the class constructor:

```scala
case class DocRepoImpl(
    metadataRepo: MetadataRepo,
    blobStorage: BlobStorage
) extends DocRepo {
  override def delete(id: String): ZIO[Any, Throwable, Unit] =
    for {
      _ <- blobStorage.delete(id)
      _ <- metadataRepo.delete(id)
    } yield ()

  override def get(id: String): ZIO[Any, Throwable, Doc] = ???

  override def save(document: Doc): ZIO[Any, Throwable, String] = ???

  override def findByTitle(title: String): ZIO[Any, Throwable, List[Doc]] = ???
}

object DocRepoImpl {
  val layer: ZLayer[BlobStorage with MetadataRepo, Nothing, DocRepo] =
    ZLayer {
      for {
        metadataRepo <- ZIO.service[MetadataRepo]
        blobStorage  <- ZIO.service[BlobStorage]
      } yield DocRepoImpl(metadataRepo, blobStorage)
    }
}
```

So keep in mind, we can't do something like this:

```scala
case class DocRepoImpl() extends DocRepo {
  override def delete(id: String): ZIO[Any, Throwable, Unit] =
    for {
      blobStorage  <- ZIO.service[BlobStorage]
      metadataRepo <- ZIO.service[MetadataRepo]
      _            <- blobStorage.delete(id)
      _            <- metadataRepo.delete(id)
    } yield ()

  override def get(id: String): ZIO[Any, Throwable, Doc] = ???

  override def save(document: Doc): ZIO[Any, Throwable, String] = ???

  override def findByTitle(title: String): ZIO[Any, Throwable, List[Doc]] = ???
}
```

## 3. Business Logic

**Finally, in the business logic we should use the ZIO environment to consume services.**

Therefore, in the last example, if we inline all accessor methods whenever we are using services, we are using the ZIO environment:

```scala

object MainApp extends ZIOAppDefault {
  val app =
    for {
      id <-
        ZIO.serviceWithZIO[DocRepo](_.save(
          Doc(
              "How to write a ZIO application?",
              "In this tutorial we will learn how to write a ZIO application.",
              "en",
              "text/plain",
              "content".getBytes()
            )
          )
        )
      doc <- ZIO.serviceWithZIO[DocRepo](_.get(id))
      _ <- Console.printLine(
        s"""
          |Downloaded the document with $id id:
          |  title: ${doc.title}
          |  description: ${doc.description}
          |  language: ${doc.language}
          |  format: ${doc.format}
          |""".stripMargin
      )
      _ <- ZIO.serviceWithZIO[DocRepo](_.delete(id))
      _ <- Console.printLine(s"Deleted the document with $id id")
    } yield ()

  def run =
    app.provide(
      DocRepoImpl.layer,
      InmemoryBlobStorage.layer,
      InmemoryMetadataRepo.layer
    )
}
```

That's it! These are the most important rules we need to know about the ZIO environment.

----------------

:::info
The remaining part of this section can be skipped if you are not an advanced ZIO user.
:::

Now let's elaborate more on the first rule. On rare occasions, all of which involve local context that is independent of implementation, it's _acceptable_ to use the environment in the definition of a service.

Here are two examples:

1. In a web application, a service may be defined only to operate in the context of an HTTP request. In such a case, the request itself could be stored in the environment: `ZIO[HttpRequest, ...]`. This is acceptable because this use of the environment is part of the semantics of the trait itself, rather than leaking an implementation detail of some particular class that implements the service trait:

```scala

type HttpApp = ZIO[HttpRequest, Throwable, HttpResponse]
type HttpRoute = Map[String, HttpApp]

case class HttpRequest(method: Int,
                       uri: URI,
                       headers: Map[String, String],
                       body: UStream[Byte])

case class HttpResponse(status: Int,
                        headers: Map[String, String],
                        body: UStream[Byte])

object HttpResponse {
  def apply(status: Int, message: String): HttpResponse =
    HttpResponse(
      status = status,
      headers = Map.empty,
      body = ZStream.fromChunk(
        Chunk.fromArray(message.getBytes(StandardCharsets.UTF_8))
      )
    )

  def ok(msg: String): HttpResponse = HttpResponse(200, msg)

  def error(msg: String): HttpResponse = HttpResponse(500, msg)
}

trait HttpServer {
  def serve(map: HttpRoute, host: String, port: Int): ZIO[Any, Throwable, Unit]
}

object HttpServer {
  def serve(map: HttpRoute, host: String, port: Int): ZIO[HttpServer, Throwable, Unit] =
    ZIO.serviceWithZIO(_.serve(map, host, port))
}

case class HttpServerLive() extends HttpServer {
  override def serve(map: HttpRoute, host: String, port: Int): ZIO[Any, Throwable, Unit] = ???
}

object HttpServerLive {
  val layer: URLayer[Any, HttpServer] = ZLayer.succeed(HttpServerLive())
}

object MainWebApp extends ZIOAppDefault {

  val myApp: ZIO[HttpServer, Throwable, Unit] = for {
    _ <- ZIO.unit
    healthcheck: HttpApp = ZIO.service[HttpRequest].map { _ =>
      HttpResponse.ok("up")
    }

    pingpong = ZIO.service[HttpRequest].flatMap { req =>
      ZIO.ifZIO(
        req.body.via(ZPipeline.utf8Decode).runHead.map(_.contains("ping"))
      )(
        onTrue = ZIO.attempt(HttpResponse.ok("pong")),
        onFalse = ZIO.attempt(HttpResponse.error("bad request"))
      )
    }

    map = Map(
      "/healthcheck" -> healthcheck,
      "/pingpong" -> pingpong
    )
    _ <- HttpServer.serve(map, "localhost", 8080)
  } yield ()

  def run = myApp.provideLayer(HttpServerLive.layer)

}
```

2. In a database application, a service may be defined only to operate in the context of a larger database transaction. In such a case, the transaction could be stored in the environment: `ZIO[DatabaseTransaction, ...]`. As in the previous example, because this is part of the semantics of the trait itself (whose functionality all operates within a transaction), this is not leaking implementation details, and therefore it is valid:

```scala
trait DatabaseTransaction {
  def get(key: String): Task[Int]
  def put(key: String, value: Int): Task[Unit]
}

object DatabaseTransaction {
  def get(key: String): ZIO[DatabaseTransaction, Throwable, Int] =
    ZIO.serviceWithZIO(_.get(key))

  def put(key: String, value: Int): ZIO[DatabaseTransaction, Throwable, Unit] =
    ZIO.serviceWithZIO(_.put(key, value))
}

trait Database {
  def atomically[E, A](zio: ZIO[DatabaseTransaction, E, A]): ZIO[Any, E, A]
}

object Database {
  def atomically[E, A](zio: ZIO[DatabaseTransaction, E, A]): ZIO[Database, E, A] =
    ZIO.serviceWithZIO(_.atomically(zio))
}

case class DatabaseLive() extends Database {
  override def atomically[E, A](zio: ZIO[DatabaseTransaction, E, A]): ZIO[Any, E, A] = ???
}

object DatabaseLive {
  val layer = ZLayer.succeed(DatabaseLive())
}

object MainDatabaseApp extends ZIOAppDefault {
  val myApp: ZIO[Database, Throwable, Unit] =
    for {
      _ <- Database.atomically(DatabaseTransaction.put("counter", 0))
      _ <- ZIO.foreachPar(List(1 to 10)) { _ =>
        Database.atomically(
          for {
            value <- DatabaseTransaction.get("counter")
            _ <- DatabaseTransaction.put("counter", value + 1)
          } yield ()
        )
      }
    } yield ()

  def run = myApp.provideLayer(DatabaseLive.layer)

}
```

So while it's better to err on the side of "don't put things into the environment of service interface", there are cases where it's acceptable.

---

## Clock

Clock service contains some functionality related to time and scheduling. 

To get the current time in a specific time unit, the `currentTime` function takes a unit as `TimeUnit` and returns `UIO[Long]`:

```scala compile-only

val inMilliseconds: UIO[Long] = Clock.currentTime(TimeUnit.MILLISECONDS)
val inDays        : UIO[Long] = Clock.currentTime(TimeUnit.DAYS)
```

To get current date time in the current timezone the `currentDateTime` function returns a ZIO effect containing `OffsetDateTime`.

Also, the Clock service has a very useful functionality for sleeping and creating a delay between jobs. The `sleep` takes a `Duration` and sleeps for the specified duration. It is analogous to `java.lang.Thread.sleep` function, but it doesn't block any underlying thread. It's completely non-blocking.

In the following example we are going to print the current time periodically by placing a one second `sleep` between each print call:

```scala compile-only

def printTimeForever: ZIO[Any, Throwable, Nothing] =
  Clock.currentDateTime.flatMap(Console.printLine(_)) *>
    ZIO.sleep(1.seconds) *> printTimeForever
```

For scheduling purposes like retry and repeats, ZIO has a great data type called [Schedule](../schedule/index.md).

---

## Console

The Console service contains simple I/O operations for reading/writing strings from/to the standard input, output, and error console.

| Function         | Input Type        | Output Type                     |
|------------------|-------------------|---------------------------------|
| `print`          | `line: => String` | `ZIO[Any, IOException, Unit]`   |
| `printError`     | `line: => String` | `ZIO[Any, IOException, Unit]`   |
| `printLine`      | `line: => String` | `ZIO[Any, IOException, Unit]`   |
| `printLineError` | `line: => String` | `ZIO[Any, IOException, Unit]`   |
| `readLine`       |                   | `ZIO[Any, IOException, String]` |

All functions of the Console service are effectful, this means they are just descriptions of reading/writing from/to the console. 

As ZIO data type supports monadic operations, we can compose these functions with for-comprehension which helps us to write our program pretty much like an imperative program:

```scala

object MyHelloApp extends ZIOAppDefault {
  val program: ZIO[Any, IOException, Unit] = for {
    _    <- printLine("Hello, what is you name?")
    name <- readLine
    _    <- printLine(s"Hello $name, welcome to ZIO!")
  } yield ()

  def run = program
}
```

Note again, every line of our `program` are descriptions, not statements. As we can see the type of our `program` is `ZIO[Any, IOException, Unit]`, it means to run `program` we do not need any environment, it may fail due to failure of `readLine` and it will produce `Unit` value.

---

## Introduction to ZIO's Built-in Services

ZIO already provides four built-in services:

1. **[Console](console.md)** — Operations for reading/writing strings from/to the standard input, output, and error console.
2. **[Clock](clock.md)** — Contains some functionality related to time and scheduling.
3. **[Random](random.md)** — Provides utilities to generate random numbers.
4. **[System](system.md)** — Contains several useful functions related to system environments and properties.

When we use these services we don't need to provide their corresponding environment explicitly. ZIO provides built-in live version of ZIO services to our effects, so we do not need to provide them manually.

```scala

object MainApp extends ZIOAppDefault {
  val myApp: ZIO[Any, IOException, Unit] = 
    for {
      date <- Clock.currentDateTime
      _    <- ZIO.logInfo(s"Application started at $date")
      _    <- Console.print("Enter your name: ")
      name <- Console.readLine
      _    <- Console.printLine(s"Hello, $name!")
    } yield ()

  def run = myApp
}
```

---

## Random

Random service provides utilities to generate random numbers. It's a functional wrapper of `scala.util.Random`. This service contains various different pseudo-random generators like `nextInt`, `nextBoolean` and `nextDouble`. Each random number generator functions return a `URIO[Random, T]` value.

```scala

for {
  randomInt    <- Random.nextInt
  _            <- Console.printLine(s"A random Int: $randomInt")
  randomChar   <- Random.nextPrintableChar
  _            <- Console.printLine(s"A random Char: $randomChar")
  randomDouble <- Random.nextDoubleBetween(1.0, 5.0)
  _            <- Console.printLine(s"A random double between 1.0 and 5.0: $randomDouble")
} yield ()
```

Random service has a `setSeed` which helps us to alter the state of the random generator. It is useful for setting up a test version of Random service when we need to reproduce always the same sequence of numbers.

```scala

for {
  _        <- Random.setSeed(0)
  nextInts <- (Random.nextInt zip Random.nextInt)
} yield assert(nextInts == (-1155484576,-723955400))
```

Also, it has a utility to shuffle a list and to generate random samples from Gaussian distribution:

* **shuffle** - Takes a list as an input and shuffles it.
* **nextGaussian** — Returns the next pseudorandom, Gaussian ("normally") distributed double value with mean 0.0 and standard deviation 1.0.

> **Note**:
>
> Random numbers that are generated via Random service are not cryptographically strong. Therefore it's not safe to use the ZIO Random service for security domains where a high level of security and randomness is required, such as password generation.

---

## System

System service contains several useful functions related to system environments and properties. Both of **system environments** and **system properties** are key/value pairs. They are used to pass user-defined information to our application.

Environment variables are global operating system level variables available to all applications running on the same machine, while properties are application-level variables provided to our application.

## System Environment
The `env` function retrieves the value of an environment variable:

```scala

for {
  user <- System.env("USER")
  _    <- user match {
            case Some(value) => 
              Console.printLine(s"The USER env is: $value")
            case None        => 
              Console.printLine("Oops! The USER env is not set")
          }
} yield ()
```

## System Property
Also, the System service has a `property` function to retrieve the value of a system property:

```scala

for {
  user <- System.property("LOG_LEVEL")
  _    <- user match {
           case Some(value) => 
             Console.printLine(s"The LOG_LEVEL property is: $value")
           case None => 
             Console.printLine("Oops! The LOG_LEVEL property is not set")
         }
} yield ()
```

## Miscellaneous

With the `lineSeparator` method, we can determine the line separator for the underlying platform:

```scala
System.lineSeparator
// res2: String = """
// """
```

---

## Fiber-local State

Both the `FiberRef` and `ZState` data types are state management tools that are scoped to a certain fiber. Their values are only accessible within the fiber that runs them.

We have a separate page for the [`FiberRef`](fiberref.md) and [`ZState`](zstate.md) data types which explain how to use them.

---

## FiberRef: Introduction to Fiber-local Storage

`FiberRef` is a data structure for managing and accessing thread-local values within a ZIO fiber. Thread-local storage (TLS) is a mechanism that provides each fiber its own separate storage space. A `FiberRef[A]` is a specialized type of mutable reference (`Ref[A]`) that allows us to store and retrieve values of type `A` that are local to a specific fiber.  

The `FiberRef` data structure allows us to perform operations such as reading the current value, updating the value, or modifying the value atomically within a fiber. It ensures thread-safety and isolation between different fibers, allowing them to have their own independent values for the `FiberRef`. Each fiber maintains its own copy of the fiber-specific variable, and modifications to the variable made by one fiber do not affect the values seen by other fibers.

By using `FiberRef`, we can maintain per-fiber context or state information, which can be useful in various scenarios such as managing resources, tracking application-specific information, or carrying contextual data throughout the execution of a fiber.

We can think of `FiberRef` as Java's `ThreadLocal` on steroids. So, just like we have `ThreadLocal` in Java we have `FiberRef` in ZIO. So as different threads have different `ThreadLocal`s, we can say different fibers have different `FiberRef`s. They don't intersect or overlap in any way. `FiberRef` is the fiber version of `ThreadLocal` with significant improvements in its semantics. A `ThreadLocal` only has a mutable state in which each thread accesses its own copy, but threads don't propagate their state to their children's.

As opposed to `Ref[A]`, the value of a `FiberRef[A]` is bound to an executing fiber. Different fibers who hold the same `FiberRef[A]` can independently set and retrieve values of the reference, without collisions.

```scala

for {
  fiberRef <- FiberRef.make[Int](0)
  _        <- fiberRef.set(10)
  v        <- fiberRef.get
} yield v == 10
```

## Motivation

Whenever we have some kind of scoped information or context, and we don't want to use the ZIO environment to store it, we can use `FiberRef` to store it.

To illustrate this, let's try to find a solution to the _Structured Logging_ problem. In structured logging, we tend to attach contextual information to log messages, such as user id, correlation id, log level, and so on.

So assume we have written the following code:

```scala

for {
  _ <- Logging.log("Hello World!")
  _ <- ZIO.foreachParDiscard(List("Jane", "John")) { name =>
    Logging.logAnnotate("name", name) {
      for {
        _ <- Logging.log(s"Received request")
        fiberId <- ZIO.fiberId.map(_.ids.head)
        _ <- Logging.logAnnotate("fiber_id", s"$fiberId")(
          Logging.log("Processing request")
        )
        _ <- Logging.log("Finished processing request")
      } yield ()
    }
  }
  _ <- Logging.log("All requests processed")
} yield ()
```

We would like to see the following log output:

```scala
Hello World!
[name=Jane] Received request
[name=John] Received request
[name=Jane] [fiber_id=7] Processing request
[name=John] [fiber_id=8] Processing request
[name=John] Finished processing request
[name=Jane] Finished processing request
All requests processed
```

In the above code, we have two users, `Jane` and `John`, and we want to handle some operations on each user, concurrently. When we perform concurrent operations, we would like to have a way to associate each concurrent operation with its corresponding user and fiber id. So when we log messages, we have all the information available for a specific event.

In order to do this, we need a context-aware logging service. This logging service needs to have a **state** that is a place to store annotations. This state can be accessed and modified **concurrently** by multiple fibers. And the important part is that each fiber should have its own isolated copy of the state, so when a fiber modifies the state, it doesn't clobber the state of other fibers.

Until now, we can categorize our requirements into two parts:
- We need a mechanism to carry some contextual information, without explicitly passing it around.
- We need a mechanism to update the state in an isolated fashion, where each fiber can update the state without affecting the state of other fibers.

## Solution

In this section, we will look at two solutions to the problem of structured logging we have mentioned above. The first solution has some limitations and drawbacks, so we will choose the second solution as the final solution.

### Solution 1: ZIO Environment

One solution is to use the ZIO environment to store the state. It addresses the first requirement, very well. ZIO environment is a nice place to store the contextual states. And to make the state isolated between fibers, we can reintroduce the new state to the environment instead of updating the environment globally:

```scala
// Solution 1: Using the ZIO environment to store the contextual state

object Logging {
  type Annotation = Map[String, String]

  def logAnnotate[R, E, A](key: String, value: String)(
    zio: ZIO[R with Annotation, E, A]
  ): ZIO[R with Annotation, E, A] = {
    for {
      s <- ZIO.service[Annotation]
      r <- zio.provideSomeLayer[R](ZLayer.succeed(s.updated(key, value)))
    } yield (r)
  }

  def log(message: String): ZIO[Annotation, Nothing, Unit] = {
    ZIO.service[Annotation].flatMap {
      case annotation if annotation.isEmpty => 
        Console.printLine(message).orDie
      case annotation =>
        val line =
          s"${annotation.map { case (k, v) => s"[$k=$v]" }.mkString(" ")} $message"
        Console.printLine(line).orDie
    }
  }
}
```

The ZIO environment solution provides an explicit method for ensuring type-safety when dealing with contextual data types. However, this increased type-safety may limit flexibility in certain scenarios. For instance, consider a situation where our workflows require multiple cross-cutting services such as `Logging`, `Config`, and `Metrics`. In this case, every instance of application logic would involve a workflow with a type signature like `ZIO[Logging & Config & Metrics & ..., IOException, Any]`. This extensive type declaration restricts easy code refactoring and maintenance, while also distracts our attention from the core business logic. Any modification to the contextual data type necessitates modifying the entire program. 

Although we successfully utilized the ZIO environment in the previous example to store the state, it is not considered the idiomatic solution. It is preferable to avoid explicitly exposing the type of the state, in this case `Annotation`, within the environment.

Anyway, despite these limitations, this solution proves especially beneficial when
  - When the contextual service holds a **crucial role** in the workflow logic
  - When there is a requirement for ensuring *type-safety on the service type** within the ZIO environment
  - When there is no sensible **default value** for such services

### Solution 2: FiberRef

The other solution is to use `FiberRef`. FiberRef is a nice way to store the contextual states and make them isolated. Any state maintained by a `FiberRef` will be isolated between fibers. Also, a nice thing about `FiberRef` is that we do not require to place the state in the environment.

Let's see how to use `FiberRef` to implement the logging service:

```scala
// Solution 2: Using the FiberRef to store the contextual state

trait Logger {
  def logAnnotate[R, E, A](key: String, value: String)(
      zio: ZIO[R, E, A]
  ): ZIO[R, E, A]
  def log(message: String): UIO[Unit]
}

object Logging extends Logger {
  def logAnnotate[R, E, A](key: String, value: String)(
      zio: ZIO[R, E, A]
  ): ZIO[R, E, A] = currentAnnotations.locallyWith(_.updated(key, value))(zio)

  def log(message: String): UIO[Unit] = {
    currentAnnotations.get.flatMap {
      case annotation if annotation.isEmpty =>
        Console.printLine(message).orDie
      case annotation =>
        val line =
          s"${annotation.map { case (k, v) => s"[$k=$v]" }.mkString(" ")} $message"
        Console.printLine(line).orDie
    }
  }

  val currentAnnotations: FiberRef[Map[String, String]] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(Map.empty[String, String])
    }

}
```

Now we can write a program that logs some information:

```scala

object FiberRefLoggingExample extends ZIOAppDefault {
  def run =
    for {
      _ <- Logging.log("Hello World!")
      _ <- ZIO.foreachParDiscard(List("Jane", "John")) { name =>
        Logging.logAnnotate("name", name) {
          for {
            _       <- Logging.log(s"Received request")
            fiberId <- ZIO.fiberId.map(_.ids.head)
            _ <- Logging.logAnnotate("fiber_id", s"$fiberId")(
              Logging.log("Processing request")
            )
            _ <- Logging.log("Finished processing request")
          } yield ()
        }
      }
      _ <- Logging.log("All requests processed")
    } yield ()
}
```

The output:

```scala
Hello World!
[name=Jane] Received request
[name=John] Received request
[name=Jane] [fiber_id=5] Processing request
[name=John] [fiber_id=6] Processing request
[name=John] Finished processing request
  [name=Jane] Finished processing request
All requests processed
```

> **Note:**
>
> In the above solution, if we replace the `FiberRef` with `Ref`, the program will not work properly, because the `Ref` is not isolated. The `Ref` will be shared between all fibers, so each fiber clobbers the other fibers' state.

To take it a step further, let's modify the previous example to allow the user to change the underlying logging service:

```scala

trait Logger {
  def logAnnotate[R, E, A](key: String, value: String)(
    zio: ZIO[R, E, A]
  ): ZIO[R, E, A]

  def log(message: String): UIO[Unit]
}
```

```scala

object Logging {

  val defaultLogger: Logger = new Logger {
    def logAnnotate[R, E, A](key: String, value: String)(
      zio: ZIO[R, E, A]
    ): ZIO[R, E, A] = currentAnnotations.locallyWith(_.updated(key, value))(zio)

    def log(message: String): UIO[Unit] = {
      currentAnnotations.get.flatMap {
        case annotation if annotation.isEmpty =>
          Console.printLine(message).orDie
        case annotation =>
          val line =
            s"${annotation.map { case (k, v) => s"[$k=$v]" }.mkString(" ")} $message"
          Console.printLine(line).orDie
      }
    }
  }

  val silentLogger: Logger = new Logger {
    def logAnnotate[R, E, A](key: String, value: String)(
      zio: ZIO[R, E, A]
    ): ZIO[R, E, A] = currentAnnotations.locallyWith(_.updated(key, value))(zio)

    def log(message: String): UIO[Unit] = ZIO.unit
  }

  def log(message: String): ZIO[Any, Nothing, Unit] =
    currentLogger.get.flatMap(_.log(message))

  def logAnnotate[R, E, A](key: String, value: String)(
    zio: ZIO[R, E, A]
  ): ZIO[R, E, A] = currentLogger.get.flatMap(_.logAnnotate(key, value)(zio))

  def locallyWithLogger[R, E, A](newLogger: Logger)(zio: ZIO[R, E, A]) = {
    currentLogger.locallyWith(_ => newLogger)(zio)
  }

  def updateLogger(logger: Logger => Logger): UIO[Unit] = currentLogger.update(logger)

  val currentLogger: FiberRef[Logger] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(defaultLogger)
    }

  val currentAnnotations: FiberRef[Map[String, String]] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(Map.empty[String, String])
    }

}
```

Now, changing the default logger is made easy with the Logging.withLogger function. Let's disable the default logger for a specific section of our example by utilizing Logging.silentLogger:

```scala

object FiberRefChangeDefaultLoggerExample extends ZIOAppDefault {
  def run = for {
    _ <- Logging.log("Hello World!")
    _ <- ZIO.foreachParDiscard(List("Jane", "John")) { name =>
      Logging.locallyWithLogger(Logging.silentLogger) {
        Logging.logAnnotate("name", name) {
          for {
            _ <- Logging.log(s"Received request")
            fiberId <- ZIO.fiberId.map(_.ids.head)
            _ <- Logging.logAnnotate("fiber_id", s"$fiberId")(
              Logging.log("Processing request")
            )
            _ <- Logging.log("Finished processing request")
          } yield ()
        }
      }
    }
    _ <- Logging.log("All requests processed")
  } yield ()
}
```

The output is:

```scala
Hello World!
All requests processed
```

The solution provided by `FiberRef` offers an **implicit** method to store and propagate contextual data or service in an untyped manner. It helps us reduce redundancy in environment types. For instance, by encoding the `Logging` and `Metrics` services using `FiberRef`, we no longer need to include the `Logging` and `Metrics` service type in the environment type of ZIO workflows. As a result, we can simplify a ZIO effect from `ZIO[Logging & Metrics & UserRepo & DocsRepo, IOException, Unit]` to `ZIO[UserRepo & DocsRepo, IOException, Unit]`. This significantly reduces boilerplate code in our workflows, which helps us to focus on maintaining application logic in a flexible manner.

Additionally, as demonstrated in the final example, `FiberRef` proves to be a valuable solution when we have a **default value** for a contextual service or data. We can start the application with default values and, whenever needed, locally or globally change the underlying service or data using `FiberRef#locallyWith` and `FiberRef#update`.

In summary, this solution is particularly advantageous in the following scenarios:

- When **encoding cross-cutting services** without the need to include them everywhere in the ZIO environment.
- When requiring **isolated states** for different fibers.
- When having a **default value** for contextual service or data.

## Use Cases

Whenever we have some kind of scoped information or context, we can think about `FiberRef` as a way to store that information. 

When developing applications, there are several use cases for `FiberRef`. Let's take a look at some of them:

1. **Resource management**: `FiberRef` can be utilized to manage resources that are specific to a particular fiber. For example, we can use it to store and access connections to a database or network resources. Each fiber can have its own dedicated resource, ensuring isolation and avoiding contention between different fibers.

2. **Configuration Settings**: It can be used to store configuration settings that are specific to a fiber. This allows different fibers to have their own configuration values, enabling fine-grained control and customization.

3. **Avoiding Synchronization**: By using `FiberRef`, we can eliminate the need for synchronization mechanisms, such as locks or atomic operations, when accessing fiber-specific data. Each fiber operates on its own private copy, avoiding contention with other fibers.

4. **Distributed Tracing**— In an architecture, where we have highly concurrent workflows and distributed services, there is a need to trace requests as they propagate through the services. To be able to trace requests, we can use `FiberRef` to design the system to automatically propagate request-scoped information.

5. **Contextual Logging**— In lot of cases, logs are not independent piece of information, but they are part of a larger context. So other than just logging the message, we also need to log some additional information such as the request ID, the user ID, the session id, and so on. So whe we collect these logs, we can correlate them based on a common data point. Instead of explicitly passing these contextual information, we can use `FiberRef`.

6. **Execution Scoped Configuration**— When we write applications, we would like to make them configurable. So we configure the application once and used it throughout the whole components. Not all configurations are global. There are certain kinds of configurations that are not global, or at least we have a default value for them globally, but we need to change them dynamically for certain regions. `FiberRef` is a nice tool to model these kind of configurations.

In ZIO we have several use cases for `FiberRef`. Let's discuss some of them:

1. Whenever we use `ZIO.withParallelism`, we can specify the parallelism factor for a region of code. So this information will be stored inside a `FiberRef`, without any need to pass it around all effects explicitly. When we exit the region, the parallelism factor will be restored to the original value:

```scala

object MainApp extends ZIOAppDefault {
  def myJob(name: String) =
    ZIO.foreachParDiscard(1 to 3)(i =>
      ZIO.debug(s"The $name-$i job started") *> ZIO.sleep(2.second)
    )

  def run =
    ZIO.withParallelismUnbounded(
      for {
        _ <- myJob("foo")
        _ <- ZIO.debug("------------------")
        _ <- ZIO.withParallelism(1)(myJob("bar"))
        _ <- ZIO.debug("------------------")
        _ <- myJob("baz")
      } yield ()
    )
}
```

2. Using `ZIOAspect.annotated` we can annotate the effect with some contextual information, e.g. the `correlation_id`. This information will be stored inside a `FiberRef`, which will be propagated to all fibers that are created from the same parent fiber. Each fiber will have its own set of annotations. When we log inside a fiber, the logging service will use the fiber's specific annotations to create the log message:

```scala

object MainApp extends ZIOAppDefault {

  def handleRequest(request: String) =
    for {
      _ <- ZIO.log(s"Received request.")
      _ <- ZIO.unit // do something with the request
      _ <- ZIO.log(s"Finished processing request")
    } yield ()

  def run =
    for {
      _ <- ZIO.log("Hello World!")
      _ <- ZIO.foreachParDiscard(List(("req1", "1"), ("req2", "2"), ("req3", "3"))){ case (req, id) =>
        handleRequest(req) @@ ZIOAspect.annotated("correlation_id", id)
      }
      _ <- ZIO.log("Goodbye!")
    } yield ()

}
```

Here is the output (extra columns were removed for better readability):

```
message="Hello World!"
message="Received request." correlation_id=2
message="Received request." correlation_id=1
message="Received request." correlation_id=3
message="Finished processing request." correlation_id=3
message="Finished processing request." correlation_id=1
message="Finished processing request." correlation_id=2
message="Goodbye!"
```

3. Log levels are also maintained by using `FiberRef`. They are stored inside a `FiberRef`, and whenever we want, we can change the log level using the `ZIO.logLevel` operator:

```scala

for {
  _ <- ZIO.log("Application started!")
  _ <- ZIO.logLevel(LogLevel.Trace) {
    for {
      _ <- ZIO.log("Entering trace log level region")
      _ <- ZIO.log("Doing something")
      _ <- ZIO.log("Leaving trace log level region")
    } yield ()
  }
  _ <- ZIO.log("Application ended!")
} yield ()
```

4. The same goes for when we access the environment (e.g. `ZIO.service`), or when we provide a layer to a ZIO effect (e.g. `ZIO#provide`). ZIO uses `FiberRef` under the hood to store the environment:

```scala

object MainApp extends ZIOAppDefault {
  private val fooLayer = ZLayer.succeed("foo")
  private val barLayer = ZLayer.succeed("bar") 
  
  def run =
    (for {
      _ <- ZIO.service[String].debug("context")
      _ <- ZIO.service[String].debug("context").provide(barLayer)
      _ <- ZIO.service[String].debug("context")
    } yield ()).provide(fooLayer)
}
// Output:
// context: foo
// context: bar
// context: foo
```

There are several other use cases for `FiberRef` in ZIO itself. We just covered some of them to get you some ideas on how they are used in the real world.

## Operations

`FiberRef[A]` has an API almost identical to `Ref[A]`. It includes well-known methods such as:

- `FiberRef#get`. Returns the current value of the reference.
- `FiberRef#set`. Sets the current value of the reference.
- `FiberRef#update` / `FiberRef#updateSome` updates the value with the specified function.
- `FiberRef#modify`/ `FiberRef#modifySome` modifies the value with the specified function, computing a return value for the operation.

You can also use `locally` to scope `FiberRef` value only for a given effect:

```scala

for {
  correlationId <- FiberRef.make[String]("")
  v1            <- correlationId.locally("my-correlation-id")(correlationId.get)
  v2            <- correlationId.get
} yield v1 == "my-correlation-id" && v2 == ""
```

## `Ref` vs. `FiberRef`

Let's explore the distinction between `Ref` and `FiberRef` through two practical examples:

```scala

object RefExample extends ZIOAppDefault {

  def run =
    for {
      ref <- Ref.make(0)
      left = ref.updateAndGet(_ + 1).debug("left1") *>
        ref.updateAndGet(_ + 1).debug("left2")
      right = ref.updateAndGet(_ + 1).debug("right1") *>
        ref.updateAndGet(_ + 3).debug("right2")
      _ <- left <&> right
    } yield ()
}
```

One potential result of running this program is as follows:

```scala
left1: 1
right1: 2
left2: 3
right2: 6
```

It is apparent that the `ref` is shared between the `left` and `right` fibers. However, when using FiberRef, each fiber has its own separate storage, isolating them from one another:

```scala

object FiberRefExample extends ZIOAppDefault {
  def run =
    for {
      ref <- FiberRef.make(0)
      left = ref.updateAndGet(_ + 1).debug("left1") *>
        ref.updateAndGet(_ + 1).debug("left2")
      right = ref.updateAndGet(_ + 1).debug("right1") *>
        ref.updateAndGet(_ + 3).debug("right2")
      _ <- left <&> right
    } yield ()
}
```

One possible output of this program is:

```scala
left1: 1
right1: 1
left2: 2
right2: 4
```

We can observe that each fiber has its own storage without interfering with the value of another fiber.

## Propagation

Let's go back to the `FiberRef`s analog called `ThreadLocal` and see how it works. If we have thread `A` with its `ThreadLocal` and thread `A` creates a new thread, let's call it thread `B`. When thread `A` sends thread `B` the same `ThreadLocal` then what value does thread `B` see inside the `ThreadLocal`? Well, it sees the default value of the `ThreadLocal`. It does not see `A`s value of the `ThreadLocal`. So in other words, `ThreadLocal`s do not propagate their values across the sort of graph of threads so when one thread creates another, the `ThreadLocal` value is not propagated from parent to child.

`FiberRef`s improve on that model quite dramatically. Basically, whenever a child's fiber is created from its parent, the `FiberRef` value of parent fiber propagated to its child fiber.

### Copy-on-Fork 

`FiberRef[A]` has *copy-on-fork* semantics for `ZIO#fork`. This essentially means that a child `Fiber` starts with `FiberRef` values of its parent. When the child sets a new value of `FiberRef`, the change is visible only to the child itself. The parent fiber still has its own value.

So if we create a `FiberRef` and set its value to `5`, and we pass this `FiberRef` to a child fiber, it sees the value `5`. If the child fiber modifies the value from `5` to `6`, the parent fiber can't see that change. So the child fiber gets its own copy of the `FiberRef`, and it can modify it locally. Those changes will not affect the parent fiber:

```scala

for {
  fiberRef <- FiberRef.make(5)
  promise <- Promise.make[Nothing, Int]
  _ <- fiberRef
    .updateAndGet(_ => 6)
    .flatMap(promise.succeed).fork
  childValue <- promise.await
  parentValue <- fiberRef.get
} yield assert(parentValue == 5 && childValue == 6)
```

## Merging FiberRefs

ZIO does not only support to propagate `FiberRef` values from parents to childs, but also to merge back these values into the current fiber. This section describes multiple variants for doing so.

### join

If we `join` a fiber then the value of its `FiberRef` is merged back into the parent fiber. The default strategy for merging back, is **replacement**. This means whenever a forked fiber joined to its parent fiber, the value of its parent will be replaced with the value of its child `FiberRef`:

```scala

for {
  fiberRef <- FiberRef.make(5)
  child <- fiberRef.set(6).fork
  _ <- child.join
  parentValue <- fiberRef.get
} yield assert(parentValue == 6)
```

So if we `fork` a fiber and that child fiber modifies a bunch of `FiberRef`s and then later we `join` it, we get those modifications merged back into the parent fiber. So that's the semantic model of ZIO on `join`. 

Each fiber has its own `FiberRef` and can modify it independently. Therefore, when multiple child fibers `join` their parent, the last child fiber will override the parent's `FiberRef` value, replacing it with its own.

As we can see, `child1` is the last fiber, so its value, which is `6`, gets merged back into its parent:

```scala

for {
  fiberRef <- FiberRef.make(5)
  child1 <- fiberRef.set(6).fork
  child2 <- fiberRef.set(7).fork
  _ <- child2.join
  _ <- child1.join
  parentValue <- fiberRef.get
} yield assert(parentValue == 6)
```

### join with Custom Merge

Furthermore, we have the ability to customize the initialization of a value when a fiber is forked, as well as the method of value combination when merging back the values. To achieve this, you can specify the desired behavior when making the `FiberRef` using `FiberRef#make`:

```scala

for {
  fiberRef <- FiberRef.make(initial = 0, join = math.max)
  child    <- fiberRef.update(_ + 1).fork
  _        <- fiberRef.update(_ + 2)
  _        <- child.join
  value    <- fiberRef.get
} yield assert(value == 2)
```

In this example, when the child fiber joins its parent, it employs the max function to determine how to merge the values. It compares the child's FiberRef value (1) with the parent's FiberRef value (2) and selects the higher value as the merged result, which in this case is 2.

### await

It is important to note that `await` has no such merge behavior. So `await` waits for the child fiber to finish and gives us its value as an `Exit`, without ever merging any `FiberRef` values back into the parent:

```scala

for {
  fiberRef <- FiberRef.make(5)
  child <- fiberRef.set(6).fork
  _ <- child.await
  parentValue <- fiberRef.get
} yield assert(parentValue == 5)
```

`join` has higher-level semantics than `await` because it will fail if the child fiber failed, it will interrupt if the child is interrupted, and it will also merge back its value to its parent.

### inheritAll

We can inherit the values from all `FiberRef`s from an existing `Fiber` using the `Fiber#inheritAll` method:

```scala

for {
  fiberRef <- FiberRef.make[Int](0)
  latch    <- Promise.make[Nothing, Unit]
  fiber    <- (fiberRef.set(10) *> latch.succeed(())).fork
  _        <- latch.await
  _        <- fiber.inheritAll
  v        <- fiberRef.get
} yield v == 10
```

Note that `inheritAll` is automatically called on `join`. However, `join` will wait for merging the **final** values, while `inheritAll` will merge the **current** values and then continue:

```scala

val withJoin =
    for {
        fiberRef <- FiberRef.make[Int](0)
        fiber    <- (fiberRef.set(10) *> fiberRef.set(20).delay(2.seconds)).fork
        _        <- fiber.join  // wait for fiber's end and copy final result 20 into fiberRef
        v        <- fiberRef.get
    } yield assert(v == 20)
```

```scala

val withoutJoin =
    for {
        fiberRef <- FiberRef.make[Int](0)
        fiber    <- (fiberRef.set(10) *> fiberRef.set(20).delay(2.seconds)).fork
        _        <- fiber.inheritAll.delay(1.second) // copy intermediate result 10 into fiberRef and continue
        v        <- fiberRef.get
    } yield assert(v == 10)
```

## Compositional Updates and Patch Theory

In the previous section, we learned the following:

1. Whenever a child fiber merges back into its parent, the value of the child fiber is, by default, replaced with the parent's value.
2. When we have multiple child fibers, and all of them join their parent, the value of the last child to join will prevail, replacing the parent's value.

Let's examine these two rules with a simple example:

```scala

object Main extends ZIOAppDefault {
  val retries: FiberRef[Int] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(3)
    }

  def run =
    for {
      _ <- ZIO.unit
      f1 = retries.set(10).debug("set 10").delay(2.seconds)
      f2 = retries.set(5).debug("set 5")
      _ <- f1 <&> f2
      _ <- retries.get.debug("final retries value")
    } yield ()

}
```

The output of this program is:

```scala
set 5: ()
set 10: ()
final retries value: 10
```

As we can see from the program's output, when we delayed the `f1` workflow, it became the last child fiber to join its parent. And guess what? Its value of 10 ended up being the winner! Why? Well, it's because the default rule is that the child's value takes over the parent's value during the merge.

### The Problem

While developing the program, we might want to add additional configurations, such as `intervals`. In this case, we can easily include another `FiberRef` that holds the `intervals` config:

```scala

object Main extends ZIOAppDefault {

  val retries: FiberRef[Int] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(2)
    }

  val intervals: FiberRef[Int] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(3)
    }

  def run =
    for {
      _ <- retries.set(5) <&> intervals.set(3)
      _ <- retries.get.debug("final retries value")
      _ <- intervals.get.debug("final intervals value")
    } yield ()

}
```

The output of this program is:

```scala
final retries value: 5
final intervals value: 3
```

This illustrates that by incorporating more `FiberRef`s, we can concurrently update the underlying configuration values without any problems.

Since the two configurations are interconnected, it might be beneficial to create a new data type utilizing `Map[String, Int]`. This approach eliminates the necessity of encoding retry configurations in two distinct `FiberRef`s:

```scala

object Main extends ZIOAppDefault {
  val retryConfig: FiberRef[Map[String, Int]] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.make(
        Map(
          "retries" -> 3,
          "intervals" -> 2
        )
      )
    }

  def withRetry(n: Int) = retryConfig.update(_.updated("retries", n))

  def withIntervals(n: Int) = retryConfig.update(_.updated("intervals", n))

  def run =
    for {
      - <- withRetry(5) <&> withIntervals(3)
      _ <- retryConfig.get.debug("retryConfig")
    } yield ()

}
```

Unfortunately, with this change, the output does not align with our intended outcome:

```scala
retryConfig: Map(retries -> 3, intervals -> 3)
```

The intervals have been successfully updated, but the retries remain unchanged. Why is this the case? It's because both fibers are overwriting the same state, resulting in the corruption of the final value. In the previous scenario, the updates are as follows:

```scala
Parent fiber: Map(retries -> 3, intervals -> 2)
Left fiber:   Map(retries -> 5, intervals -> 2)
right fiber:  Map(retries -> 3, intervals -> 3)

Parent fiber joins the left fiber:  Map(retries -> 5, intervals -> 2)
Parent fiber joins the right fiber: Map(retries -> 3, intervals -> 3)
```

So, this is the reason why the retries value is not updated and ends up with the wrong value. The retries value is clobbered by the right fiber when it joins the parent fiber.

To solve this problem, we need a way to compose the updates. We need to be able to say, 'update the retries to 5 and then update the intervals to 3' or, conversely, 'update the intervals to 3 and then update the retries to 5'. We need to be able to compose updates. This is where compositional updates and the patch theory come into play.

### Differ and Patch

Before we dive into the code, let's try to understand some terminology:

```scala
trait Differ[Value, Patch] {
  def combine(first: Patch, second: Patch): Patch
  def diff(oldValue: Value, newValue: Value): Patch
  def empty: Patch
  def patch(patch: Patch)(oldValue: Value): Value
}
```

By having an instance of `Differ[Value, Patch]`, we have the ability to do the following:

  1. We can **diff** two values of type `Value` to generate a `Patch`. What does `Patch` mean? `Patch` is a data type that signifies the modifications made from one value to another. We can envision a patch as a "diff" between two values.
  2. With **combine** function we can provide two `Patch`s and combines them into a single `Patch`. This is useful for composing updates. For example, if we have a `Patch` that updates the `retries` to 5 and another `Patch` that updates the `intervals` to 3, we can combine them into a single `Patch` that updates both the `retries` and the `intervals`.
  3. Using the **patch** function we can apply a `Patch` to a value to produce a new value.
  4. The **empty** function gives us a `Patch` that represents no changes.

To implement a `Differ` for a data type, we need to implement these 4 functions. We have four laws associated with any `Differ` value of type `Differ[Value, Patch]`:

  1. The `combine` function is associative, which means that combining two patches and then combining the result with a third patch is the same as combining the first patch with the combination of the second and third patches.
  2. Combining a patch with an empty patch is the same as the patch itself.
  3. Diffing a value with itself produces an empty patch.
  4. Diffing two values and then patching the first value with the resulting patch results in the second value.
  5. Patching a value with an empty patch results in the original value.

ZIO includes some utilities which helps us to create `Differ` instances for more complex data types:

- Instances of `Differ` for common data types, such as `Map`, `Set`, and `Chunk`.
- `Differ.update[A]`, which constructs a differ that diffs two values by returning a function that sets the value to the new value.
- `Differ.map`, which constructs a map differ from a differ which knows how to diff the values of the map.
- `Differ#zip`, is used to combine two differs into a single differ that works on a tuple of values.
- `Differ#orElseEither`, is used to combine two differs into a single differ that works on an `Either` of two values.
- Using `Differ#transform`, we can convert a differ of one type (Value1) to a differ of another type (Value2) by providing two functions: one to convert the Value1 to Value2 and another to convert the Value2 to Value1.

Let's implement a `Differ` for `retryConfig` which is a FiberRef of type `Map[String, Int]`:

```scala

val differ   = Differ.map[String, Int, Int => Int](Differ.update[Int])
val patch1   = differ.diff(Map("retries" -> 3), Map("retries" -> 5))
val patch2   = differ.diff(Map("intervals" -> 2), Map("intervals" -> 3))
val combined = differ.combine(patch1, patch2)
val result   = differ.patch(combined)(Map("retries" -> 3, "intervals" -> 2))
println(result)
```

The output is as follows:

```scala
Map(retries -> 5, intervals -> 3)
```

### First Solution: Compositional Updates For `Map[String, Int]` Data Type

In previous section, we have successfully updated the `retries` and `intervals` values using compositional updates. Now we can use this differ to make the updates of our `FiberRef` composable:

```scala

object Main extends ZIOAppDefault {

  val differ = Differ.map[String, Int, Int => Int](Differ.update[Int])

  val retryConfig: FiberRef[Map[String, Int]] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.makePatch[Map[String, Int], Differ.MapPatch[
        String,
        Int,
        Int => Int
      ]](
        Map(
          "retries" -> 3,
          "intervals" -> 2
        ),
        differ = differ,
        fork0 = differ.empty
      )
    }

  def withRetry(n: Int): UIO[Unit] =
    retryConfig.update(_.updated("retries", n))

  def withIntervals(n: Int): UIO[Unit] =
    retryConfig.update(_.updated("intervals", n))

  def run = {
    for {
      _ <- withRetry(5) <&> withIntervals(3)
      _ <- retryConfig.get.debug("retryConfig")
    } yield ()

  }
}
```

The output is as follows:

```scala
retryConfig: Map(retries -> 5, intervals -> 3)
```

:::note
Please note that as the `combine` operation of `Differ` is associative, the order of the updates does not change the result. This is a very important property of compositional updates in concurrent environments where multiple fibers updating the same value when they join, but the order of the joins is not deterministic.
:::

### Second Solution: Compositional Updates For The `RetryConfig` Case Class 

We can take this example one step further and create a type-safe configuration data type for `RetryConfig` using scala case classes:

```scala
case class RetryConfig(
    retries: Int,
    intervals: Int
)
```

We can create a `Differ` for `RetryConfig` using the `Differ#transform` function:

```scala

val differ: Differ[RetryConfig, (Int => Int, Int => Int)] =
  Differ
    .update[Int]
    .zip(Differ.update[Int])
    .transform(
      { case (x, y) => RetryConfig.apply(x, y) },
      retryConfig => (retryConfig.retries, retryConfig.intervals)
    )
```

Now, as same as before, we can use this `differ` to make the updates of our new `FiberRef` composable:

```scala

object Main extends ZIOAppDefault {

  val retryConfig: FiberRef[RetryConfig] =
    Unsafe.unsafe { implicit unsafe =>
      FiberRef.unsafe.makePatch[RetryConfig, (Int => Int, Int => Int)](
        initialValue0 = RetryConfig(
          retries = 3,
          intervals = 2
        ),
        differ = differ,
        fork0 = differ.empty
      )
    }

  def withRetry(n: Int) = retryConfig.update(_.copy(retries = n))

  def withIntervals(n: Int) = retryConfig.update(_.copy(intervals = n))

  def run =
    for {
      _ <- withRetry(5) <&> withIntervals(3)
      _ <- retryConfig.get.debug("retryConfig")
    } yield ()
    
}
```

---

## Global Shared State Using Ref

One of the common use cases for `Ref` is to manage the state of applications, especially in concurrent environments. We can use the `Ref` data type, which is a purely functional description of a mutable reference.

> **Note:**
>
> In this section, we will only cover the basic usage of the `Ref` data type. To learn more details about the `Ref`, especially its usage in concurrent programming, please refer to the [`Ref`](../concurrency/ref.md) page on the concurrency section.

In the previous page, we have learned how to use recursive functions to manage the state of our application. However, this approach has the following drawbacks:
- We cannot share the state between multiple fibers.
- Sometime, writing the application logic is a bit tedious. It is somehow awkward to pass the state using function parameters.

Thanks to the `Ref` data type, we can easily use the `Ref` data type to manage the state of our application, whether we need concurrency or not.

In the previous section, we learned that we can have state management, even for effectful operations. Here is the last example we tried:

```scala

def inputNames: ZIO[Any, String, List[String]] = {
  def loop(names: List[String]): ZIO[Any, String, List[String]] = {
    Console.readLine("Please enter a name or `q` to exit: ").orDie.flatMap {
      case "q" =>
        ZIO.succeed(names)
      case name =>
        loop(names appended name)
    }
  }

  loop(List.empty[String])
}
```

This code can be rewritten using the `Ref` type, which is simpler than the previous one:

```scala

def getNames: ZIO[Any, String, List[String]] =
  Ref.make(List.empty[String])
    .flatMap { ref =>
      Console
        .readLine("Please enter a name or 'q' to exit: ")
        .orDie
        .repeatWhileZIO {
          case "q" => ZIO.succeed(false)
          case name => ref.update(_ appended name).as(true)
        } *> ref.get
    }
```

First, we created a mutable reference to the initial state value, which is an empty list. Then, we read from the console repeatedly until the user enters the "q" command. Finally, we got the value of the reference and returned it.

> **Note:**
>
> All the operations on the `Ref` data type are effectful. So when we are reading from or writing to a `Ref`, we are performing an effectful operation.

Now that we have learned how to use the `Ref` data type, we can use it to manage the state concurrently. For example, assume while we are reading from the console, we have another fiber that is trying to update the state from a different source:

```scala

def getNames: ZIO[Any, String, List[String]] =
  for {
    ref <- Ref.make(List.empty[String])
    f1 <- Console
      .readLine("Please enter a name or 'q' to exit: ")
      .orDie
      .repeatWhileZIO {
        case "q"  => ZIO.succeed(false)
        case name => ref.update(_ appended name).as(true)
      }.fork 
      f2 <- ZIO.foreachDiscard(Seq("John", "Jane", "Joe", "Tom")) { name =>
        ref.update(_ appended name) *> ZIO.sleep(1.second)
      }
      .fork
    _ <- f1.join
    _ <- f2.join
    v <- ref.get
  } yield v
```

## Counter Example

Let's write a counter using the `Ref` data type:

```scala

case class Counter(value: Ref[Int]) {
  def inc: UIO[Unit] = value.update(_ + 1)
  def dec: UIO[Unit] = value.update(_ - 1)
  def get: UIO[Int] = value.get
}

object Counter {
  def make: UIO[Counter] = Ref.make(0).map(Counter(_))
}
```

Here is the usage example of the `Counter`:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      c <- Counter.make
      _ <- c.inc
      _ <- c.inc
      _ <- c.dec
      _ <- c.inc
      v <- c.get
      _ <- ZIO.debug(s"This counter has a value of $v.")
    } yield ()
}
```

We can use this counter in a concurrent environment, e.g. in a RESTful API to count the number of requests. But for just an example, let's concurrently update the counter:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      c <- Counter.make
      _ <- c.inc <&> c.inc <&> c.dec <&> c.inc
      v <- c.get
      _ <- ZIO.debug(s"This counter has a value of $v.")
    } yield ()
}
```

---

## Introduction to State Management in ZIO

When we write a program, more often we need to keep track of some sort of state during the execution of the program. If an object has a state, its behavior is influenced by passing the time.

Here are some examples:

- **Counter**— Assume a RESTful API, which has a set of endpoints, and wants to keep track of how many requests have been made to each endpoint.
- **Bank Account Balance**— Each bank account has a balance, and it can be deposited or withdrawn. So its value is changing over time.
- **Temperature**— The temperature of a room is changing over time.
- **List length**— When we are iterating over a list of items, we might need to keep track of the number of items we have seen so far. So during the calculation of the length of the list, we need an intermediate state that records the number of items we have seen so far.

In imperative programming, one common way to store the state is using a variable. So we can update their values in place. But this approach can introduce bugs, especially when the state is shared between multiple components. So it is better to avoid using variables to keep track of the state.

From the aspect of concurrency, we have two general approaches to maintaining the state in functional programming:
1. **[Recursion](state-management-using-recursion.md)**— In this approach, we can update the state by passing the new state to the next component. This is a very easy way to maintain the state, but it can't be used in a concurrent environment, because we can't share the state between multiple fibers.

2. Concurrent— The concurrent state management is also has two variant, global and fiber-local state management:

   1. **[Global Shared State](global-shared-state.md)**- ZIO has a powerful data type called `Ref`, which is the description of a mutable reference. We can use `Ref` to share the state between multiple fibers, e.g. producer and consumer components.

   2. **[Fiber-local State](fiber-local-state.md)**— ZIO provides two data types called `FiberRef` and `ZState` that can be used to maintain the state in a concurrent environment, but each fiber has its own state. Their states are not shared between other fibers. This prevents them from clobbering each other's state.

In this section, we will talk about these approaches.

---

## State Management Using Recursion

This is a very common pattern to use variables to keep track of the state. For example, to calculate the length of a list, we can store intermediate results inside the `count` variable:

```scala
def length[T](list: List[T]): Int = {
  var count = 0
  for (_ <- list) count += 1
  count
}
```

But in functional programming, we avoid using variables to keep track of the state. Instead, we use other techniques.

One common technique is to pass the new state as an argument to the next function. After we created a new version of the state, we pass it to the next function. We do this until we have reached the final state.

Assume we have the following code:

```scala
var state = 5
state = state + 1
state = state * 2
state = state * state

println(state)
// Output: 144
```

We can rewrite it as a series of transformations, in which new states are passed to the next function:

```scala
def foo(state: Int): Int = bar(state + 1)
def bar(state: Int): Int = baz(state * 2)
def baz(state: Int): Int = state * state

println(foo(5)) 
// 144
// Output: 144
```

Now, what if we wanted to apply the transformation multiple times to a given state? We can combine this technique with recursive functions and call the function multiple times.

For example, we use function arguments to pass the state to the next function, using recursive calls. Assume we have a `length` function that returns the length of a list as below:

```scala
def length[T](list: List[T]): Int = {
  var count = 0
  for (_ <- list) count += 1
  count
}
```

If we want to convert the above function to a series of state transformations, first, we need to diagnose what is the state?

One obvious answer is the `count` variable. The `count` variable is the state of the above function that keeps track of the length of the list. Another non-obvious state is the remainder of the list that is not yet processed during the list traversal.

So we can model the state composed of the `count` and the `remainder` of the list:

```scala
case class State[T](count: Int, remainder: List[T])
```

Now we are ready to write the state transformation function called `loop` as below:

```scala
case class State[T](count: Int, remainder: List[T])

def loop[T](state: State[T]): State[T] = {
  state.remainder match {
    case Nil => state
    case _ :: tail => loop(State(state.count + 1, tail))
  }
}
```

Here are series of `loop` calls, when we call it with the `State(0, List("a", "b", "c", "d"))` state:

```scala
loop(State(0, List("a", "b", "c", "d")))
loop(State(1, List("b", "c", "d")))
loop(State(2, List("c", "d")))
loop(State(3, List("d")))
loop(State(4, List()))
// Output:
// State(4, List())
```

Let's do some small modification to the `loop` function and use it inside the `length` function:

```scala
def length[T](list: List[T]): Int = {
  def loop(list: List[T], count: Int): Int = {
    list match {
      case Nil => count
      case _ :: tail => loop(tail, count + 1)
    }
  }

  loop(list, 0)
}
```

The same pattern can be used when we have side effects. Assume we have a function that tries to read names from the input, until the user enters the "q" command indicating the end of the input. We can write the function like this:

```scala

def getNames: List[String] = {
  def getName() = readLine("Please enter a name or 'q' to exit: ")
  var names = List.empty[String]
  var input = getName()
  while (input != "q") {
    names = names appended input
    input = getName()
  }
  names
}
```

Using the previous pattern, we can eliminate the need to use variables:

```scala

def getNames: Seq[String] = {
  def loop(names: List[String]): List[String] = {
    val name = readLine("Please enter a name or 'q' to exit: ")
    if (name == "q") names else loop(names appended name)
  }
  loop(List.empty[String])
}
```

But, there is also a problem with the previous solution. The `getName` is not referentially transparent. In order to make it free of side effects, we can use `ZIO` to describe any effectual operation:

```scala

def inputNames: ZIO[Any, String, List[String]] = {
  def loop(names: List[String]): ZIO[Any, String, List[String]] = {
    Console.readLine("Please enter a name or `q` to exit: ").orDie.flatMap {
      case "q" =>
        ZIO.succeed(names)
      case name =>
        loop(names appended name)
    }
  }

  loop(List.empty[String])
}
```

On this page, we have learned how to have stateful computations in our programs using recursion. However, this approach is not suitable for concurrent programs, where multiple fibers want to change the state of the program concurrently. Let's move on to the next page, where we will discuss stateful computation over concurrent programs.

---

## ZState

`ZState[S]` models a value of type `S` that can be read from and written to during the execution of an effect. This is a higher-level construct built on top of [`FiberRef`](fiberref.md) and the environment type to support using ZIO where we might have traditionally used state monad transformers.

Let's try a simple example of using `ZState`:

```scala

object ZStateExample extends zio.ZIOAppDefault {
  val myApp: ZIO[ZState[Int], IOException, Unit] = for {
    s <- ZIO.service[ZState[Int]]
    _ <- s.update(_ + 1)
    _ <- s.update(_ + 2)
    state <- s.get
    _ <- Console.printLine(s"current state: $state")
  } yield ()

  def run = ZIO.stateful(0)(myApp)
}
```

The idiomatic way to work with `ZState` is as part of the environment using operators defined on `ZIO` to access the `ZState` from the environment, and finally, allocate the initial state using the `ZIO.stateful` operator.

Because we typically use `ZState` as part of the environment, it is recommended to define our own state type `S` such as `MyState` rather than using a type such as `Int` to avoid the risk of ambiguity:

```scala

final case class MyState(counter: Int)

object ZStateExample extends zio.ZIOAppDefault {

  val myApp: ZIO[ZState[MyState], IOException, Unit] =
    for {
      counter <- ZIO.service[ZState[MyState]]
      _ <- counter.update(state => state.copy(counter = state.counter + 1))
      _ <- counter.update(state => state.copy(counter = state.counter + 2))
      state <- counter.get
      _ <- Console.printLine(s"Current state: $state")
    } yield ()

  def run = ZIO.stateful(MyState(0))(myApp)
}
```

The `ZIO` data type also has some helper methods to work with `ZState` as the environment of `ZIO` effect such as `ZIO.updateState`, `ZIO.getState`, and `ZIO.getStateWith`:

```scala

final case class MyState(counter: Int)

val myApp: ZIO[ZState[MyState], IOException, Int] =
  for {
    _ <- ZIO.updateState[MyState](state => state.copy(counter = state.counter + 1))
    _ <- ZIO.updateState[MyState](state => state.copy(counter = state.counter + 2))
    state <- ZIO.getStateWith[MyState](_.counter)
    _ <- Console.printLine(s"Current state: $state")
  } yield state
```

An important note about `ZState` is that it is on top of the `FiberRef` data type. So it will inherit its behavior from the `FiberRef`.

For example, when a fiber is going to join to its parent fiber, its state will be merged with its parent state:

```scala

case class MyState(counter: Int)

object ZStateExample extends ZIOAppDefault {
  val myApp = for {
    _ <- ZIO.updateState[MyState](state => state.copy(counter = state.counter + 1))
    fiber <-
      (for {
        _ <- ZIO.updateState[MyState](state => state.copy(counter = state.counter + 1))
        state <- ZIO.getState[MyState]
        _ <- Console.printLine(s"Current state inside the forked fiber: $state")
      } yield ()).fork
    _ <- ZIO.updateState[MyState](state => state.copy(counter = state.counter + 5))
    state1 <- ZIO.getState[MyState]
    _ <- Console.printLine(s"Current state before merging the fiber: $state1")
    _ <- fiber.join
    state2 <- ZIO.getState[MyState]
    _ <- Console.printLine(s"The final state: $state2")
  } yield ()

  def run =
    ZIO.stateful(MyState(0))(myApp)
}
```

The output of running this snippet code would be as below:

```
Current state before merging the fiber: MyState(6)
Current state inside the forked fiber: MyState(2)
The final state: MyState(2)
```

---

## Introduction to Software Transactional Memory

## Overview

ZIO supports Software Transactional Memory (STM) which is a modular composable concurrency data structure. It allows us to combine and compose a group of memory operations and perform all of them in one single atomic operation.

Software Transactional Memory is an abstraction for concurrent communications. The main benefits of STM are composability and modularity. We can write concurrent abstractions that can be composed with any other abstraction built using STM, without exposing the details of how our abstraction ensures safety. This is typically not the case with the locking mechanism.

The idea of the transactional operation is not new, they have been the fundamental of distributed systems, and those databases that guarantee us an ACID property. Software transactional memory is just all about memory operations. All operations are performed on memory. It is not related to a remote system or a database. Very similar to the database concept of ACID property, but the _durability_, is missing which doesn't make sense for in-memory operations.

In transactional memory, we get these aspects of ACID properties:

- **Atomicity** — On write operations, we want _atomic update_, which means the update operation either should run at once or not at all.
- **Consistency** — On read operations, we want a _consistent view_ of the state of the program that ensures us all references to the state, get the same value whenever they get the state.
- **Isolated** — If we have multiple updates, we need to perform these updates in isolated transactions. So each transaction doesn't affect other concurrent transactions. No matter how many fibers are running any number of transactions. None of them have to worry about what is happening in the other transactions.

The ZIO STM API is inspired by Haskell's [STM library](http://hackage.haskell.org/package/stm-2.5.0.0/docs/Control-Concurrent-STM.html) although the implementation in ZIO is completely different.

## The Problem

Let's start from a simple `inc` function, which takes a mutable reference of `Int` and increases it by `amount`:

```scala
def inc(counter: Ref[Int], amount: Int) = for {
  c <- counter.get
  _ <- counter.set(c + amount)
} yield c
```

If there is only one fiber in the world, it is not a problem. This function sounds correct. But what happens if in between reading the value of the counter and setting a new value, another fiber comes and mutates the value of the counter? Another fiber is just updating the counter just after we read the counter. So this function is subject to a race condition, we can test that with the following program:

```scala
for {
  counter <- Ref.make(0)
  _ <- ZIO.collectAllPar(ZIO.replicate(10)(inc(counter, 1)))
  value <- counter.get
} yield (value)
```

The above program runs 10 concurrent fibers to increase the counter value. However, we cannot expect this program to always return 10 as a result. 

To fix this issue, we need to perform the `get` and `set` operations atomically. The `Ref` data type some other API like `update`, `updateAndGet`, and `modify` which perform the reading and writing atomically:

```scala
def inc(counter: Ref[Int], amount: Int) = counter.updateAndGet(_ + amount)
```

The most important note about the `modify` operation is that it doesn't use pessimistic locking. It doesn't use any locking primitives for the critical section. It has an optimistic assumption about occurring collisions.

The `modify` function takes these three steps:

1. It assumes that other fibers don't change the shared state and don't interfere in most cases. So it read the shared state without using any locking primitives.
2. It should prepare itself for the worst-case scenarios. If another fiber is accessing the data at the same time, what would happen? Therefore, when we write a new value, it should check everything. It must ensure that it sees a consistent state of the universe, and if it does, then it can change that value.
4. If it encounters an inconsistent value, it shouldn't continue. So it aborts updating the shared state with an invalidated assumption. It should retry the `modify` operation with an updated state.

Let's see how the `modify` function of `Ref` is implemented without any locking mechanism:

```scala
  final case class Ref[A](value: AtomicReference[A]) { self =>
    def modify[B](f: A => (B, A)): UIO[B] = ZIO.succeed {
      var loop = true
      var b: B = null.asInstanceOf[B]
      while (loop) {
        val current = value.get
        val tuple   = f(current)
        b = tuple._1
        loop = !value.compareAndSet(current, tuple._2)
      }
      b
    }
 }
```

As we see, the `modify` operation is implemented in terms of the `compare-and-swap` operation which helps us to perform read and update atomically.

Let's rename the `inc` function to the `deposit` as follows to try the classic problem of transferring money from one account to another:

```scala
def deposit(accountBalance: Ref[Int], amount: Int) = accountBalance.update(_ + amount)
```

And the `withdraw` function:

```scala
def withdraw(accountBalance: Ref[Int], amount: Int) = accountBalance.update(_ - amount) 
```

It seems pretty good, but we also need to check that there is sufficient balance in the account to withdraw. So let's add an invariant to check that:

```scala
def withdraw(accountBalance: Ref[Int], amount: Int) = for {
  balance <- accountBalance.get
  _ <- if (balance < amount) ZIO.fail("Insufficient funds in you account") else
    accountBalance.update(_ - amount)
} yield ()
```

What if in between checking and updating the balance, another fiber comes and withdraws all money in the account? This solution has a bug. It has the potential to reach a negative balance. 

Suppose we finally reached a solution to do withdraw atomically, the problem remains. We need a way to compose `withdraw` with `deposit` atomically to create a `transfer function:

```scala
def transfer(from: Ref[Int], to: Ref[Int], amount: Int) = for {
  _ <- withdraw(from, amount)
  _ <- deposit(to, amount)
} yield ()
```

In the above example, even if we assume that the `withdraw` and `deposit` are atomic, we can't compose these two transactions. They produce bugs in a concurrent environment. This code doesn't guarantee us that both `withdraw` and `deposit` are performed in one single atomic operation. Other fibers which are executing this `transfer` method can override the shared state and introduce a race condition.

We need a solution to **atomically compose transactions**. This is where software transactional memory comes into play.

## Composable Concurrency

Software transactional memory provides us a way to compose multiple transactions and perform them in one single transaction.

Let's continue our last effort to convert our `withdraw` method to be one atomic operation. To solve the problem using STM, we replace `Ref` with `TRef`. `TRef` stands for _Transactional Reference_; it is a mutable reference contained in the `STM` world. `STM` is a monadic data structure that represents an effect that can be performed transactionally:

```scala
def withdraw(accountBalance: TRef[Int], amount: Int): STM[String, Unit] =
  for {
    balance <- accountBalance.get
    _ <- if (balance < amount)
      STM.fail("Insufficient funds in you account")
    else
      accountBalance.update(_ - amount)
  } yield ()
```

Although the `deposit` operation is atomic, to be able to compose with `withdraw` we need to refactor it to take `TRef` and return `STM`:

```scala
def deposit(accountBalance: TRef[Int], amount: Int): STM[Nothing, Unit] =
  accountBalance.update(_ + amount)
```

In the `STM` world we can compose all operations and at the end of the world, we perform all of them in one single operation atomically. To be able to compose `withdraw` with `deposit` we need to stay in the `STM` world. Therefore, we didn't perform `STM.atomically` or `STM#commit` methods on each of them.

Now we can define the `transfer` method by composing these two functions in the `STM` world and converting them into the `IO` atomically:

```scala
def transfer(from: TRef[Int], to: TRef[Int], amount: Int): IO[String, Unit] =
  STM.atomically {
    for {
      _ <- withdraw(from, amount)
      _ <- deposit(to, amount)
    } yield ()
  }
```

Assume we are in the middle of transferring money from one account to the other. If we withdraw the first account but haven't deposited the second account, that kind of intermediate state is not visible to any external fibers. The transaction is completely successful if there are no conflicting changes. If there are any conflicts or conflicting changes then the whole transaction, the entire STM will be retried.

## How Does it Work?

The `STM` uses the same idea as the `Ref#modify` function but with a composability feature. The main goal of `STM` is to provide a mechanism to compose multiple transactions and perform them in one single atomic operation.

The mechanism behind the compositional part is obvious. The `STM` has its own world. It has lots of useful combinators like `flatMap` and `orElse` to compose multiple `STM`s and create more elegant ones. After we perform a transaction with `STM#commit` or `STM.atomically` the runtime system does the following steps. These steps are not exactly accurate, but they draw an outline of what happens during the transaction:

1. **Starting a Transaction** — When we start a transaction, the runtime system creates a virtual space to keep track of the transaction logs which is build up by recording the reads and tentative writes that the transaction will perform during the transaction steps.
2. **Virtual Execution** — The runtime starts speculating the execution of transactions on every read and write operation. It has two internal logs;  the read and the write log. On the read log, it saves the version of all variables it reads during the intermediate steps, and on the write log, it saves the intermediate result of the transaction. It doesn't change the shared state on the main memory. Anything that is inside an atomic block is not executed immediately, it's executed in the virtual world, just by putting stuff in the internal log, not in the main memory. In this particular model, we guarantee that all computations are isolated from one another.
3. **Commit Phase (Real Execution)** — When it comes to the end of the transaction the runtime system should check everything it has read. It should make sure that it sees a consistent state of the universe and if it has, then it atomically commits. As the STM is optimistic, it assumes that in the middle of a transaction, the chance of interfering with the shared state by other fibers is very rare. But it must ready itself for the worst cases. It should validate its assumption in the final stage. It checks whether the transactional variables involved were modified by any other threads or not. If its assumption got invalidated in the meanwhile of the transaction, it should abandon the transaction and retry it again. It jumps to the start of the transaction with the original and default values and tries again until it succeeds; This is necessary to resolve conflicts. Otherwise, if there is no conflict, it commits the final value atomically to the memory and succeeds. From the point of view of other fibers, all values in memory exchange in one blink of an eye. It's all atomic.

Everything done within a transaction to other transactions looks like it happens at once or not at all. So no matter how many pieces of memory it touches during the transaction. From the other transaction perspective, all of these changes happen at once.

## STM Data Types

Like the `ZIO` data type, the `ZSTM` has some type aliases as follows:

```scala
type RSTM[-R, +A]  = ZSTM[R, Throwable, A]
type URSTM[-R, +A] = ZSTM[R, Nothing, A]
type STM[+E, +A]   = ZSTM[Any, E, A]
type USTM[+A]      = ZSTM[Any, Nothing, A]
type TaskSTM[+A]   = ZSTM[Any, Throwable, A]
```

There are a variety of transactional data structures that can take part in an STM transaction:

- **[TArray](tarray.md)** - A `TArray[A]` is an array of mutable references that can participate in transactions.
- **[TRandom](trandom.md)** — `TRandom` is a random service that provides utilities to generate random numbers, which can participate in STM transactions.
- **[TSet](tset.md)** - A `TSet` is a mutable set that can participate in transactions.
- **[TMap](tmap.md)** - A `TMap[A]` is a mutable map that can participate in transactions.
- **[TRef](tref.md)** - A `TRef` is a mutable reference to an immutable value that can participate in transactions.
- **[TPriorityQueue](tpriorityqueue.md)** - A `TPriorityQueue[A]` is a mutable priority queue that can participate in transactions.
- **[TPromise](tpromise.md)** - A `TPromise` is a mutable reference that can be set exactly once and can participate in transactions.
- **[TQueue](tqueue.md)** - A `TQueue` is a mutable queue that can participate in transactions.
- **[TReentrantLock](treentrantlock.md)** - A `TReentrantLock` is a reentrant read / write lock that can be composed.
- **[TSemaphore](tsemaphore.md)** - A `TSemaphore` is a semaphore that can participate in transactions.
- **[THub](thub.md)** - A `THub` is a hub that can participate in STM transactions.

Since STM places a great emphasis on compositionality, we can build upon these data structures and define our very own concurrent data structures. For example, we can build a transactional priority queue using `TRef`, `TMap` and `TQueue`.

## Advantage of Using STM

1. **Composable Transaction** — Combining atomic operations using locking-oriented programming is almost impossible. ZIO provides the `STM` data type, which has lots of combinators to compose transactions.
2. **Declarative** — ZIO STM is completely declarative. It doesn't require us to think about low-level primitives. It doesn't force us to think about the ordering of locks. Reasoning concurrent programs in a declarative fashion is very simple. We can just focus on the logic of our program and run it in a concurrent environment deterministically. The user code is much simpler of course because it doesn't have to deal with the concurrency at all. 
3. **Optimistic Concurrency** — In most cases, we are allowed to be optimistic unless there is tremendous contention. So if we haven't tremendous contention it really pays to be optimistic. It allows a higher volume of concurrent transactions.
4. **Lock-Free** — All operations are non-blocking using lock-free algorithms.
5. **Fine-Grained Locking**— Coarse-grained locking is very simple to implement, but it has a negative impact on performance, while fine-grained locking significantly has better performance, but it is very cumbersome, sophisticated, and error-prone even for experienced programmers. We would like to have the ease of use of coarse-grain locking, but at the same time, we would like to have the efficiency of fine-grain locking. ZIO provides several data types which are a very coarse way of using concurrency, but they are implemented as if every single word were lockable. So the granularity of concurrency is fine-grained. It increases the performance and concurrency. For example, if we have two fibers accessing the same `TArray`, one of them reads and writes on the first index of our array, and another one reads and writes to the second index of that array, they will not conflict. It is just like as if we were locking the indices, not the whole array. 

## Implication of Using STM

1. **Running I/O Inside STM**— There is a strict boundary between the `STM` world and the `ZIO` world. This boundary propagates even deeper because we are not allowed to execute arbitrary effects in the `STM` universe. Performing side effects and I/O operations inside a transaction is problematic. In the `STM` the only effect that exists is the `STM` itself. We cannot print something or launch a missile inside a transaction as it will nondeterministically get printed on every reties that transaction does that.
2. **Large Allocations** — We should be very careful in choosing the best data structure for using STM operations. For example, if we use a single data structure with `TRef` and that data structure occupies a big chunk of memory. Every time we are updating this data structure during the transaction, the runtime system needs a fresh copy of this chunk of memory.
3. **Running Expensive Operations**— The beautiful feature of the `retry` combinator is when we decide to retry the transaction, the `retry` avoids the busy loop. It waits until any of the underlying transactional variables have changed. However, we should be careful about running expensive operations multiple times.

---

## STM

An `STM[E, A]` represents an effect that can be performed transactionally resulting in a failure `E` or a success `A`. There is a more powerful variant `ZSTM[R, E, A]` which supports an environment type `R` like `ZIO[R, E, A]`.

The `STM` (and `ZSTM` variant) data-type is _not_ as powerful as the `ZIO[R, E, A]` datatype as it does not allow you to perform arbitrary effects. These are because actions inside STM actions can be executed an arbitrary amount of times (and rolled-back as well). Only STM actions and pure computation may be performed inside a memory transaction. 

No STM actions can be performed outside a transaction, so you cannot accidentally read or write a transactional data structure outside the protection of `STM.atomically` (or without explicitly `commit`ting the transaction). For example:

```scala

def transferMoney(from: TRef[Long], to: TRef[Long], amount: Long): STM[String, Long] =
  for {
    senderBal <- from.get
    _         <- if (senderBal < amount) STM.fail("Not enough money")
                 else STM.unit
    _         <- from.update(existing => existing - amount)
    _         <- to.update(existing => existing + amount)
    recvBal   <- to.get
  } yield recvBal

val program: IO[String, Long] = for {
  sndAcc  <- STM.atomically(TRef.make(1000L))
  rcvAcc  <- STM.atomically(TRef.make(0L))
  recvAmt <- STM.atomically(transferMoney(sndAcc, rcvAcc, 500L))
} yield recvAmt
```

`transferMoney` describes an atomic transfer process between a sender and a receiver. The transaction will fail if the sender does not have enough of money in their account. This means that individual accounts will be debited and credited atomically. If the transaction fails in the middle, the entire process will be rolled back, and it will appear that  nothing has happened.

Here, we see that `STM` effects compose using a for-comprehension and that wrapping an `STM` effect with `STM.atomically` (or calling `commit` on any STM effect) turns the `STM` effect into a `ZIO` effect which can be executed. 

STM transactions compose sequentially. By using `STM.atomically` (or `commit`), the programmer identifies atomic transaction in the sense that the entire set of operations within `STM.atomically` appears to take place indivisibly.

## Errors

`STM` supports errors just like `ZIO` via the error channel. In `transferMoney`, we saw an example of an error (`STM.fail`). 

Errors in `STM` have abort semantics: if an atomic transaction encounters an error, the transaction is rolled back with no effect.

## `retry`

`STM.retry` is central to making transactions composable when they may block. For example, if we wanted to ensure that the money transfer took place when the sender had enough of money (instead of failing right away), we can use `STM.retry` instead:

```scala
def transferMoneyNoMatterWhat(from: TRef[Long], to: TRef[Long], amount: Long): STM[String, Long] =
  for {
    senderBal <- from.get
    _         <- if (senderBal < amount) STM.retry else STM.unit
    _         <- from.update(existing => existing - amount)
    _         <- to.update(existing => existing + amount)
    recvBal   <- to.get
  } yield recvBal
```

`STM.retry` will abort and retry the entire transaction until it succeeds (instead of failing like the previous example).

Note that the transaction will only be retried when one of the underlying transactional data structures have been changed.

There are many other variants of the `STM.retry` combinator like `STM.check` so rather than writing `if (senderBal < amount) STM.retry else STM.unit`, you can replace it with `STM.check(senderBal < amount)`.

## Composing alternatives

STM transactions compose sequentially so that both STM effects are executed. However, STM transactions can also compose transactions as alternatives so that only one STM effect is executed by making use of `orTry` on STM effects. 

Provided we have two STM effects `sA` and `sB`, you can express that you would like to compose the two using `sA orTry sB`. The transaction would first attempt to run `sA` and if it retries then `sA` is abandoned with no effect and then `sB` runs. Now if `sB` also retries then the entire call retries. However, it waits for the transactional data structures to change that are involved in either `sA` or `sB`. 

Using `orTry` is an elegant technique that can be used to determine whether or not an STM transaction needs to block. For example, we can take `transferMoneyNoMatterWhat` and turn it into an STM transaction that will fail immediately if the sender does not have enough of money instead of retrying by doing:

```scala
def transferMoneyFailFast(from: TRef[Long], to: TRef[Long], amount: Long): STM[String, Long] =
    transferMoneyNoMatterWhat(from, to, amount) orTry STM.fail("Sender does not have enough of money")
```

This will cause the transfer to fail immediately if the sender does not have money because of the semantics of `orTry`.

---

## TArray

`TArray` is an array of mutable references that can participate in transactions in STM.

## Create a TArray

Creating an empty `TArray`:

```scala

val emptyTArray: STM[Nothing, TArray[Int]] = TArray.empty[Int]
```

Or creating a `TArray` with specified values:

```scala

val specifiedValuesTArray: STM[Nothing, TArray[Int]] = TArray.make(1, 2, 3)
```

Alternatively, you can create a `TArray` by providing a collection of values:

```scala

val iterableTArray: STM[Nothing, TArray[Int]] = TArray.fromIterable(List(1, 2, 3))
```

## Retrieve the value from a TArray

The n-th element of the array can be obtained as follows:

```scala

val tArrayGetElem: UIO[Int] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  elem   <- tArray(2)
} yield elem).commit
```

Accessing the non-existing indexes aborts the transaction with `ArrayIndexOutOfBoundsException`.

## Update the value of a TArray

Updating the n-th element of an array can be done as follows:

```scala

val tArrayUpdateElem: UIO[TArray[Int]] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  _      <- tArray.update(2, el => el + 10)
} yield tArray).commit
```

Updating the n-th element of an array can be done effectfully via `updateSTM`:

```scala

val tArrayUpdateMElem: UIO[TArray[Int]] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  _      <- tArray.updateSTM(2, el => STM.succeed(el + 10))
} yield tArray).commit
```

Updating the non-existing indexes aborts the transaction with `ArrayIndexOutOfBoundsException`.

## Transform elements of a TArray

The transform function `A => A` allows computing a new value for every element in the array: 

```scala

val transformTArray: UIO[TArray[Int]] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  _      <- tArray.transform(a => a * a)
} yield tArray).commit
```

The elements can be mapped effectfully via `transformSTM`:

```scala

val transformSTMTArray: UIO[TArray[Int]] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  _      <- tArray.transformSTM(a => STM.succeed(a * a))
} yield tArray).commit
```

Folds the elements of a `TArray` using the specified associative binary operator:

```scala

val foldTArray: UIO[Int] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  sum    <- tArray.fold(0)(_ + _)
} yield sum).commit
```

The elements can be folded effectfully via `foldSTM`:

```scala

val foldSTMTArray: UIO[Int] = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  sum    <- tArray.foldSTM(0)((acc, el) => STM.succeed(acc + el))
} yield sum).commit
```

## Perform effects for TArray elements

`foreach` is used for performing an STM effect for each element in the array:

```scala

val foreachTArray = (for {
  tArray <- TArray.make(1, 2, 3, 4)
  tQueue <- TQueue.unbounded[Int]
  _      <- tArray.foreach(a => tQueue.offer(a).unit)
} yield tArray).commit
```

---

## THub

A `THub` is a transactional message hub. Publishers can publish messages to the hub and subscribers can subscribe to take messages from the hub.

A `THub` is an asynchronous message hub like `Hub` but it can participate in STM transactions. APIs are almost identical, but they are in the `STM` world rather than the `ZIO` world.

The fundamental operators on a `THub` are `publish` and `subscribe`:

```scala
trait THub[A] {
  def publish(a: A): USTM[Boolean]
  def subscribe: USTM[TDequeue[B]]
}
```

---

## TMap

A `TMap[A]` is a mutable map that can participate in transactions in STM.

## Create a TMap

Creating an empty `TMap`:

```scala

val emptyTMap: STM[Nothing, TMap[String, Int]] = TMap.empty[String, Int]
```

Or creating a `TMap` with specified values:

```scala

val specifiedValuesTMap: STM[Nothing, TMap[String, Int]] = TMap.make(("a", 1), ("b", 2), ("c", 3))
```

Alternatively, you can create a `TMap` by providing a collection of tuple values:

```scala

val iterableTMap: STM[Nothing, TMap[String, Int]] = TMap.fromIterable(List(("a", 1), ("b", 2), ("c", 3)))
```

## Put a key-value pair to a TMap

New key-value pair can be added to the map in the following way:

```scala

val putElem: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2))
  _    <- tMap.put("c", 3)
} yield tMap).commit
```

Another way of adding an entry in the map is by using `merge`:

```scala

val mergeElem: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.merge("c", 4)((x, y) => x * y)
} yield tMap).commit
```

If the key is not present in the map it behaves as a simple `put` method. It merges the existing value with the new one using the provided function otherwise.

## Remove an element from a TMap

The simplest way to remove a key-value pair from `TMap` is using `delete` method that takes key:

```scala

val deleteElem: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.delete("b")
} yield tMap).commit
```

Also, it is possible to remove every key-value pairs that satisfy provided predicate:

```scala

val removedEvenValues: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3), ("d", 4))
  _    <- tMap.removeIf((_, v) => v % 2 == 0)
} yield tMap).commit
```

Or you can keep all key-value pairs that match predicate function:

```scala

val retainedEvenValues: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3), ("d", 4))
  _    <- tMap.retainIf((_, v) => v % 2 == 0)
} yield tMap).commit
```

Note that `retainIf` and `removeIf` serve the same purpose as `filter` and `filterNot`. The reason for naming them differently was to emphasize a distinction in their nature. Namely, both `retainIf` and `removeIf` are destructive - calling them can modify the collection.

## Retrieve the value from a TMap

Value associated with the key can be obtained as follows: 

```scala

val elemGet: UIO[Option[Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  elem <- tMap.get("c")
} yield elem).commit
```

Alternatively, you can provide a default value if entry by key is not present in the map:

```scala

val elemGetOrElse: UIO[Int] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  elem <- tMap.getOrElse("d", 4)
} yield elem).commit
```

## Transform entries of a TMap

The transform function `(K, V) => (K, V)` allows computing a new value for every entry in the map: 

```scala

val transformTMap: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.transform((k, v) => k -> v * v)
} yield tMap).commit
```

Note that it is possible to shrink a `TMap`:

```scala

val shrinkTMap: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.transform((_, v) => "d" -> v)
} yield tMap).commit
```

The entries can be mapped effectfully via `transformSTM`:

```scala

val transformSTMTMap: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.transformSTM((k, v) => STM.succeed(k -> v * v))
} yield tMap).commit
```

The `transformValues` function `V => V` allows computing a new value for every value in the map: 

```scala

val transformValuesTMap: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.transformValues(v => v * v)
} yield tMap).commit
```

The values can be mapped effectfully via `transformValuesSTM`:

```scala

val transformValuesMTMap: UIO[TMap[String, Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  _    <- tMap.transformValuesSTM(v => STM.succeed(v * v))
} yield tMap).commit
```

Note that both `transform` and `transformValues` serve the same purpose as `map` and `mapValues`. The reason for naming them differently was to emphasize a distinction in their nature. Namely, both `transform` and `transformValues` are destructive - calling them can modify the collection.

Folds the elements of a `TMap` using the specified associative binary operator:

```scala

val foldTMap: UIO[Int] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  sum  <- tMap.fold(0) { case (acc, (_, v)) => acc + v }
} yield sum).commit
```

The elements can be folded effectfully via `foldSTM`:

```scala

val foldSTMTMap: UIO[Int] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  sum  <- tMap.foldSTM(0) { case (acc, (_, v)) => STM.succeed(acc + v) }
} yield sum).commit
```

## Perform effects for TMap key-value pairs

`foreach` is used for performing an STM effect for each key-value pair in the map:

```scala

val foreachTMap = (for {
  tMap   <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  tQueue <- TQueue.unbounded[String]
  _      <- tMap.foreach((k, v) => tQueue.offer(s"$k -> $v").unit)
} yield tMap).commit
```

## Check TMap membership

Checking whether key-value pair is present in a `TMap`:

```scala

val tMapContainsValue: UIO[Boolean] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  res  <- tMap.contains("a")
} yield res).commit
```

## Convert TMap to a List

List of tuples can be obtained as follows:

```scala

val tMapTuplesList: UIO[List[(String, Int)]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  list <- tMap.toList
} yield list).commit
```

List of keys can be obtained as follows:

```scala

val tMapKeysList: UIO[List[String]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  list <- tMap.keys
} yield list).commit
```

List of values can be obtained as follows:

```scala

val tMapValuesList: UIO[List[Int]] = (for {
  tMap <- TMap.make(("a", 1), ("b", 2), ("c", 3))
  list <- tMap.values
} yield list).commit
```

---

## TPriorityQueue

A `TPriorityQueue[A]` is a mutable queue that can participate in STM transactions. A `TPriorityQueue` contains values of type `A` for which an `Ordering` is defined. Unlike a `TQueue`, `take` returns the highest priority value (the value that is first in the specified ordering) as opposed to the first value offered to the queue. The ordering of elements sharing the same priority when taken from the queue is not guaranteed.

## Creating a TPriorityQueue

You can create an empty `TPriorityQueue` using the `empty` constructor:

```scala

val minQueue: STM[Nothing, TPriorityQueue[Int]] =
  TPriorityQueue.empty
```

Notice that a `TPriorityQueue` is created with an implicit `Ordering`. By default, `take` will return the value that is first in the specified ordering. For example, in a queue of events ordered by time the earliest event would be taken first. If you want a different behavior you can use a custom `Ordering`.

```scala
val maxQueue: STM[Nothing, TPriorityQueue[Int]] =
  TPriorityQueue.empty(Ordering[Int].reverse)
```

You can also create a `TPriorityQueue` initialized with specified elements using the `fromIterable` or `make` constructors". The `fromIterable` constructor takes a `Iterable` while the `make` constructor takes a variable arguments sequence of elements.

## Offering elements to a TPriorityQueue

You can offer elements to a `TPriorityQueue` using the `offer` or `offerAll` methods. The `offerAll` method is more efficient if you want to offer more than one element to the queue at the same time.

```scala
val queue: STM[Nothing, TPriorityQueue[Int]] =
  for {
    queue <- TPriorityQueue.empty[Int]
    _     <- queue.offerAll(List(2, 4, 6, 3, 5, 6))
  } yield queue
```

## Taking elements from a TPriorityQueue

Take an element from a `TPriorityQueue` using the `take`. `take` will semantically block until there is at least one value in the queue to take. You can also use `takeAll` to immediately take all values that are currently in the queue, or `takeUpTo` to immediately take up to the specified number of elements from the queue.

```scala
val sorted: STM[Nothing, Chunk[Int]] =
  for {
    queue  <- TPriorityQueue.empty[Int]
    _      <- queue.offerAll(List(2, 4, 6, 3, 5, 6))
    sorted <- queue.takeAll
  } yield sorted
```

You can also use `takeOption` method to take the first value from the queue if it exists without suspending or the `peek` method to observe the first element of the queue if it exists without removing it from the queue.

Sometimes you want to take a snapshot of the current state of the queue without modifying it. For this the `toChunk` combinator or its variants `toList` or `toVector` are extremely helpful. These will return an immutable collection that consists of all of the elements currently in the queue, leaving the state of the queue unchanged.

## Size of a TPriorityQueue

You can check the size of the `TPriorityQueue` using the `size` method:

```scala

val size: STM[Nothing, Int] =
  for {
    queue <- TPriorityQueue.empty[Int]
    _     <- queue.offerAll(List(2, 4, 6, 3, 5, 6))
    size  <- queue.size
  } yield size
```

---

## TPromise

`TPromise` is a mutable reference that can be set exactly once and can participate in transactions in STM.

## Create a TPromise

Creating a `TPromise`:

```scala

val tPromise: STM[Nothing, TPromise[String, Int]] = TPromise.make[String, Int]
```

## Complete a TPromise

In order to successfully complete a `TPromise`:

```scala

val tPromiseSucceed: UIO[TPromise[String, Int]] = for {
  tPromise <- TPromise.make[String, Int].commit
  _        <- tPromise.succeed(0).commit
} yield tPromise
```

In order to fail a `TPromise` use:

```scala

val tPromiseFail: UIO[TPromise[String, Int]] = for {
  tPromise <- TPromise.make[String, Int].commit
  _        <- tPromise.fail("failed").commit
} yield tPromise
```

Alternatively, you can use `done` combinator and complete the promise by passing it `Either[E, A]`:

```scala

val tPromiseDoneSucceed: UIO[TPromise[String, Int]] = for {
  tPromise <- TPromise.make[String, Int].commit
  _        <- tPromise.done(Right(0)).commit
} yield tPromise

val tPromiseDoneFail: UIO[TPromise[String, Int]] = for {
  tPromise <- TPromise.make[String, Int].commit
  _        <- tPromise.done(Left("failed")).commit
} yield tPromise
```

Once the value is set, any following attempts to set it will result in `false`.

## Retrieve the value of a TPromise

Returns the result if the promise has already been completed or a `None` otherwise:

```scala

val tPromiseOptionValue: UIO[Option[Either[String, Int]]] = for {
  tPromise <- TPromise.make[String, Int].commit
  _        <- tPromise.succeed(0).commit
  res      <- tPromise.poll.commit
} yield res
```

Alternatively, you can wait for the promise to be completed and return the value:

```scala

val tPromiseValue: IO[String, Int] = for {
  tPromise <- TPromise.make[String, Int].commit
  _        <- tPromise.succeed(0).commit
  res      <- tPromise.await.commit
} yield res
```

---

## TQueue

A `TQueue[A]` is a mutable queue that can participate in transactions in STM.

## Create a TQueue

Creating an empty bounded `TQueue` with specified capacity:

```scala

val tQueueBounded: STM[Nothing, TQueue[Int]] = TQueue.bounded[Int](5)
```

Creating an empty unbounded `TQueue`:

```scala

val tQueueUnbounded: STM[Nothing, TQueue[Int]] = TQueue.unbounded[Int]
```

## Put element(s) in a TQueue

In order to put an element to a `TQueue`:

```scala

val tQueueOffer: UIO[TQueue[Int]] = (for {
  tQueue <- TQueue.bounded[Int](3)
  _      <- tQueue.offer(1)
} yield tQueue).commit
```

The specified element will be successfully added to a queue if the queue is not full.
It will wait for an empty slot in the queue otherwise.

Alternatively, you can provide a list of elements:

```scala

val tQueueOfferAll: UIO[TQueue[Int]] = (for {
  tQueue <- TQueue.bounded[Int](3)
  _      <- tQueue.offerAll(List(1, 2))
} yield tQueue).commit
```

## Retrieve element(s) from a TQueue

The first element of the queue can be obtained as follows:

```scala

val tQueueTake: UIO[Int] = (for {
  tQueue <- TQueue.bounded[Int](3)
  _      <- tQueue.offerAll(List(1, 2))
  res    <- tQueue.take
} yield res).commit
```

In case the queue is empty it will block execution waiting for the element you're asking for.

This behavior can be avoided by using `poll` method that will return an element if exists or `None` otherwise:

```scala

val tQueuePoll: UIO[Option[Int]] = (for {
  tQueue <- TQueue.bounded[Int](3)
  res    <- tQueue.poll
} yield res).commit
```

Retrieving first `n` elements of the queue:

```scala

val tQueueTakeUpTo: UIO[Chunk[Int]] = (for {
  tQueue <- TQueue.bounded[Int](4)
  _      <- tQueue.offerAll(List(1, 2))
  res    <- tQueue.takeUpTo(3)
} yield res).commit
```

All elements of the queue can be obtained as follows:

```scala

val tQueueTakeAll: UIO[Chunk[Int]] = (for {
  tQueue <- TQueue.bounded[Int](4)
  _      <- tQueue.offerAll(List(1, 2))
  res    <- tQueue.takeAll
} yield res).commit
```

## Size of a TQueue

The number of elements in the queue can be obtained as follows:

```scala

val tQueueSize: UIO[Int] = (for {
  tQueue <- TQueue.bounded[Int](3)
  _      <- tQueue.offerAll(List(1, 2))
  size   <- tQueue.size
} yield size).commit
```

---

## TRandom

`TRandom` is a random service like [Random](../services/random.md) that provides utilities to generate random numbers, but they can participate in STM transactions.

The `TRandom` service is the same as the `Random` service. There are no differences in operations, but all return types are in the `STM` world rather than the `ZIO` world:

| Function      | Input Type    | Output Type                   |
| --------------| ------------- | ----------------------------- |
| `nextBoolean` |               | `URSTM[TRandom, Boolean]`     |
| `nextBytes`   | `length: Int` | `URSTM[TRandom, Chunk[Byte]]` |
| `nextDouble`  |               | `URSTM[TRandom, Double]`      |
| `nextInt`     |               | `URSTM[TRandom, Int]`         |
| ...           | ...           | ...                           |

When we use operations of the `TRandom` service, they add `TRandom` dependency on our `STM` data type. After committing all the transactions, we can `inject`/`provide` a `TRandom` implementation into our effect:

```scala
myApp.provide(TRandom.live)
```

---

## TReentrantLock

A `TReentrantLock` allows safe concurrent access to some mutable state efficiently, allowing multiple fibers to read the 
state (because that is safe to do) but only one fiber to modify the state (to prevent data corruption). Also, even though 
the `TReentrantLock` is implemented using `STM`; reads and writes can be committed, allowing this to be used as a building 
block for solutions that expose purely ZIO effects and internally allow locking on more than one piece of state in a 
simple and composable way (thanks to STM).

A `TReentrantLock` is a _reentrant_ read/write lock. A reentrant lock is one where a fiber can claim the lock multiple 
times without blocking on itself. It's useful in situations where it's not easy to keep track of whether you have already 
grabbed a lock. If a lock is non re-entrant you could grab the lock, then block when you go to grab it again, effectively 
causing a deadlock. 

## Semantics 

This lock allows both readers and writers to reacquire read or write locks with reentrancy guarantees. Readers are not 
allowed until all write locks held by the writing fiber have been released. Writers are not allowed unless there are no 
other locks or the fiber wanting to hold a write lock already has a read lock and there are no other fibers holding a 
read lock. 

This lock also allows upgrading from a read lock to a write lock (automatically) and downgrading 
from a write lock to a read lock (automatically provided that you upgraded from a read lock to a write lock).

## Creating a reentrant lock

```scala

val reentrantLock = TReentrantLock.make
```

## Acquiring a read lock

```scala

val program =
  (for {
    lock <- TReentrantLock.make
    _    <- lock.acquireRead
    rst  <- lock.readLocked  // lock is read-locked once transaction completes
    wst  <- lock.writeLocked // lock is not write-locked
  } yield rst && !wst).commit
```

## Acquiring a write lock

```scala

val writeLockProgram: UIO[Boolean] =
  (for {
    lock <- TReentrantLock.make
    _    <- lock.acquireWrite
    wst  <- lock.writeLocked // lock is write-locked once transaction completes
    rst  <- lock.readLocked  // lock is not read-locked
  } yield !rst && wst).commit
```

## Multiple fibers can hold read locks

```scala

val multipleReadLocksProgram: UIO[(Int, Int)] = for {
  lock          <- TReentrantLock.make.commit
  fiber0        <- lock.acquireRead.commit.fork // fiber0 acquires a read-lock
  currentState1 <- fiber0.join                  // 1 read lock held
  fiber1        <- lock.acquireRead.commit.fork // fiber1 acquires a read-lock
  currentState2 <- fiber1.join                  // 2 read locks held 
} yield (currentState1, currentState2)
```

## Upgrading and downgrading locks

If your fiber already has a read lock then it is possible to upgrade the lock to a write lock provided that no other
reader (other than your fiber) holds a lock
```scala

val upgradeDowngradeProgram: UIO[(Boolean, Boolean, Boolean, Boolean)] = for {
  lock               <- TReentrantLock.make.commit
  _                  <- lock.acquireRead.commit
  _                  <- lock.acquireWrite.commit  // upgrade
  isWriteLocked      <- lock.writeLocked.commit   // now write-locked
  isReadLocked       <- lock.readLocked.commit    // and read-locked
  _                  <- lock.releaseWrite.commit  // downgrade
  isWriteLockedAfter <- lock.writeLocked.commit   // no longer write-locked
  isReadLockedAfter  <- lock.readLocked.commit    // still read-locked
} yield (isWriteLocked, isReadLocked, isWriteLockedAfter, isReadLockedAfter)
```

## Acquiring a write lock in a contentious scenario

A write lock can be acquired immediately only if one of the following conditions are satisfied:
1. There are no other holders of the lock
2. The current fiber is already holding a read lock and there are no other parties holding a read lock

If either of the above scenarios are untrue then attempting to acquire a write lock will semantically block the fiber.
Here is an example which demonstrates that a write lock can only be obtained by the fiber once all other readers (except 
the fiber attempting to acquire the write lock) have released their hold on the (read or write) lock.

```scala

val writeLockDemoProgram: UIO[Unit] = for {
  l  <- TReentrantLock.make.commit
  _  <- printLine("Beginning test").orDie
  f1 <- (l.acquireRead.commit *> ZIO.sleep(5.seconds) *> l.releaseRead.commit).fork
  f2 <- (l.acquireRead.commit *> printLine("read-lock").orDie *> l.acquireWrite.commit *> printLine("I have upgraded!").orDie).fork
  _  <- (f1 zip f2).join
} yield ()
```

Here fiber `f1` acquires a read lock and sleeps for 5 seconds before releasing it. Fiber `f2` also acquires a read
lock and immediately tries to acquire a write lock. However, `f2` will have to semantically block for approximately 5 
seconds to obtain a write lock because `f1` will release its hold on the lock and only then can `f2` acquire a hold for
the write lock. 

## Safer methods  (`readLock` and `writeLock`)

Using `acquireRead`, `acquireWrite`, `releaseRead` and `releaseWrite` should be avoided for simple use cases relying on
methods like `readLock` and `writeLock` instead. `readLock` and `writeLock` automatically acquire and release the lock
thanks to the `Scope` construct. The program described below is a safer version of the program above and ensures we 
don't hold onto any resources once we are done using the reentrant lock.

```scala

val saferProgram: UIO[Unit] = for {
  lock <- TReentrantLock.make.commit
  f1   <- ZIO.scoped(lock.readLock *> ZIO.sleep(5.seconds) *> printLine("Powering down").orDie).fork
  f2   <- ZIO.scoped(lock.readLock *> lock.writeLock *> printLine("Huzzah, writes are mine").orDie).fork
  _    <- (f1 zip f2).join
} yield ()
```

---

## TRef

A `TRef[A]` is a mutable reference to an immutable value, which can participate in transactions in STM. The mutable reference can be retrieved and set from within transactions, with strong guarantees for atomicity, consistency, and isolation from other transactions.

`TRef` provides the low-level machinery to create transactions from modifications of STM memory.

## Create a TRef

Creating a `TRef` inside a transaction:

```scala

val createTRef: STM[Nothing, TRef[Int]] = TRef.make(10)
```

Or creating a `TRef` inside a transaction, and immediately committing the transaction, which allows you to store and pass along the reference.

```scala

val commitTRef: UIO[TRef[Int]] = TRef.makeCommit(10)
```

## Retrieve the value out of a TRef

Retrieving the value in a single transaction: 

```scala

val retrieveSingle: UIO[Int] = (for {
  tRef <- TRef.make(10)
  value <- tRef.get
} yield value).commit
```

Or on multiple transactional statements:

```scala

val retrieveMultiple: UIO[Int] = for {
  tRef <- TRef.makeCommit(10)
  value <- tRef.get.commit
} yield value
```

## Set a value to a TRef

Setting the value overwrites the existing content of a reference.

Setting the value in a single transaction:

```scala

val setSingle: UIO[Int] = (for {
  tRef <- TRef.make(10)
  _ <- tRef.set(20)
  nValue <- tRef.get
} yield nValue).commit
```

Or on multiple transactions:

```scala

val setMultiple: UIO[Int] = for {
  tRef <- TRef.makeCommit(10)
  nValue <- tRef.set(20).flatMap(_ => tRef.get).commit
} yield nValue
```

## Update the value of the TRef

The update function `A => A` allows computing a new value for the `TRef` using the old value.

Updating the value in a single transaction:

```scala

val updateSingle: UIO[Int] = (for {
  tRef <- TRef.make(10)
  nValue <- tRef.updateAndGet(_ + 20)
} yield nValue).commit
```

Or on multiple transactions:

```scala

val updateMultiple: UIO[Int] = for {
  tRef <- TRef.makeCommit(10)
  nValue <- tRef.updateAndGet(_ + 20).commit
} yield nValue
```

## Modify the value of the TRef

The modify function `A => (B, A): B` works similar to `update`, but allows extracting some information (the `B`) out of the update operation. 

Modify the value in a single transaction:

```scala

val modifySingle: UIO[(String, Int)] = (for {
  tRef <- TRef.make(10)
  mValue <- tRef.modify(v => ("Zee-Oh", v + 10))
  nValue <- tRef.get
} yield (mValue, nValue)).commit
```

Or on multiple transactions:

```scala

val modifyMultiple: UIO[(String, Int)] = for {
  tRef <- TRef.makeCommit(10)
  tuple2 <- tRef.modify(v => ("Zee-Oh", v + 10)).zip(tRef.get).commit
} yield tuple2
```

## Example usage

Here is a scenario where we use a `TRef` to hand-off a value between two `Fiber`s

```scala

def transfer(tSender: TRef[Int],
             tReceiver: TRef[Int],
             amount: Int): UIO[Int] = {
  STM.atomically {
    for {
      _ <- tSender.get.retryUntil(_ >= amount)
      _ <- tSender.update(_ - amount)
      nAmount <- tReceiver.updateAndGet(_ + amount)
    } yield nAmount
  }
}

val transferredMoney: UIO[String] = for {
  tSender <- TRef.makeCommit(50)
  tReceiver <- TRef.makeCommit(100)
  _ <- transfer(tSender, tReceiver, 50).fork
  _ <- tSender.get.retryUntil(_ == 0).commit
  tuple2 <- tSender.get.zip(tReceiver.get).commit
  (senderBalance, receiverBalance) = tuple2
} yield s"sender: $senderBalance & receiver: $receiverBalance"
```

In this example, we create and commit two transactional references for the sender and receiver to be able to extract their value. 
On the following step, we create an atomic transactional that updates both accounts only when there is sufficient balance available in the sender account. In the end, we fork to run asynchronously.
On the running fiber, we suspend until the sender balance suffers changes, in this case, to reach `zero`. Finally, we extract the new values out of the accounts and combine them in one result.

---

## TSemaphore

`TSemaphore` is a semaphore with transactional semantics that can be used to control access to a common resource. It 
holds a certain number of permits, and permits may be acquired or released.

## Create a TSemaphore
Creating a `TSemaphore` with 10 permits:
```scala

val tSemaphoreCreate: STM[Nothing, TSemaphore] = TSemaphore.make(10L)
```

## Acquire a permit
Acquiring a permit reduces the number of remaining permits that the `TSemaphore` contains. Acquiring a permit is done 
when a user wants to access a common shared resource:
```scala

val tSemaphoreAcq: STM[Nothing, TSemaphore] = for {
  tSem <- TSemaphore.make(2L)
  _    <- tSem.acquire
} yield tSem

tSemaphoreAcq.commit
```
Note that if you try to acquire a permit when there are no more remaining permits in the semaphore then execution will be blocked semantically until a permit is ready to be acquired. Note that semantic blocking does not block threads and the STM transaction will only be retried when a permit is released.

## Release a permit
Once you have finished accessing the shared resource, you must release your permit so other parties can access the 
shared resource:
```scala

val tSemaphoreRelease: STM[Nothing, TSemaphore] = for {
  tSem <- TSemaphore.make(1L)
  _    <- tSem.acquire
  _    <- tSem.release
} yield tSem

tSemaphoreRelease.commit
```

## Retrieve available permits
You can query for the remaining amount of permits in the TSemaphore by using `available`:
```scala

val tSemaphoreAvailable: STM[Nothing, Long] = for {
  tSem <- TSemaphore.make(2L)
  _    <- tSem.acquire
  cap  <- tSem.available
} yield cap

tSemaphoreAvailable.commit
```
The above code creates a TSemaphore with two permits and acquires one permit without releasing it. Here, `available`
will report that there is a single permit left.

## Execute an arbitrary STM action with automatic acquire and release
You can choose to execute any arbitrary STM action that requires acquiring and releasing permit on TSemaphore as part
of the same transaction. Rather than doing:
```scala

def yourSTMAction: STM[Nothing, Unit] = STM.unit

val tSemaphoreWithoutPermit: STM[Nothing, Unit] = 
  for {
    sem <- TSemaphore.make(1L)
    _   <- sem.acquire
    a   <- yourSTMAction
    _   <- sem.release
  } yield a

tSemaphoreWithoutPermit.commit
```
You can simply use `withPermit` instead:
```scala

val tSemaphoreWithPermit: IO[Nothing, Unit] =
  for {
    sem <- TSemaphore.make(1L).commit
    a   <- sem.withPermit(yourSTMAction.commit)
  } yield a
```

It is considered best practice to use `withPermit` over using an `acquire` and a `release` directly unless dealing with more complicated use cases that involve multiple STM actions where `acquire` is not at the start and `release` is not at the end of the STM transaction.

## Acquire and release multiple permits
It is possible to acquire and release multiple permits at a time using `acquireN` and `releaseN`:
```scala

val tSemaphoreAcquireNReleaseN: STM[Nothing, Boolean] = for {
  sem <- TSemaphore.make(3L)
  _   <- sem.acquireN(3L)
  cap <- sem.available 
  _   <- sem.releaseN(3L)
} yield cap == 0

tSemaphoreAcquireNReleaseN.commit
```

---

## TSet

A `TSet[A]` is a mutable set that can participate in transactions in STM.

## Create a TSet

Creating an empty `TSet`:

```scala

val emptyTSet: STM[Nothing, TSet[Int]] = TSet.empty[Int]
```

Or creating a `TSet` with specified values:

```scala

val specifiedValuesTSet: STM[Nothing, TSet[Int]] = TSet.make(1, 2, 3)
```

Alternatively, you can create a `TSet` by providing a collection of values:

```scala

val iterableTSet: STM[Nothing, TSet[Int]] = TSet.fromIterable(List(1, 2, 3))
```

In case there are duplicates provided, the last one is taken.

## Put an element to a TSet

The new element can be added to the set in the following way:

```scala

val putElem: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2)
  _    <- tSet.put(3)
} yield tSet).commit
```

In case the set already contains the element, no modification will happen.

## Remove an element from a TSet

The simplest way to remove an element from `TSet` is using `delete` method:

```scala

val deleteElem: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2, 3)
  _    <- tSet.delete(1)
} yield tSet).commit
```

Also, it is possible to remove every element that satisfies provided predicate:

```scala

val removedEvenElems: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  _    <- tSet.removeIf(_ % 2 == 0)
} yield tSet).commit
```

Or you can keep all the elements that match predicate function:

```scala

val retainedEvenElems: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  _    <- tSet.retainIf(_ % 2 == 0)
} yield tSet).commit
```

Note that `retainIf` and `removeIf` serve the same purpose as `filter` and `filterNot`. The reason for naming them differently was to emphasize a distinction in their nature. Namely, both `retainIf` and `removeIf` are destructive - calling them can modify the collection.

## Union of a TSet

Union of the sets A and B represents the set of elements belonging to set A or set B, or both.
Using `A union B` method modifies set `A`.

```scala

// unionTSet = {1, 2, 3, 4, 5, 6}
val unionTSet: UIO[TSet[Int]] = (for {
  tSetA <- TSet.make(1, 2, 3, 4)
  tSetB <- TSet.make(3, 4, 5, 6)
  _     <- tSetA.union(tSetB)
} yield tSetA).commit
```

## Intersection of a TSet

The intersection of the sets A and B is the set of elements belonging to both A and B.
Using `A intersect B` method modifies set `A`.

```scala

// intersectionTSet = {3, 4}
val intersectionTSet: UIO[TSet[Int]] = (for {
  tSetA <- TSet.make(1, 2, 3, 4)
  tSetB <- TSet.make(3, 4, 5, 6)
  _     <- tSetA.intersect(tSetB)
} yield tSetA).commit
```

## Difference of a TSet

The difference between sets A and B is the set containing elements of set A but not in B.
Using `A diff B` method modifies set `A`.

```scala

// diffTSet = {1, 2}
val diffTSet: UIO[TSet[Int]] = (for {
  tSetA <- TSet.make(1, 2, 3, 4)
  tSetB <- TSet.make(3, 4, 5, 6)
  _     <- tSetA.diff(tSetB)
} yield tSetA).commit
```

## Transform elements of a TSet

The transform function `A => A` allows computing a new value for every element in the set: 

```scala

val transformTSet: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  _    <- tSet.transform(a => a * a)
} yield tSet).commit
```

Note that it is possible to shrink a `TSet`:

```scala

val shrinkTSet: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  _    <- tSet.transform(_ => 1)
} yield tSet).commit
```
Resulting set in example above has only one element.

Note that `transform` serves the same purpose as `map`. The reason for naming it differently was to emphasize a distinction in its nature. Namely, `transform` is destructive - calling it can modify the collection.

The elements can be mapped effectfully via `transformSTM`:

```scala

val transformSTMTSet: UIO[TSet[Int]] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  _    <- tSet.transformSTM(a => STM.succeed(a * a))
} yield tSet).commit
```

Folds the elements of a `TSet` using the specified associative binary operator:

```scala

val foldTSet: UIO[Int] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  sum  <- tSet.fold(0)(_ + _)
} yield sum).commit
```

The elements can be folded effectfully via `foldSTM`:

```scala

val foldSTMTSet: UIO[Int] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  sum  <- tSet.foldSTM(0)((acc, el) => STM.succeed(acc + el))
} yield sum).commit
```

## Perform effects for TSet elements

`foreach` is used for performing an STM effect for each element in set:

```scala

val foreachTSet = (for {
  tSet   <- TSet.make(1, 2, 3, 4)
  tQueue <- TQueue.unbounded[Int]
  _      <- tSet.foreach(a => tQueue.offer(a).unit)
} yield tSet).commit
```

## Check TSet membership 

Checking whether the element is present in a `TSet`:

```scala

val tSetContainsElem: UIO[Boolean] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  res  <- tSet.contains(3)
} yield res).commit
```

## Convert TSet to a List

List of set elements can be obtained as follows:

```scala

val tSetToList: UIO[List[Int]] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  list <- tSet.toList
} yield list).commit
```

## Size of a TSet

Set's size can be obtained as follows:

```scala

val tSetSize: UIO[Int] = (for {
  tSet <- TSet.make(1, 2, 3, 4)
  size <- tSet.size
} yield size).commit
```

---

## Chunk

A `Chunk[A]` represents a chunk of values of type `A`. Chunks are usually backed by arrays, but expose a purely functional, safe interface to the underlying elements, and they become lazy on operations that would be costly with arrays, such as repeated concatenation. Like lists and arrays, Chunk is an ordered collection. 

## Why Chunk?
Arrays are fast and don’t box primitive values but due to `ClassTag` requirements and mutability they are painful to use and don't integrate well into functional code. ZIO chunks are backed by arrays so they also have zero boxing for primitives while providing an immutable interface and avoiding `ClassTag` requirements.

Let's get more details behind why Chunk was invented:

### Immutability
In Scala, there is no immutable data type that can efficiently represent primitive data types. There is Array, but Array is a mutable interface. The Array data type can efficiently represent primitives without boxing, but only by exposing some unsafe mutable methods like `update`.

### Ergonomic Design
Every time, when we create an array of generic types in Scala, we need a [ClassTag](https://www.scala-lang.org/api/current/scala/reflect/ClassTag.html) to provide runtime information about that generic type, which is very inconvenient and isn't ergonomic. It leads us to a very cumbersome API.

Chunk does not have the inconvenience of Array in Scala. **Chunk dispenses with the need to have ClassTags**. It utilizes a different approach to solve that problem. 

### High Performance
In addition to being an immutable array and zero boxing of Chunks that leads us to a high performant data type, Chunk has specialized operations for things like appending a single element or concatenating two Chunks together which have significantly higher performance than doing these same operations on the Array. Many Chunk methods have been handwritten to achieve better performance than their corresponding Array implementations in the Scala standard library.

Although Chunk is a common data type in ZIO, it exists primarily to support streaming use cases. 

When we are doing data streaming, a lot of times the source stream is a stream of bytes. Hence, internally we use a Chunk of bytes to represent that, so we don't have to box the bytes. Of course, it can be utilized for Chunks of Ints and many other types. Using Chunk is especially common when we are encoding and decoding at the level of streams. It is a very efficient, high-performance data type. 

## Operations

### Creating a Chunk

Creating empty `Chunk`:
```
val emptyChunk = Chunk.empty
```

Creating a `Chunk` with specified values:
```scala
val specifiedValuesChunk = Chunk(1,2,3)
// specifiedValuesChunk: Chunk[Int] = IndexedSeq(1, 2, 3)
```

Alternatively, we can create a `Chunk` by providing a collection of values:
```scala
val fromIterableChunk: Chunk[Int] = Chunk.fromIterable(List(1, 2, 3))
// fromIterableChunk: Chunk[Int] = IndexedSeq(1, 2, 3)
val fromArrayChunk: Chunk[Int] = Chunk.fromArray(Array(1, 2, 3))
// fromArrayChunk: Chunk[Int] = IndexedSeq(1, 2, 3)
```

Creating a `Chunk` using filling same n element into it:
```scala
val chunk: Chunk[Int] = Chunk.fill(3)(0)
// chunk: Chunk[Int] = IndexedSeq(0, 0, 0)
```

Creating a `Chunk` using unfold method by repeatedly applying the given function, as long as it returns Some:
```scala
val unfolded = Chunk.unfold(0)(n => if (n < 8) Some((n*2, n+2)) else None)
// unfolded: Chunk[Int] = IndexedSeq(0, 4, 8, 12)
```

### Concatenating chunk

`++` Returns the concatenation of this chunk with the specified chunk. For example:

```scala
Chunk(1,2,3) ++ Chunk(4,5,6)
// res0: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
```

### Collecting chunk

`collect` Returns a filtered, mapped subset of the elements of this chunk.
How to use `collect` function to cherry-pick all strings from Chunk[A]:

```scala
val collectChunk = Chunk("Hello ZIO", 1.5, "Hello ZIO NIO", 2.0, "Some string", 2.5)
// collectChunk: Chunk[Any] = IndexedSeq(
//   "Hello ZIO",
//   1.5,
//   "Hello ZIO NIO",
//   2.0,
//   "Some string",
//   2.5
// )

collectChunk.collect { case string: String => string }
// res1: Chunk[String] = IndexedSeq(
//   "Hello ZIO",
//   "Hello ZIO NIO",
//   "Some string"
// )
```
How to use `collect` function to cherry-pick all the digits from Chunk[A]:

```scala
collectChunk.collect { case digit: Double => digit }
// res2: Chunk[Double] = IndexedSeq(1.5, 2.0, 2.5)
```

`collectWhile` collects the elements (from left to right) until the predicate returns "false" for the first time:

```scala
Chunk("Sarah", "Bob", "Jane").collectWhile { case element if element != "Bob" => true }
// res3: Chunk[Boolean] = IndexedSeq(true)
```
or another example:

```scala
Chunk(9, 2, 5, 1, 6).collectWhile { case element if element >= 2 => true }
// res4: Chunk[Boolean] = IndexedSeq(true, true, true)
```
### Dropping chunk

`drop` drops the first `n` elements of the chunk:

```scala
Chunk("Sarah", "Bob", "Jane").drop(1)
// res5: Chunk[String] = IndexedSeq("Bob", "Jane")
```

`dropWhile` drops all elements so long as the predicate returns true:

```scala
Chunk(9, 2, 5, 1, 6).dropWhile(_ >= 2)
// res6: Chunk[Int] = IndexedSeq(1, 6)
```

### Comparing chunks

```scala
Chunk("A","B") == Chunk("A", "C")
// res7: Boolean = false
```

### Converting chunks

`toArray` converts the chunk into an Array.

```scala
Chunk(1,2,3).toArray
```

`toSeq` converts the chunk into `Seq`.

``` scala mdoc
Chunk(1,2,3).toSeq
```

---

## Introduction to ZIO Streams

The primary goal of a streaming library is to introduce **a high-level API that abstracts the mechanism of reading and writing operations using data sources and destinations**.

A streaming library helps us to concentrate on the business logic and separates us from low-level implementation details.

## Use Cases

There are lots of examples of streaming that people might not recognize, this is a common problem especially for beginners. A beginner might say "I don't need a streaming library. Why should I use that?". It's because they don't see streams. Once we use a streaming library, we start to see streams everywhere but until then we don't understand where they are. 

Before diving into ZIO Streams, let's list some use cases of a streaming solution and see why we would want to program in a streaming fashion:

- **Files** — Every time an old school API interacting with a file has very low-level operators like "Open a file, get me an InputStream, and a method to read the next chunk from that InputStream, and also another method to close the file". Although that is a very low-level imperative API, there is a way to see files as streams of bytes. 

- **Sockets** — Instead of working with low-level APIs, we can use streams to provide a stream-based implementation of server socket that hides the low-level implementation details of sockets. We could model socket communication as a function from a stream of bytes to a stream of bytes. We can view the input of that socket as being a stream, and its output as being another stream.

- **Event-Sourcing** — In these days and age, it is common to want to build event-sourced applications which work on events or messages in a queuing system like Kafka or AMQP systems and so forth. The foundation of this architecture is streaming. Also, they are useful when we want to do a lot of data analytics and so forth.  

- **UI Applications** — Streams are the foundation of almost every single modern UI application. Every time we click on something, under the hood that is an event. We can use low-level APIs like subscribing callbacks to the user events but also we can view those as streams of events. So we can model subscriptions as streams of events in UI applications. 

- **HTTP Server** —  An HTTP Server can be viewed as a stream. We have a stream of requests that are being transformed to a stream of responses; a function from a stream of bytes that go to a stream of bytes.

So streams are everywhere. We can see all of these different things as being streams. Everywhere we look we can find streams. Basically, all data-driven applications, almost all data-driven applications can benefit from streams. 

## Motivation

Assume, we would like to take a list of numbers and grab all the prime numbers and then do some more hard work on each of these prime numbers. We can do it using `ZIO.foreachPar` and `ZIO.filterPar` operators like this:

```scala

def isPrime(number: Int): Task[Boolean] = ZIO.succeed(???)
def moreHardWork(i: Int): Task[Boolean] = ZIO.succeed(???)

val numbers = 1 to 1000

for {
  primes <- ZIO.filterPar(numbers)(isPrime)
  _      <- ZIO.foreachPar(primes)(moreHardWork) @@ parallel(20)
} yield ()
```

This processes the list in parallel and filters all the prime numbers, then takes all the prime numbers and does some more hard work on them.

There are two problems with this example:

- **High Latency** — We are not getting any pipelining, we are doing batch processing. We need to wait for the entire list to be processed in the first step before we can continue to the next step. This can lead to a pretty severe loss of performance.

- **Limited Memory** — We need to keep the entire list in memory as we process it and this doesn't work if we are working with an infinite data stream.

With ZIO stream we can change this program to the following code:

```scala
def prime(number: Int): Task[(Boolean, Int)] = ZIO.succeed(???)

ZStream.fromIterable(numbers)
  .mapZIOParUnordered(20)(prime(_))
  .filter(_._1).map(_._2)
  .mapZIOParUnordered(20)(moreHardWork(_))
```

We converted the list of numbers using `ZStream.fromIterable` into a `ZStream`, then we mapped it in parallel, twenty items at a time, and then we performed the hard work problem, twenty items of a time. This is a pipeline, and this easily works for an infinite list.

One might ask, "Okay, I can get the pipelining by using fibers and queues. So why should I use ZIO streams?". It is extremely tempting to write up the pipeline look like this. We can create a bunch of queues and fibers, then we have fibers that copy information between the queues and perform the processing concurrently. It ends up something like this:

```scala
def writeToInput(q: Queue[Int]): Task[Unit]                            = ZIO.succeed(???)
def processBetweenQueues(from: Queue[Int], to: Queue[Int]): Task[Unit] = ZIO.succeed(???)
def printElements(q: Queue[Int]): Task[Unit]                           = ZIO.succeed(???)

for {
  input  <- Queue.bounded[Int](16)
  middle <- Queue.bounded[Int](16)
  output <- Queue.bounded[Int](16)
  _      <- writeToInput(input).fork
  _      <- processBetweenQueues(input, middle).fork
  _      <- processBetweenQueues(middle, output).fork
  _      <- printElements(output).fork
} yield ()
```

We created a bunch of queues for buffering source, destination elements, and intermediate results.

There are some problems with this solution. As fibers are low-level concurrency tools, using them to create a data pipeline is not straightforward. We need to handle interruptions properly. We should care about resources and prevent them to leak. We need to shutdown the pipeline in a right way by waiting for queues to be drained.

Although fibers are very efficient and more performant than threads. They are advanced concurrency tools. So it is better to avoid using them to do manual pipelining. Instead, we can use ZIO streams:

```scala
def generateElement: Task[Int]    = ZIO.succeed(???)
def process(i: Int): Task[Int]    = ZIO.succeed(???)
def printElem(i: Int): Task[Unit] = ZIO.succeed(???)

ZStream
  .repeatZIO(generateElement)
  .buffer(16)
  .mapZIO(process(_))
  .buffer(16)
  .mapZIO(process(_))
  .buffer(16)
  .tap(printElem(_))
```

We have a buffer in between each step. We performed our computations in between. This is everything we need to get that pipelining in the same fashion that it looked before.

## Why Streams?

ZIO stream has super compelling advantages of using high-level streams. ZIO solution to streaming solves a lot of common streaming pain points. It shines in the following topics:

### 1. High-level and Declarative

This means in a very short snippet of a fluent code we can solve very outrageously complicated problems with just a few simple lines.

### 2. Asynchronous and Non-blocking

They're reactive streams, they don't block threads. They're super-efficient and very scalable. We can minimize our application latency and increase its performance. We can avoid wasting precious thread resources by using non-blocking and asynchronous ZIO streams. 

### 3. Concurrency and Parallelism

Streams are concurrent. They have a lot of concurrent operators. All the operations on them are safe to use in presence of concurrency. And also just like ZIO gives us parallel operators with everything, there are lots of parallel operators. We can use the parallel version of operators, like `mapZIOPar`, `flatMapPar`.

Parallel operators allow us to fully saturate and utilize all CPU cores of our machine. If we need to do bulk processing on a lot of data and use all the cores on our machine, so we can speed up the process by using these parallel operators. 

### 4. Resource Safety

Resource safety is not a simple thing to guarantee. Assume when we have several streams, some of them are sockets and files, some of them are API calls and database queries. When we have all these streams, and we are transforming and combining them, and we are timing some out, and also some of them are doing concurrent merges; what happens when things go wrong in one part of that stream graph? ZIO streams provides the guarantee that it will never leak resources. 

So when streams have to be terminated for error or timeout or interruption reasons or whatever, ZIO will always safely shutdown and release the resources associated with that stream usage. 

We don't have to worry about resource management anymore. We can work at high-level and just declaratively describe our stream graph and then ZIO will handle the tricky job of executing that and taking care to make sure that no resources are leaked in an event of something bad happens or even just a timeout, or interruption, or just we are done with a result. So resources are always safely released without any leaks. 

### 5. High Performance and Efficiency

When we are doing an I/O job, the granularity of data is not at the level of a single byte. For example, we never read or write a single element from/to a file descriptor. We always use multiple elements. So when we are doing an I/O operation it is a poor practice to read/write element by element and this decreases the performance of our program. 

In order to achieve high efficiency, ZIO stream implicitly chunks everything, but it still presents us with a nice API that is at the level of every single element. So we can always deal with streams of individual elements even though behind-the-scenes ZIO is doing some chunking to make that performant. This is one of the tricks that enables ZIO streams to have such great performance. 

ZIO Streams are working at the level of chunks. Every time we are working with ZIO streams, we are also working with chunks implicitly. So there are no streams with individual elements. Streams always use chunks. Every time we pull an element out of a ZIO stream, we end up pulling a chunk of elements under the hood. 

### 6. Seamless Integration with ZIO

ZIO stream has a powerful seamless integrated support for ZIO. It uses `Scope`, `Schedule`, and any other powerful data types in ZIO. So we can stay within the same ecosystem and get all its significant benefits.

### 7. Back-Pressure

We get back-pressuring for free. With ZIO streams it is actually not a back-pressuring, but it is equivalent. In push-based streams like Akka Streams, streams are push-based; when an element comes in, it is pushed downward in the pipeline. That is what leads to the need for back-pressuring. Back-pressuring makes the push-based stream much more complicated than it needs to be. 

Push-based streams are good at splitting streams because we have one element, and we can push it to two different places. That is nice and elegant, but they're terrible at merging streams and that is because you end up needing to use queues, and then we run into a problem. In the case of using queues, we need back-pressuring, which leads to a complicated architecture. 

In ZIO when we merge streams, ZIO uses pull-based streams. They need minimal computation because we pull elements at the end of our data pipeline when needed. When the sink asks for one element, then that ripples all the way back through the very edges of the system. 

So when we pull one element at the end, no additional computation takes place until we pull the next element or decide that we are done pulling, and we close the stream. It causes the minimum amount of computation necessary to produce the result. 

Using the pull-based mechanism we have no producers, and it prevents producing more events than necessary. So ZIO streams does not need back-pressure even though it provides a form of that because it is lazy and on-demand and uses pull-based streams. 

So ZIO stream gives us the benefits of back-pressuring, but in a cleaner conceptual model that is very efficient and does not require all these levels of buffering.

### 8. Infinite Data using Finite Memory

Streams let us work on infinite data in a finite amount of memory. When we are writing streaming logic, we don't have to worry about how much data we are ultimately going to be processed.

That is because we are just building a workflow, a description of the processing. We are not manually loading up everything into memory, into a list, and then doing our processing on a list. That doesn't work very well because we can only fit a finite amount of memory into our computer at one time. 

ZIO streams enable us just concentrate on our business problem, and not on how much memory this program is going to consume. So we can write these computations that work over streams that are totally infinite but in a finite amount of memory and ZIO handles that for us.

Assume we have the following code. This is a snippet of a code that reads a file into a string and splits the string into new lines, then iterates over lines and prints them out. It is pretty simple and easy to read and also it is simple to understand:

```scala
for (line <- FileUtils.readFileToString(new File("file.txt")).split('\n'))
  println(line)
```

The only problem here is that if we run this code with a file that is very large which is bigger than our memory, that is not going to work. Instead, we can reach the same functionality, by using the stream API:

```scala
ZStream.fromFileName("file.txt")
  .via(ZPipeline.utf8Decode >>> ZPipeline.splitLines)
  .foreach(printLine(_))
```

By using ZIO streams, we do not care how big is a file, we just concentrate on the logic of our application.

## Core Abstractions

To define a stream workflow there are three core abstraction in ZIO stream; _Streams_, _Sinks_, and _Pipelines_:

1. **[ZStream](zstream/index.md)** — Streams act as _sources_ of values. We get elements from them. They produce values.

2. **[ZSink](zsink/index.md)** — Sinks act as _receptacles_ or _sinks_ for values. They consume values.

3. **[ZPipeline](zpipeline.md)** — Pipelines act as _transformers_ of values. They take individual values, and they transform or decode them. 

### Stream

The `ZStream` data type similar to the `ZIO` effect has `R`, `E`, and `A`. It has environment, error, and element type. 

The difference between the `ZIO` and `ZStream` is that:

- A `ZIO` effect will always succeed or fail. If it succeeds, it will succeed with a single element.

- A `ZStream` can succeed with zero or more elements. So we can have an _empty stream_. A `ZStream[R, E, A]` doesn't necessarily produce any `A`s, it produces zero or more `A`s. 

So, that is a big difference. There is no such thing as a non-empty `ZStream`. All `ZStreams` are empty, they can produce any number of `A`s, which could be an infinite number of `A`s. 

There is no way to check to see if a stream is empty or not, because that computation hasn't started. Streams are super lazy, so there is no way to say "Oh! does this stream contain anything?" No! We can't figure that out. We have to use it and try to do something with it, and then we are going to figure out whether it had something.

### Sink

The basic idea behind the `Sink` is that **it consumes values of some type, and then it ends up when it is done. When the sink is done, it produces the value of a different type**. 

Sinks are a bit like **parsers**; they consume some input, when they're done, they produce a value. Also, they are like **databases**; they read enough from input when they don't want anymore, they can produce some value or return unit.

Some sinks will produce nothing as their return type parameter is `Nothing`, which means that the sink is always going to accept more and more input; it is never ever going to be done. 

Just like Streams, sinks are super compositional. Sink's operators allow us to combine two sinks together or transform them. That allows us to generate a vast variety of sinks.

Streams and Sinks are duals in category theory. One produces values, and the other one consumes them. They are mere images of each other. They both have to exist. A streaming library cannot be complete unless it has streams and sinks. That is why ZIO has a sort of better design than FS2 because FS2 has a stream, but it doesn't have a sink. Its Sink is just faked. It doesn't actually have a real sink. ZIO has a real sink, and we can compose them to generate new sinks.

### Pipeline

With `Pipeline`s, we can transform streams from one type to another, in a **stateful fashion**, which is sometimes necessary when we are doing encoding and decoding. 

Pipeline is a transformer of element types. Pipeline accepts some element of type `A` and produces some element of type `B`, and it may fail along the way or use the environment. It just transforms elements from one type to another type in a stateful way. 

For example, we can write counter with pipelines. We take strings and then split them into lines, and then we take the lines, and we split them into words, and then we count them. 

Another common use case of pipelines is **writing codecs**. We can use them to decode the bytes into strings. We have a bunch of bytes, and we want to end up with a JSON and then once we are in JSON land we want to go from JSON to our user-defined data type. So, by writing a pipeline we can convert that JSON to our user-defined data type.

Pipelines can be thought of as **element transformers**. They transform elements of a stream:

1. We can take a pipeline, and we can stack it onto a stream to change the element type. For example, we have a Stream of `A`s, and a pipeline that goes from `A` to `B`, so we can take that pipeline from `A` to `B` and stack it on the stream to get back a stream of `B`s. 

2. Also, we can stack a pipeline onto the front of a sink to change the input element type. If some sink consumes `B`s, and we have a pipeline from `A` to `B` we can take that pipeline stack it onto the front of the sink and get back a new sink that consumes `A`s. 

Assume we are building the data pipeline, the elements come from the far left, and they end up on the far right. Events come from the stream, they end up on the sink, along the way they're transformed by pipelines. **Pipelines are the middle section of the pipe that keep on transforming those elements in a stateful way**.

---

## Installing ZIO Streams

In order to use ZIO Streaming, we need to add the required configuration in our SBT settings:

```scala
libraryDependencies += Seq(
  "dev.zio" %% "zio"         % "2.1.26" % Test
  "dev.zio" %% "zio-streams" % "2.1.26" % Test
)
```

---

## SubscriptionRef

A `SubscriptionRef[A]` is a `Ref` that lets us subscribe to receive the current value along with all changes to that value.

```scala

trait SubscriptionRef[A] extends Ref.Synchronized[A] {
  def changes: ZStream[Any, Nothing, A]
}
```

We can use all the normal methods on `Ref.Synchronized` to `get`, `set`, or `modify` the current value.

The `changes` stream can be consumed to observe the current value as well as all changes to that value. Since `changes` is just a description of a stream, each time we run the stream we will observe the current value as of that point in time as well as all changes after that.

To create a `SubscriptionRef` you can use the `make` constructor, which makes a new `SubscriptionRef` with the specified initial value.

```scala
object SubscriptionRef {
  def make[A](a: A): UIO[SubscriptionRef[A]] =
    ???
}
```

A `SubscriptionRef` can be extremely useful to model some shared state where one or more observers must perform some action for all changes in that shared state. For example, in a functional reactive programming context the value of the `SubscriptionRef` might represent one part of the application state and each observer would need to update various user interface elements based on changes in that state.

To see how this works, let's create a simple example where a "server" repeatedly updates a value that is observed by multiple "clients".

```scala
def server(ref: Ref[Long]): UIO[Nothing] =
  ref.update(_ + 1).forever
```

Notice that `server` just takes a `Ref` and does not need to know anything about `SubscriptionRef`. From its perspective it is just updating a value.

```scala
def client(changes: ZStream[Any, Nothing, Long]): UIO[Chunk[Long]] =
  for {
    n     <- Random.nextLongBetween(1, 200)
    chunk <- changes.take(n).runCollect
  } yield chunk
```

Similarly `client` just takes a `ZStream` of values and does not have to know anything about the source of these values. In this case we will simply observe a fixed number of values.

To wire everything together, we start the server, then start multiple instances of the client in parallel, and finally shut down the server when we are done. We also actually create the `SubscriptionRef` here.

```scala
for {
  subscriptionRef <- SubscriptionRef.make(0L)
  server          <- server(subscriptionRef).fork
  chunks          <- ZIO.collectAllPar(List.fill(100)(client(subscriptionRef.changes)))
  _               <- server.interrupt
  _               <- ZIO.foreach(chunks)(chunk => Console.printLine(chunk))
} yield ()
```

This will ensure that each client observes the current value when it starts and all changes to the value after that.

Since the changes are just streams it is also easy to build much more complex programs using all the stream operators we are accustomed to. For example, we can transform these streams, filter them, or merge them with other streams.

---

## Channel Interruption

We can interrupt a channel using the `ZChannel.interruptWhen` operator. It takes a ZIO effect that will be evaluated, if it finishes before the channel is closed, it will interrupt the channel, and the terminal value of the returned channel will be the success value of the effect:

```scala

def randomNumbers: ZChannel[Any, Any, Any, Any, Nothing, Int, Nothing] =
  ZChannel
    .fromZIO(Random.nextIntBounded(100))
    .flatMap(ZChannel.write) *>
    ZChannel.fromZIO(ZIO.sleep(1.second)) *> randomNumbers

randomNumbers.interruptWhen(ZIO.sleep(3.seconds).as("Done!")).runCollect.debug
// One output: (Chunk(84,57,70),Done!)
```

Another version of `interruptWhen` takes a `Promise` as an argument. It will interrupt the channel when the promise is fulfilled:

```scala

for {
  p <- Promise.make[Nothing, Unit]
  f <- randomNumbers
    .interruptWhen(p)
    .mapOutZIO(e => Console.printLine(e))
    .runDrain
    .fork
  _ <- p.succeed(()).delay(5.seconds)
  _ <- f.join
} yield ()

// Output:
// 74
// 60
// 52
// 52
// 79
```

---

## Channel Operations

## Piping

The values from the output port of the first channel are passed to the input port of the second channel when we pipe a channel to another channel:

```scala

(ZChannel.writeAll(1,2,3) >>> (ZChannel.read[Int] <*> ZChannel.read[Int])).runCollect.debug
// Output: (Chunk(),(1,2))
```

## Sequencing

In order to sequence channels, we can use the `ZChannel#flatMap` operator. When we use the `flatMap` operator, we have the ability to chain two channels together. After the first channel is finished, we can create a new channel based on the terminal value of the first channel:

```scala

ZChannel
  .fromZIO(
    Console.readLine("Please enter a number: ").map(_.toInt)
  )
  .flatMap {
    case n if n < 0 => ZChannel.fail("Number must be positive")
    case n          => ZChannel.writeAll((0 to n): _*)
  }
  .runCollect
  .debug
// Sample Output:
// Please enter a number: 5
// (Chunk(0,1,2,3,4,5),())
```

## Concatenating

Suppose there is a channel that creates a new channel for each element of the outer channel and emits them to the output port. We can use `concatOut` to concatenate all the inner channels into a single channel:

```scala

ZChannel
  .writeAll("a", "b", "c")
  .mapOut { l =>
    ZChannel.writeAll((1 to 3).map(i => s"$l$i"):_*) 
  }
  .concatOut
  .runCollect
  .debug
// Output: (Chunk(a1,a2,a3,b1,b2,b3,c1,c2,c3),())
```

We can do the same with `ZChannel.concatAll`:

```scala

ZChannel
  .concatAll(
    ZChannel
      .writeAll("a", "b", "c")
      .mapOut { l =>
        ZChannel.writeAll((1 to 3).map(i => s"$l$i"): _*)
      }
  )
  .runCollect
  .debug
  
// Output: (Chunk(a1,a2,a3,b1,b2,b3,c1,c2,c3),())
```

## Zipping

We have two categories of `zip` operators: ordinary `zipXYZ` operators which run sequentially, and parallel `zipXYZ` operators which run in parallel.

1. `zip`/`<*>` operator:

```scala

val first = ZChannel.write(1,2,3) *> ZChannel.succeed("Done!")
val second = ZChannel.write(4,5,6) *> ZChannel.succeed("Bye!")

(first <*> second).runCollect.debug
// Output: (Chunk((1,2,3),(4,5,6)),(Done!,Bye!))
```

2. `zipRight`/`*>` operator:

```scala
(first *> second).runCollect.debug
```

3. `zipLeft`/`<*` operator:

```scala
(first <* second).runCollect.debug
```

## Mapping

### Mapping The Terminal Done Value (`OutDone`)

The ordinary `map` operator is used to map the done value of a channel:

```scala

ZChannel.writeAll(1, 2, 3).map(_ => 5).runCollect.debug 
// (Chunk(1,2,3),5)
```

### Mapping The Done Value of The Input Port (`InDone`)

To map the done value of the input port, we use the `contramap` operator:

```scala

(ZChannel.succeed("5") >>>
  ZChannel
    .readWith(
      (i: Int) => ZChannel.write(ZChannel.write(i)),
      (_: Any) => ZChannel.unit,
      (d: Int) => ZChannel.succeed(d * 2)
    )
    .contramap[String](_.toInt)).runCollect.debug
// Output: (Chunk(),(10))
```

### Mapping The Error Value of The Output Port (`OutErr`)

To map the failure value of a channel, we use the `mapError` operator:

```scala

val channel =
  ZChannel
    .fromZIO(Console.readLine("Please enter you name: "))
    .mapError(_.toString)
```

### Mapping The Output Elements of a Channel (`OutElem`)

To map the output elements of a channel, we use the `mapOutput` operator:

```scala

ZChannel.writeAll(1,2,3).mapOut(_ * 2).runCollect.debug
// Output: (Chunk(2,4,6),())
```

### Mapping The Input Elements of a Channel (`InElem`)

To map the input elements of a channel, we use the `contramapIn` operator:

```scala

(ZChannel.write("123") >>> ZChannel.read[Int].contramapIn[String](_.toInt * 2)).runCollect.debug
// Output: (Chunk(),(246))
```

## Merging

Merge operators are used to merging multiple channels into a single channel. They are used to combine the output port of channels concurrently. Every time any of the channels produces a value, the output port of the resulting channel will produce a value.

Assume we have the following channel:

```scala

def iterate(
    from: Int,
    to: Int
): ZChannel[Any, Any, Any, Any, Nothing, Int, Unit] =
  if (from <= to)
    ZChannel.write(from) *>
      ZChannel.fromZIO(
        Random
          .nextLongBounded(1000)
          .flatMap(delay => ZIO.sleep(Duration.fromMillis(delay)))
      ) *> iterate(from + 1, to)
  else ZChannel.unit
```

Now let's merge some channels:

```scala

ZChannel
  .mergeAllUnbounded(
    ZChannel.writeAll(
      iterate(1, 3),
      iterate(4, 6),
      iterate(6, 9)
    )
  )
  .mapOutZIO(i => Console.print(i + " "))
  .runDrain
// Sample output: 1 4 6 7 8 2 3 5 6 9
```

The `ZChannel.mergeAllUnbounded` uses the maximum buffer size, which is `Int.MaxValue` by default. This means that if we use this operator for long-running channels, which produce a lot of values, it can cause the program to run out of memory.

We have another operator called `ZChannel.mergeAll`, which allows us to specify the buffer size, the concurrency level, and also the strategy for merging the channels.

Note that if we want to merge channels sequentially, we can use the `zip` or `flatMap` operators:

```scala

(iterate(1, 3) <*> iterate(4, 6) <*> iterate(6, 9)).runCollect.debug
// Output: (Chunk(1,2,3,4,5,6,7,8,9),())
```

## Collecting

1. `collectElements` collects all the elements of the channel along with its done value as a tuple and returns a new channel with a terminal value of that tuple:

```scala

ZChannel.writeAll(1,2,3,4,5)
  .collectElements
  .runCollect
  .debug
// Output: (Chunk(),(Chunk(1,2,3,4,5),()))
```

2. `emitCollect` is like the `collectElements` operator, but it emits the result of the collection to the output port of the new channel:

```scala

ZChannel.writeAll(1,2,3,4,5)
  .emitCollect
  .runCollect
  .debug
// Output: (Chunk((Chunk(1,2,3,4,5),())),())
```

## Converting

We can convert a channel to other data types using the `ZChannel.toXYZ` methods:

- `ZChannel#toStream`
- `ZChannel#toPipeline`
- `ZChannel#toSink`
- `ZChannel#toPull`
- `ZChannel#toQueue`

## concatMap

`concatMap` is a combination of two operators: mapping and concatenation. Using this operator, we can map every emitted element of a channel (outer channel) to a new channel (inner channels), and then concatenate all the inner channels into a single channel. The concatenation is done **sequentially**, so we use this operator when the order of the elements is important:

```scala

ZChannel
  .writeAll("a", "b", "c")
  .concatMap { l =>
    def inner(from: Int, to: Int): ZChannel[Any, Any, Any, Any, Nothing, String, Unit] =
      if (from <= to) ZChannel.write(s"$l$from") *> inner(from + 1, to)
      else ZChannel.unit  
    inner(0, 5)
  }
  .runCollect
  .debug
// Output: (Chunk(a0,a1,a2,a3,a4,a5,b0,b1,b2,b3,b4,b5,c0,c1,c2,c3,c4,c5),())
```

In the above example, we create a new channel for every element of the outer channel. The new inner channel is responsible for emitting from zero to five with the label of the outer channel. When an inner channel is done, it moves to the next inner channel sequentially. There is a similar operator called `mergeMap` that works in parallel and doesn't preserve the order of the elements.

## mergeMap

`mergeMap` is a combination of two operators: mapping and merging. Using this operator, we can map every emitted element of a channel (outer channel) to a new channel (inner channel), and then run all the inner channels in parallel and merge them into a single channel. The merge operation is done **in parallel**, so we use this operator when the order of the elements is not important, and we want to process all inner channels in parallel:

```scala

ZChannel
  .writeAll("a", "b", "c")
  .mergeMap(8, 1, MergeStrategy.BackPressure) { l =>
    def inner(
        from: Int,
        to: Int
    ): ZChannel[Any, Any, Any, Any, Nothing, String, Unit] =
      if (from <= to) ZChannel.write(s"$l$from") *> inner(from + 1, to)
      else ZChannel.unit
    inner(0, 5)
  }
  .runCollect
  .debug
// Non-deterministic output: (Chunk(a0,a1,a2,b0,b1,b2,b3,c0,b4,c1,a3,c2,b5,a4,c3,c4,a5,c5),())
```

## collect

`collect` is a combination of two operations: filtering and mapping. Using this operator, we can filter the elements of a channel using a partial function, and then map the filtered elements:

```scala

ZChannel
  .writeAll((1 to 10): _*)
  .collect { case i if i % 3 == 0 => i * 2 }
  .runCollect
  .debug
// Output: (Chunk(6,12,18),())
```

---

## Composing Channels

We can write more complex channels by using `read` operators and composing them recursively. 

Let's try some examples:

## Simple Echo Channel

Assume we want to read a value from the input port and then print it to the console, we can use the `ZChannel.readWith` operator to do this:

```scala

val producer = 
  ZChannel.write(1)
  
val consumer = 
  ZChannel.readWith(
    (i: Int) => ZChannel.fromZIO(Console.printLine("Consumed: " + i)),
    (_: Any) => ZChannel.unit,
    (_: Any) => ZChannel.unit
  )

(producer >>> consumer).run
// Output:
// Consumed: 1
```

## Echo Channel Forever

We can also recursively compose channels to create a more complex channel. In the following example, we are going to continuously read values from the console and write them back to the console:

```scala

object MainApp extends ZIOAppDefault {
  val producer: ZChannel[Any, Any, Any, Any, IOException, String, Nothing] =
    ZChannel
      .fromZIO(Console.readLine("Please enter some text: "))
      .flatMap(i => ZChannel.write(i) *> producer)

  val consumer: ZChannel[Any, Any, String, Any, IOException, Nothing, Unit] =
    ZChannel.readWith(
      (i: String) => i match {
        case "exit" => ZChannel.unit
        case _ => ZChannel.fromZIO(Console.printLine("Consumed: " + i)) *> consumer
      },
      (_: Any) => ZChannel.unit,
      (_: Any) => ZChannel.unit
    )

  def run = (producer >>> consumer).run
}

// Output:
// Please enter some text: Foo
// Consumed: Foo
// Please enter some text: Bar
// Consumed: Bar
// Please enter some text: Baz
// Consumed: Baz
// Please enter some text: exit
```

## Replicator Channel

In this example, we are going to create a channel that replicates any input values to the output port.

```scala

object MainApp extends ZIOAppDefault {
  lazy val doubler: ZChannel[Any, Any, Int, Any, Nothing, Int, Unit] =
    ZChannel.readWith(
      (i: Int) => ZChannel.writeAll(i, i) *> doubler,
      (_: Any) => ZChannel.unit,
      (_: Any) => ZChannel.unit
    )
  def run = (ZChannel.writeAll(1,2,3,4,5) >>> doubler).runCollect.debug
}
// Output:
//   (Chunk(1,1,2,2,3,3,4,4,5,5),())
```

## Counter Channel

We can also use `Ref` to create a channel with an updatable state. For example, we can create a channel that keeps track number of all the values that it has read and finally returns it as the done value:

```scala

object MainApp extends ZIOAppDefault {
  val counter = {
      def count(c: Int): ZChannel[Any, Any, Int, Any, String, Int, Int] =
        ZChannel.readWith(
          (i: Int) => ZChannel.write(i) *> count(c + 1),
          (_: Any) => ZChannel.fail("error"),
          (_: Any) => ZChannel.succeed(c)
        )

      count(0)
    }

  def run = (ZChannel.writeAll(1, 2, 3, 4, 5) >>> counter).runCollect.debug
}

// Output:
// (Chunk(1,2,3,4,5), 5)
```

## Dedupe Channel

Sometimes we want to remove duplicate values from the input port. We need to have a state that keeps track of the values that have been seen. So if a value is seen for the first time, we can write it to the output port. If a value is duplicated, we can ignore it:

```scala

object MainApp extends ZIOAppDefault {
  val dedup =
    ZChannel.fromZIO(Ref.make[HashSet[Int]](HashSet.empty)).flatMap { ref =>
      lazy val inner: ZChannel[Any, Any, Int, Any, Nothing, Int, Unit] =
        ZChannel.readWith(
          (i: Int) =>
            ZChannel
              .fromZIO(ref.modify(s => (s contains i, s incl i)))
              .flatMap {
                case true  => ZChannel.unit
                case false => ZChannel.write(i)
              } *> inner,
          (_: Any) => ZChannel.unit,
          (_: Any) => ZChannel.unit
        )
      inner
    }

  def run =
    (ZChannel.writeAll(1, 2, 2, 3, 3, 4, 2, 5, 5) >>> dedup).runCollect.debug
}
// Output:
// (Chunk(1,2,3,4,5),())
```

## Buffered Channel

With help of `ZChannel.buffer` or `ZChannel.bufferChunk`, we can create a channel backed by a buffer.
- If the buffer is full, the channel puts the values in the buffer to the output port.
- If the buffer is empty, the channel reads the value from the input port and puts it in the output port.

Assume we have a channel written as follows:

```scala

def buffered(input: Int) =
  ZChannel
    .fromZIO(Ref.make(input))
    .flatMap { ref =>
      ZChannel.buffer[Any, Int, Unit](
        0,
        i => if (i == 0) true else false,
        ref
      )
    }
```

If the buffer is empty (zero value), the `buffered` channel passes the `1` to the output port:

```scala
(ZChannel.write(1) >>> buffered(0)).runCollect.debug
```

If the buffer is full, the channel puts the value from the buffer to the output port:

```scala
(ZChannel.write(1) >>> buffered(0)).runCollect.debug
```

---

## Creating Channels

`ZChannel` have several constructors and also built-in channels, where suitable to create more complex channels.

Without further ado, let's learn them one by one:

## `ZChannel.succeed`

Creates a channel that succeeds with a given done value, e.g. `ZChannel.succeed(42)`:

```scala

val channel: ZChannel[Any, Any, Any, Any, Nothing, Nothing, Int] = 
  ZChannel.succeed(42)
```

This channel doesn't produce any data but succeeds with a done value of type `Int`. Let's try to `runCollect` this channel and see what happens:

```scala
channel.runCollect.debug
```

The output of the `runCollect` operation is a tuple of two elements: the first is a chunk of data that the channel produced, and the second is the done value. Because this channel doesn't produce any data, the first element is an empty chunk, but it has a 42 as the done value in the second element.

## `ZChannel.fail`

Creates a channel that fails with a given error, e.g. `ZChannel.fail(new Exception("error"))`:

```scala

val channel: ZChannel[Any, Any, Any, Any, Exception, Nothing, Nothing] = 
  ZChannel.fail(new Exception("error"))
```

## `ZChannel.write*`

Create a channel that writes given elements to the output port:

```scala

ZChannel.write(1).runCollect.debug
// Output: (Chunk(1),()) 

ZChannel.writeAll(1, 2, 3).runCollect.debug
// Output: (Chunk(1,2,3),()) 

ZChannel.writeChunk(Chunk(1, 2, 3)).runCollect.debug
// Output: (Chunk(1,2,3),())
```

## `ZChannel.read*`

Create a channel that reads elements from the input port and returns that as a done value:

Let's start with the simplest read operation, `ZChannel.read`:

```scala

val read: ZChannel[Any, Any, Int, Any, None.type, Nothing, Int] = 
  ZChannel.read[Int]
```

To test this channel, we can create a writer channel and then pipe that to the reader channel:

```scala

val read = ZChannel.read[Int] 

(ZChannel.write(1) >>> read).runCollect.debug
// Output: (Chunk(0),1)
```

In the above example, the writer channel writes the value 1 to the output port, and the reader channel reads the value from the input port and then returns it as a done value.

If we compose multiple read operations, we can read more values from the input port:

```scala

val read = ZChannel.read[Int] 

(ZChannel.writeAll(1, 2, 3) >>> (read *> read)).runCollect.debug
// Output: (Chunk(),2) 

(ZChannel.writeAll(1, 2, 3) >>> (read *> read *> read)).runCollect.debug
// Output: (Chunk(),3)
```

Another useful read operation is `ZChannel.readWith`. Using this operator, after reading a value from the input port, instead of returning it as a done value, we have the ability to pass the input value to another channel.

---

## Introduction To ZChannels

Channels are the nexus of communications, which support both reading and writing. They allow us to have a unidirectional flow of data from the input to the output.

A `ZChannel[-Env, -InErr, -InElem, -InDone, +OutErr, +OutElem, +OutDone]` requires some environment `Env` and have two main operations:

- It can read some data `InElem` from the input port, and finally can terminate with a done value of type `InDone`. If the read operation fails, the channel will terminate with an error of type `InErr`.

- It can write some data `OutElem` to the output port, and finally terminate the channel with a done value of type `OutDone`. If the write operation fails, the channel will terminate with an error of type `OutErr`.

They are an underlying abstraction for `ZStream`, `ZPipeline`, and `ZSink`. In ZIO Streams, we call the input port `ZStream`, the output port `ZSink`, and the middle part `ZPipeline`:

- A `Channel` can write some elements to the _output_, and it can terminate with some sort of _done_ value. The `Channel` uses this _done_ value to notify the downstream `Channel` that its emission of elements is finished. In ZIO, the `ZStream` is encoded as an output side of the `Channel`.

- A `Channel` can read from its input, and it can also terminate with some sort of _done_ value, which is an upstream result. So a `Channel` has the _input type_, and the _input done type_. The `Channel` uses this _done_ value to determine when the upstream `Channel` finishes its emission. In ZIO, the `ZSink` is encoded as an input side of the `Channel`.

- A `Channel` can read from its input, do some transformation on the elements, and write to its output. In ZIO, the `ZPipeline` is encoded as a middle part of both sides of the `Channel`. Pipelines accept a stream as input and return the transformed stream as output.

:::caution

`ZChannel` is an underlying abstraction. So we do not usually need to use it directly. So if you are learning ZIO Streams, we recommend you to focus on `ZStream`, `ZPipeline`, and `ZSink` data types.
:::

Let's take a look at how `ZStream`, `ZPipeline` and `ZSink` are defined using `ZChannel`:

```scala
trait ZChannel[-Env, -InErr, -InElem, -InDone, +OutErr, +OutElem, +OutDone] 

case class ZStream[-R, +E, +A] (
  val channel: ZChannel[R, Any, Any, Any, E, Chunk[A], Any]
)

case class ZSink[-R, +E, -In, +L, +Z] (
  val channel: ZChannel[R, ZNothing, Chunk[In], Any, E, Chunk[L], Z]
)

case class ZPipeline[-R, +E, -In, +Out] (
  val channel: ZChannel[R, ZNothing, Chunk[In], Any, E, Chunk[Out], Any]
)
```

So we can say that:

- `ZStream[R, E, A]` is a channel that requires an environment `R`, emits elements of type `Chunk[A]`, and terminates, if at all, with either an error of type `E` or a done value of type `Any`.

- `ZPipeline[R, Err, In, Out]` is a channel that uses `R` as its environment, consumes `Chunk[In]` from its input port, and produces `Chunk[Out]` to its output port.

- `ZSink[R, E, In, L , Z]` is a channel that uses `R` as its environment, consumes `Chunk[In]` from its input port, and produces `Chunk[L]` to its output port as its leftovers, and can terminate with a success value of type `Z` or can terminate with a failure of type `E`.

![ZIO Streams 2.x](/img/assets/zio-streams-2.x.svg)

Channels compose in a variety of ways:

- **Piping**— One channel can be piped to another channel, assuming the input type of the second is the same as the output type of the first. We can pipe data from a channel that reads from the input port to a channel that writes to the output port, by using the `pipeTo` or `>>>` operator.

- **Sequencing**— The terminal value of one channel can be used to create another channel, and both the first channel and the function that makes the second channel can be composed into a channel. We use the `ZChannel#flatMap` to sequence the channels.

- **Concating**— The output of one channel can be used to create other channels, which are all concatenated together. The first channel and the function that makes the other channels can be composed into a channel. We use `ZChannel#concat*` operators to do this.

Finally, we can run a channel by using the `ZChannel#run*` operators.

---

## Running a Channel

To run a channel, we can use the `ZChannel.runXYZ` methods:

- `ZChannel#run`— The `run` method is the simplest way to run a channel. It only runs a channel that doesn't read any input or write any output.
- `ZChannel#runCollect`— It will run a channel and collects the output and finally returns it along with the done value of the channel.
- `ZChannel#runDrain`— It will run a channel and ignore any emitted output.
- `ZChannel#runScoped`— Using this method, we can run a channel in a scope. So all the finalizers of the scope will be run before the channel is closed.

---

## ZPipeline

## Introduction

A `ZPipeline[-Env, +Err, -In, +Out]` is a stream transformer. Pipelines accept a stream as input and return the transformed stream as output.

ZPipelines can be thought of as a recipe for calling a bunch of methods on a source stream to yield a new (transformed) stream. A nice mental model is the following type alias:

```scala
type ZPipeline[Env, Err, In, Out] = ZStream[Env, Err, In] => ZStream[Env, Err, Out]
```

There is no fundamental requirement for pipelines to exist, because everything pipelines do can be done directly on a stream. However, because pipelines separate the stream transformation from the source stream itself, it becomes possible to abstract over stream transformations at the level of values, creating, storing, and passing around reusable transformation pipelines that can be applied to many different streams. 

## Creation

### From Function

By using `ZPipeline.map` we convert a function into a pipeline. Let's create a pipeline which converts a stream of strings into a stream of characters:

```scala
val chars = 
 ZPipeline.map[String, Chunk[Char]](s => Chunk.fromArray(s.toArray)) >>>
   ZPipeline.mapChunks[Chunk[Char], Char](_.flatten)
```

There is also a `ZPipeline.mapZIO` which is an effectful version of this constructor.

### From Custom Channels

For stateful transformations that can't be expressed with `map` or `mapZIO`, you can build pipelines directly from `ZChannel` using `ZChannel.readWithCause`. Here is a pipeline that pairs each element with its predecessor:

```scala

def pairwise[A]: ZPipeline[Any, Nothing, A, (A, A)] =
  ZPipeline.fromChannel(pairwiseGo[A](None))

def pairwiseGo[A](
  prev: Option[A]
): ZChannel[Any, ZNothing, Chunk[A], Any, ZNothing, Chunk[(A, A)], Any] =
  ZChannel.readWithCause(
    (in: Chunk[A]) => {
      val buf  = Chunk.newBuilder[(A, A)]
      var last = prev
      in.foreach { a =>
        last.foreach(p => buf += ((p, a)))
        last = Some(a)
      }
      val out = buf.result()
      (if (out.nonEmpty) ZChannel.write(out) else ZChannel.unit) *> pairwiseGo(last)
    },
    (err: Cause[ZNothing]) => ZChannel.refailCause(err),
    (_: Any) => ZChannel.unit
  )
```

The three-case `readWithCause` pattern handles:
- **Case 1 (in):** Process incoming chunk, emit outputs, recurse to read next chunk
- **Case 2 (err):** Handle errors from upstream (usually propagate with `refailCause`)
- **Case 3 (done):** Handle stream completion (finalize any pending state)

:::note
Use `ZNothing` (not `Nothing`) as the error type in channel signatures. `ZNothing` is ZIO's abstract bottom type that avoids Scala type inference issues when composing channels with `readWithCause`.
:::

:::info
**Why channels instead of functions?** Pipelines are channels because they need to compose seamlessly with sinks and other pipelines. A function-based pipeline couldn't be composed with a sink—the channel abstraction provides a unified interface for all streaming components.
:::

## Built-in Pipelines

### Identity

The identity pipeline passes elements through without any modification:

```scala
ZStream(1,2,3).via(ZPipeline.identity[Int])
// Ouput: 1, 2, 3
```

### Splitting

**ZPipeline.splitOn** — A pipeline that splits strings on a delimiter:

```scala
ZStream(
  "5-6-7-8", 
  "-9-10-1",
  "1-12-13"
)
  .via(ZPipeline.splitOn("-"))
  .map(_.toInt)
// Ouput: 5, 6, 7, 8, 9, 10, 11, 12, 13
```

**ZPipeline.splitLines** — A pipeline that splits strings on newlines. Handles both Windows newlines (`\r\n`) and UNIX newlines (`\n`):

```scala
ZStream("This is the first line.\nSecond line.\nAnd the last line.")
  .via(ZPipeline.splitLines)
// Output: "This is the first line.", "Second line.", "And the last line."
```

**ZPipeline.splitOnChunk** — A pipeline that splits elements on a delimiter and transforms the splits into desired output:

```scala
ZStream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
  .via(ZPipeline.splitOnChunk(Chunk(4, 5, 6)))
// Output: Chunk(1, 2, 3), Chunk(7, 8, 9, 10)
```

### Dropping

**ZPipeline.dropWhile** — Creates a pipeline that starts consuming values as soon as one fails the given predicate:

```scala
ZStream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
  .via(ZPipeline.dropWhile((x: Int) => x <= 5))
// Output: 6, 7, 8, 9, 10
```

The `ZPipeline` also has `dropWhileZIO` which takes an effectful predicate `p: I => ZIO[R, E, Boolean]`.

### Prepending

The `ZPipeline.prepend` creates a pipeline that emits the provided chunks before emitting any other values:

```scala
ZStream(2, 3, 4).via(
  ZPipeline.prepend(Chunk(0, 1))
)
// Output: 0, 1, 2, 3, 4
```

### Compression

**ZPipeline.deflate** — The `deflate` pipeline compresses a stream of bytes as specified by [RFC 1951](https://tools.ietf.org/html/rfc1951).

```scala

def compressWithDeflate(clearText: ZStream[Any, Nothing, Byte]): ZStream[Any, Nothing, Byte] = {
  val bufferSize: Int = 64 * 1024 // Internal buffer size. Few times bigger than upstream chunks should work well.
  val noWrap: Boolean = false // For HTTP Content-Encoding should be false.
  val level: CompressionLevel = CompressionLevel.DefaultCompression
  val strategy: CompressionStrategy = CompressionStrategy.DefaultStrategy
  val flushMode: FlushMode = FlushMode.NoFlush
  clearText.via(deflate(bufferSize, noWrap, level, strategy, flushMode))
}

def deflateWithDefaultParameters(clearText: ZStream[Any, Nothing, Byte]): ZStream[Any, Nothing, Byte] =
  clearText.via(deflate())
```

**ZPipeline.gzip** — The `gzip` pipeline compresses a stream of bytes as using _gzip_ method:

```scala

ZStream
  .fromFileName("file.txt")
  .via(
    ZPipeline.gzip(
      bufferSize = 64 * 1024,
      level = CompressionLevel.DefaultCompression,
      strategy = CompressionStrategy.DefaultStrategy,
      flushMode = FlushMode.NoFlush
    )
  )
  .run(
    ZSink.fromFileName("file.gz")
  )
```

### Decompression

If we are reading `Content-Encoding: deflate`, `Content-Encoding: gzip` streams, or other such streams of compressed data, the following pipelines can be helpful. Both decompression methods will fail with `CompressionException` when input wasn't properly compressed:

**ZPipeline.inflate** — This pipeline allows decompressing stream of _deflated_ inputs, according to [RFC 1951](https://tools.ietf.org/html/rfc1951).

```scala

def decompressDeflated(deflated: ZStream[Any, Nothing, Byte]): ZStream[Any, CompressionException, Byte] = {
  val bufferSize: Int = 64 * 1024 // Internal buffer size. Few times bigger than upstream chunks should work well.
  val noWrap: Boolean = false     // For HTTP Content-Encoding should be false.
  deflated.via(inflate(bufferSize, noWrap))
}
```

**ZPipeline.gunzip** — This pipeline can be used to decompress stream of _gzipped_ inputs, according to [RFC 1952](https://tools.ietf.org/html/rfc1952):

```scala

def decompressGzipped(gzipped: ZStream[Any, Nothing, Byte]): ZStream[Any, CompressionException, Byte] = {
  val bufferSize: Int = 64 * 1024 // Internal buffer size. Few times bigger than upstream chunks should work well.
  gzipped.via(gunzip(bufferSize))
}
```

**ZPipeline.gunzipAuto** — This pipeline can be used to decompress stream of *possibly* _gzipped_ inputs, according to [RFC 1952](https://tools.ietf.org/html/rfc1952). If the input is gzipped, it will be decompressed; if not, it will be passed downstream as-is:

```scala

def decompressMaybeGzipped(maybeGzipped: ZStream[Any, Nothing, Byte]): ZStream[Any, CompressionException, Byte] = {
  val bufferSize: Int = 64 * 1024 // Internal buffer size. Few times bigger than upstream chunks should work well.
  maybeGzipped.via(gunzipAuto(bufferSize))
}
```

### Decoders

ZIO stream has a wide variety of pipelines to decode chunks of bytes into strings:

| Decoder                     | Input          | Output |
|-----------------------------|----------------|--------|
| `ZPipeline.utfDecode`     | Unicode bytes  | String |
| `ZPipeline.utf8Decode`    | UTF-8 bytes    | String |
| `ZPipeline.utf16Decode`   | UTF-16         | String |
| `ZPipeline.utf16BEDecode` | UTF-16BE bytes | String |
| `ZPipeline.utf16LEDecode` | UTF-16LE bytes | String |
| `ZPipeline.utf32Decode`   | UTF-32 bytes   | String |
| `ZPipeline.utf32BEDecode` | UTF-32BE bytes | String |
| `ZPipeline.utf32LEDecode` | UTF-32LE bytes | String |
| `ZPipeline.usASCIIDecode` | US-ASCII bytes | String |

## Operations

### Output Transformation (Mapping)

To transform the _outputs_ of the pipeline, we can use the `ZPipeline#map` combinator for the success channel, and the `ZPipeline#mapError` combinator for the failure channel. Also, the `ZPipeline.mapChunks` takes a function of type `Chunk[O] => Chunk[O2]` and transforms chunks emitted by the pipeline.

### Input Transformation (Contramap)

To transform the _inputs_ of the pipeline, we can use the `ZPipeline#contramap` combinator. It takes a map function of type `J => I` and convert a `ZPipeline[R, E, I, O]` to `ZPipeline[R, E, J, O]`:

```scala
class ZPipeline[-R, +E, -I, +O] {
  final def contramap[J](f: J => I): ZPipeline[R, E, J, O] = ???
}
```

Let's create an integer parser pipeline using `ZPipeline.contramap`:

```scala
val numbers: ZStream[Any, Nothing, Int] =
 ZStream("1-2-3-4-5")
   .mapConcat(_.split("-"))
   .via(
     ZPipeline.map[String, Int](_.toInt)
   )
```

### Composing

We can compose pipelines in two ways:

1. **Composing Two Pipelines** — One pipeline can be composed with another pipeline, resulting in a composite pipeline:

```scala
val lines: ZStream[Any, Throwable, String] =
  ZStream
    .fromFileName("file.txt")
    .via(
      ZPipeline.utf8Decode >>> ZPipeline.splitLines
    )
```

2. **Composing ZPipeline with ZSink** — One pipeline can be composed with a sink, resulting in a sink that processes elements by piping them through the pipeline and piping the results into the sink:

```scala

val refine: ZIO[Any, Throwable, Long] = {
  val stream: ZStream[Any, Throwable, Byte] = ZStream.fromFileName("file.txt")
  val pipeline: ZPipeline[Any, CharacterCodingException, Byte, String] =
    ZPipeline.utf8Decode >>> ZPipeline.splitLines >>> ZPipeline.filter[String](_.contains('₿'))
  val fileSink: ZSink[Any, Throwable, String, Byte, Long] = ZSink
    .fromFileName("file.refined.txt")
    .contramapChunks[String](
      _.flatMap(line => (line + System.lineSeparator()).getBytes())
    )
  val pipeSink: ZSink[Any, Throwable, Byte, Byte, Long] = pipeline >>> fileSink
  stream >>> pipeSink
}
```

---

## Parallel Operators

## Parallel Zipping

Like `ZStream`, two `ZSink` can be zipped together. Both of them will be run in parallel, and their results will be combined in a tuple:

```scala

val kafkaSink: ZSink[Any, Throwable, Record, Record, Unit] =
  ZSink.foreach[Any, Throwable, Record](record => ZIO.attempt(???))

val pulsarSink: ZSink[Any, Throwable, Record, Record, Unit] =
  ZSink.foreach[Any, Throwable, Record](record => ZIO.attempt(???))

val stream: ZSink[Any, Throwable, Record, Record, Unit] =
  kafkaSink zipPar pulsarSink 
```

## Racing

We are able to `race` multiple sinks, they will run in parallel, and the one that wins will provide the result of our program:

```scala
val stream: ZSink[Any, Throwable, Record, Record, Unit] =
  kafkaSink race pulsarSink 
```

To determine which one succeeded, we should use the `ZSink#raceBoth` combinator, it returns an `Either` result.

---

## Creating Sinks

The `zio.stream` provides numerous kinds of sinks to use.

### Common Constructors

**ZSink.head** — It creates a sink containing the first element, returns `None` for empty streams:

```scala
val sink: ZSink[Any, Nothing, Int, Int, Option[Int]] = ZSink.head[Int]
val head: ZIO[Any, Nothing, Option[Int]]             = ZStream(1, 2, 3, 4).run(sink)
// Result: Some(1)
``` 

**ZSink.last** — It consumes all elements of a stream and returns the last element of the stream:

```scala
val sink: ZSink[Any, Nothing, Int, Nothing, Option[Int]] = ZSink.last[Int]
val last: ZIO[Any, Nothing, Option[Int]]                 = ZStream(1, 2, 3, 4).run(sink)
// Result: Some(4)
```

**ZSink.count** — A sink that consumes all elements of the stream and counts the number of elements fed to it:

```scala
val sink : ZSink[Any, Nothing, Int, Nothing, Long] = ZSink.count
val count: ZIO[Any, Nothing, Long]                 = ZStream(1, 2, 3, 4, 5).run(sink)
// Result: 5
```

**ZSink.sum** — A sink that consumes all elements of the stream and sums incoming numeric values:

```scala
val sink : ZSink[Any, Nothing, Int, Nothing, Int] = ZSink.sum[Int]
val sum: ZIO[Any, Nothing, Int]                 = ZStream(1, 2, 3, 4, 5).run(sink)
// Result: 15
```

**ZSink.take** — A sink that takes the specified number of values and results in a `Chunk` data type:

```scala
val sink  : ZSink[Any, Nothing, Int, Int, Chunk[Int]] = ZSink.take[Int](3)
val stream: ZIO[Any, Nothing, Chunk[Int]]             = ZStream(1, 2, 3, 4, 5).run(sink)
// Result: Chunk(1, 2, 3)
```

**ZSink.drain** — A sink that ignores its inputs:

```scala
val drain: ZSink[Any, Nothing, Any, Nothing, Unit] = ZSink.drain
```

**ZSink.timed** — A sink that executes the stream and times its execution:

```scala
val timed: ZSink[Any, Nothing, Any, Nothing, Duration] = ZSink.timed
val stream: ZIO[Any, Nothing, Long] =
  ZStream(1, 2, 3, 4, 5)
    .schedule(Schedule.fixed(2.seconds))
    .run(timed)
    .map(_.getSeconds)
// Result: 10
```

**ZSink.foreach** — A sink that executes the provided effectful function for every element fed to it:

```scala
val printer: ZSink[Any, IOException, Int, Int, Unit] =
  ZSink.foreach((i: Int) => printLine(i))
val stream : ZIO[Any, IOException, Unit]             =
  ZStream(1, 2, 3, 4, 5).run(printer)
```

### From Success and Failure

Similar to the `ZStream` data type, we can create a `ZSink` using `fail` and `succeed` methods.

A sink that doesn't consume any element from its upstream and succeeds with a value of `Int` type:

```scala
val succeed: ZSink[Any, Any, Any, Nothing, Int] = ZSink.succeed(5)
```

A sink that doesn't consume any element from its upstream and intentionally fails with a message of `String` type:

```scala
val failed : ZSink[Any, String, Any, Nothing, Nothing] = ZSink.fail("fail!")
```

### Collecting

To create a sink that collects all elements of a stream into a `Chunk[A]`, we can use `ZSink.collectAll`:

```scala
val stream    : UStream[Int]    = ZStream(1, 2, 3, 4, 5)
val collection: UIO[Chunk[Int]] = stream.run(ZSink.collectAll[Int])
// Output: Chunk(1, 2, 3, 4, 5)
```

We can collect all elements into a `Set`:

```scala
val collectAllToSet: ZSink[Any, Nothing, Int, Nothing, Set[Int]] = ZSink.collectAllToSet[Int]
val stream: ZIO[Any, Nothing, Set[Int]] = ZStream(1, 3, 2, 3, 1, 5, 1).run(collectAllToSet)
// Output: Set(1, 3, 2, 5)
```

Or we can collect and merge them into a `Map[K, A]` using a merge function. In the following example, we use `(_:Int) % 3` to determine map keys, and we provide the `_ + _` function to merge multiple elements with the same key:

```scala
val collectAllToMap: ZSink[Any, Nothing, Int, Nothing, Map[Int, Int]] = ZSink.collectAllToMap((_: Int) % 3)(_ + _)
val stream: ZIO[Any, Nothing, Map[Int, Int]] = ZStream(1, 3, 2, 3, 1, 5, 1).run(collectAllToMap)
// Output: Map(1 -> 3, 0 -> 6, 2 -> 7)
```

**ZSink.collectAllN** — Collects incoming values into a chunk of maximum size of `n`:

```scala
ZStream(1, 2, 3, 4, 5).run(
  ZSink.collectAllN(3)
)
// Output: Chunk(1,2,3), Chunk(4,5)
```

**ZSink.collectAllWhile** — Accumulates incoming elements into a chunk as long as they verify the given predicate:

```scala
ZStream(1, 2, 0, 4, 0, 6, 7).run(
  ZSink.collectAllWhile(_ != 0)
)
// Output: Chunk(1,2), Chunk(4), Chunk(6,7)
```

**ZSink.collectAllToMapN** — Creates a sink accumulating incoming values into maps of up to `n` keys. Elements are mapped to keys using the function `key`; elements mapped to the same key will be merged with the function `f`:

```scala
object ZSink {
  def collectAllToMapN[Err, In, K](
    n: Long
  )(key: In => K)(f: (In, In) => In): ZSink[Any, Err, In, Err, In, Map[K, In]]
}
```

Let's do an example:

```scala
ZStream(1, 2, 0, 4, 5).run(
  ZSink.collectAllToMapN[Nothing, Int, Int](10)(_ % 3)(_ + _)
)
// Output: Map(1 -> 5, 2 -> 7, 0 -> 0)
```

**ZSink.collectAllToSetN** — Creates a sink accumulating incoming values into sets of maximum size `n`:

```scala
ZStream(1, 2, 1, 2, 1, 3, 0, 5, 0, 2).run(
  ZSink.collectAllToSetN(3)
)
// Output: Set(1,2,3), Set(0,5,2), Set(1)
```

### Folding

Basic fold accumulation of received elements:

```scala
ZSink.foldLeft[Int, Int](0)(_ + _)
```

A fold with short-circuiting has a termination predicate that determines the end of the folding process:

```scala
ZStream.iterate(0)(_ + 1).run(
  ZSink.fold(0)(sum => sum <= 10)((acc, n: Int) => acc + n)
)
// Output: 15
```

**ZSink.foldWeighted** — Creates a sink that folds incoming elements until it reaches the `max` worth of elements determined by the `costFn`, then the pipeline emits the computed value and restarts the folding process:

```scala
object ZSink {
  def foldWeighted[In, S](z: => S)(costFn: (S, In) => Long, max: Long)(
    f: (S, In) => S
  ): ZSink[Any, Nothing, In, In, S] = ???
}
```

In the following example, each time we consume a new element, we return one as the weight of that element using the cost function. After three times, the sum of the weights reaches the `max` number, and the folding process restarts. So we expect this pipeline to group each three elements in one `Chunk`:

```scala
ZStream(3, 2, 4, 1, 5, 6, 2, 1, 3, 5, 6)
  .transduce(
    ZSink
      .foldWeighted(Chunk[Int]())(
        (_, _: Int) => 1,
        3
      ) { (acc, el) =>
        acc ++ Chunk(el)
      }
  )
// Output: Chunk(3,2,4),Chunk(1,5,6),Chunk(2,1,3),Chunk(5,6)
```

Another example is when we want to group elements whose sum is equal to or less than a specific number:

```scala
ZStream(1, 2, 2, 4, 2, 1, 1, 1, 0, 2, 1, 2)
  .transduce(
    ZSink
      .foldWeighted(Chunk[Int]())(
        (_, i: Int) => i.toLong,
        5
      ) { (acc, el) =>
        acc ++ Chunk(el)
      }
  )
// Output: Chunk(1,2,2),Chunk(4),Chunk(2,1,1,1,0),Chunk(2,1,2)
```

:::caution
The `ZSink.foldWeighted` cannot decompose elements whose weight is more than the `max` number. So elements that have an individual cost larger than `max` will force the pipeline to cross the `max` cost. In the last example, if the source stream was `ZStream(1, 2, 2, 4, 2, 1, 6, 1, 0, 2, 1, 2)` the output would be `Chunk(1,2,2),Chunk(4),Chunk(2,1),Chunk(6),Chunk(1,0,2,1),Chunk(2)`. As we see, the `6` element crossed the `max` cost.

To decompose these elements, we should use the `ZSink.foldWeightedDecompose` function.
:::

**ZSink.foldWeightedDecompose** — As we saw in the previous section, we need a way to decompose elements — whose cause the output aggregate cross the `max` — into smaller elements. This version of fold takes the `decompose` function and enables us to do that:

```scala
object ZSink {
  def foldWeightedDecompose[Err, In, S](
     z: S
   )(costFn: (S, In) => Long, max: Long, decompose: In => Chunk[In])(
     f: (S, In) => S
   ): ZSink[Any, Err, In, Err, In, S] = ???
}
```

In the following example, we are break down elements that are bigger than 5, using the `decompose` function:

```scala
ZStream(1, 2, 2, 2, 1, 6, 1, 7, 2, 1, 2)
  .transduce(
    ZSink
      .foldWeightedDecompose(Chunk[Int]())(
        (_, i: Int) => i.toLong,
        5,
        (i: Int) =>
          if (i > 5) Chunk(i - 1, 1) else Chunk(i)
      )((acc, el) => acc ++ Chunk.succeed(el))
  )
// Ouput: Chunk(1,2,2),Chunk(2,1),Chunk(5),Chunk(1,1),Chunk(5),Chunk(1,1,2,1),Chunk(2)
```

**ZSink.foldUntil** — Creates a sink that folds incoming elements until specific `max` elements have been folded:

```scala
ZStream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
  .run(ZSink.foldUntil(0, 3)(_ + _))
// Output: 6, 15, 24, 10
```

**ZSink.foldLeft** — This sink will fold the inputs until the stream ends, resulting in one element:

```scala
val stream: ZIO[Any, Nothing, Int] = 
  ZStream(1, 2, 3, 4).run(ZSink.foldLeft[Int, Int](0)(_ + _))
// Output: 10
```

### From ZIO

The `ZSink.fromZIO` creates a single-value sink produced from a ZIO workflow:

```scala
val sink = ZSink.fromZIO(ZIO.succeed(1))
```

### From File

The `ZSink.fromPath` creates a file sink that consumes byte chunks and writes them to the specified file:

```scala
def fileSink(path: Path): ZSink[Any, Throwable, String, Byte, Long] =
  ZSink
    .fromPath(path)
    .contramapChunks[String](_.flatMap(_.getBytes))

val result = ZStream("Hello", "ZIO", "World!")
  .intersperse("\n")
  .run(fileSink(Paths.get("file.txt")))
```

### From OutputStream

The `ZSink.fromOutputStream` creates a sink that consumes byte chunks and writes them to the `OutputStream`:

```scala
ZStream("Application", "Error", "Logs")
  .intersperse("\n")
  .run(
    ZSink
      .fromOutputStream(java.lang.System.err)
      .contramapChunks[String](_.flatMap(_.getBytes))
  )
```

### From Queue

A queue has a finite or infinite buffer size, so they are useful in situations where we need to consume streams as fast as we can and then do some batching operations on consumed messages. By using `ZSink.fromQueue`, we can create a sink that is backed by a queue; it enqueues each element into the specified queue:

```scala
val myApp: IO[IOException, Unit] =
  for {
    queue    <- Queue.bounded[Int](32)
    producer <- ZStream
      .iterate(1)(_ + 1)
      .schedule(Schedule.fixed(200.millis))
      .run(ZSink.fromQueue(queue))
      .fork
    consumer <- queue.take.flatMap(printLine(_)).forever
    _        <- producer.zip(consumer).join
  } yield ()
```

### From Hub

`Hub` is an asynchronous data type in which a publisher can publish their messages to that and subscribers can subscribe to take messages from the `Hub`. The `ZSink.fromHub` takes a `Hub` and returns a `ZSink` which publishes each element to that `Hub`.

In the following example, the `sink` consumes elements of the `producer` stream and publishes them to the `hub`. We have two consumers that are subscribed to that hub, and they are taking its elements forever:

```scala
val myApp: ZIO[Any, IOException, Unit] =
  for {
    promise <- Promise.make[Nothing, Unit]
    hub <- Hub.bounded[Int](1)
    sink <- ZIO.succeed(ZSink.fromHub(hub))
    producer <- ZStream
      .iterate(0)(_ + 1)
      .schedule(Schedule.fixed(1.seconds))
      .run(sink)
      .fork
    consumers <- ZIO.scoped {
      hub.subscribe.zip(hub.subscribe).flatMap { case (left, right) =>
        for {
          _ <- promise.succeed(())
          f1 <- left.take.flatMap(e => printLine(s"Left Queue: $e")).forever.fork
          f2 <- right.take.flatMap(e => printLine(s"Right Queue: $e")).forever.fork
          _ <- f1.zip(f2).join
        } yield ()
      }
    }.fork
    _ <- promise.await
    _ <- producer.zip(consumers).join
  } yield ()
```

---

## Introduction to ZSink

A `ZSink[R, E, I, L, Z]` is used to consume elements produced by a `ZStream`. You can think of a sink as a function that will consume a variable amount of `I` elements (could be 0, 1, or many!), might fail with an error of type `E`, and will eventually yield a value of type `Z` together with a remainder of type `L` as leftover.

To consume a stream using `ZSink` we can pass `ZSink` to the `ZStream#run` function:

```scala

val stream = ZStream.fromIterable(1 to 1000)
val sink   = ZSink.sum[Int]
val sum    = stream.run(sink)
```

`ZSink` has one type alias called `Sink`. `Sink[E, A, L, B]` is a type alias for `ZSink[Any, E, A, L, B]`. We can think of a `Sink` as a function that does not require any services and will consume a variable amount of `A` elements (could be 0, 1, or many!), might fail with an error of type `E`, and will eventually yield a value of type `B`. The `L` is the type of elements in the leftover.

```scala
type Sink[+E, A, +L, +B] = ZSink[Any, E, A, L, B]
```

---

## Leftovers

## Collecting Leftovers

A sink consumes a variable amount of `I` elements (zero or more) from the upstream. If the upstream is finite, we can collect leftover values by calling `ZSink#collectLeftover`. It returns a tuple that contains the result of the previous sink and its leftovers:

```scala
val s1: ZIO[Any, Nothing, (Chunk[Int], Chunk[Int])] =
  ZStream(1, 2, 3, 4, 5).run(
    ZSink.take(3).collectLeftover
  )
// Output: (Chunk(1, 2, 3), Chunk(4, 5))

val s2: ZIO[Any, Nothing, (Option[Int], Chunk[Int])] =
  ZStream(1, 2, 3, 4, 5).run(
    ZSink.head[Int].collectLeftover
  )
// Output: (Some(1), Chunk(2, 3, 4, 5))
```

## Ignoring Leftovers

If we don't need leftovers, we can drop them by using `ZSink#ignoreLeftover`:

```scala
ZSink.take[Int](3).ignoreLeftover
```

---

## Sink Operations

Having created the sink, we can transform it with provided operations.

## contramap

Contramap is a simple combinator to change the domain of an existing function. While _map_ changes the co-domain of a function, the _contramap_ changes the domain of a function. So the _contramap_ takes a function and maps over its input.

This is useful when we have a fixed output, and our existing function cannot consume those outputs. So we can use _contramap_ to create a new function that can consume that fixed output. Assume we have a `ZSink.sum` that sums incoming numeric values, but we have a `ZStream` of `String` values. We can convert the `ZSink.sum` to a sink that can consume `String` values;

```scala

val numericSum: ZSink[Any, Nothing, Int, Nothing, Int]    = 
  ZSink.sum[Int]
val stringSum : ZSink[Any, Nothing, String, Nothing, Int] = 
  numericSum.contramap((x: String) => x.toInt)

val sum: ZIO[Any, Nothing, Int] =
  ZStream("1", "2", "3", "4", "5").run(stringSum)
// Output: 15
```

## dimap

A `dimap` is an extended `contramap` that additionally transforms sink's output:

```scala

// Convert its input to integers, do the computation and then convert them back to a string
val sumSink: ZSink[Any, Nothing, String, Nothing, String] =
  numericSum.dimap[String, String](_.toInt, _.toString)
  
val sum: ZIO[Any, Nothing, String] =
  ZStream("1", "2", "3", "4", "5").run(sumSink)
// Output: 15
```

## Filtering

Sinks have `ZSink#filterInput` for filtering incoming elements:

```scala

ZStream(1, -2, 0, 1, 3, -3, 4, 2, 0, 1, -3, 1, 1, 6)
  .transduce(
    ZSink
      .collectAllN[Int](3)
      .filterInput[Int](_ > 0)
  )
// Output: Chunk(Chunk(1,1,3),Chunk(4,2,1),Chunk(1,1,6),Chunk())
```

---

## Consuming Streams

```scala

val result: Task[Unit] = ZStream.fromIterable(0 to 100).foreach(printLine(_))
```

### Using a Sink

To consume a stream using `ZSink` we can pass `ZSink` to the `ZStream#run` function:

```scala
val sum: UIO[Int] = ZStream(1,2,3).run(ZSink.sum)
```

### Using fold

The `ZStream#fold` method executes the fold operation over the stream of values and returns a `ZIO` effect containing the result:

```scala
val s1: ZIO[Any, Nothing, Int] = ZStream(1, 2, 3, 4, 5).runFold(0)(_ + _)
val s2: ZIO[Any, Nothing, Int] = ZStream.iterate(1)(_ + 1).runFoldWhile(0)(_ <= 5)(_ + _)
```

### Using foreach

Using `ZStream#foreach` is another way of consuming elements of a stream. It takes a callback of type `O => ZIO[R1, E1, Any]` which passes each element of a stream to this callback:

```scala
ZStream(1, 2, 3).foreach(printLine(_))
```

---

## Creating ZIO Streams

There are several ways to create ZIO Stream. In this section, we are going to enumerate some of the important ways of creating `ZStream`.

## Common Constructors

**ZStream.apply** — Creates a pure stream from a variable list of values:

```scala
val stream: ZStream[Any, Nothing, Int] = ZStream(1, 2, 3)
```

**ZStream.unit** — A stream that contains a single `Unit` value:

```scala
val unit: ZStream[Any, Nothing, Unit] = ZStream.unit
```

**ZStream.never** — A stream that produces no value or fails with an error:

```scala
val never: ZStream[Any, Nothing, Nothing] = ZStream.never
```

**ZStream.repeat** — A stream that repeats using the specified schedule:
```scala
val repeat: ZStream[Any, Nothing, Int] = 
  ZStream(1).repeat(Schedule.forever)
```

**ZStream.iterate** — Takes an initial value and applies the given function to the initial value iteratively. The initial value is the first value produced by the stream, followed by f(init), f(f(init)), ...

```scala
val nats: ZStream[Any, Nothing, Int] = 
  ZStream.iterate(1)(_ + 1) // 1, 2, 3, ...
```

**ZStream.range** — A stream from a range of integers `[min, max)`:

```scala
val range: ZStream[Any, Nothing, Int] = ZStream.range(1, 5) // 1, 2, 3, 4
```

**ZStream.service[R]** — Create a stream that extract the requested service from the environment:

```scala
trait Foo

val fooStream: ZStream[Foo, Nothing, Foo] = ZStream.service[Foo]
```

**ZStream.scoped** — Creates a single-valued stream from a scoped resource:

```scala
val scopedStream: ZStream[Any, Throwable, BufferedReader] =
  ZStream.scoped(
    ZIO.fromAutoCloseable(
      ZIO.attemptBlocking(
        Files.newBufferedReader(java.nio.file.Paths.get("file.txt"))
      )
    )
  )
```

## From Success and Failure

Similar to `ZIO` data type, we can create a `ZStream` using `fail` and `succeed` methods:

```scala
val s1: ZStream[Any, String, Nothing] = ZStream.fail("Uh oh!")
val s2: ZStream[Any, Nothing, Int]    = ZStream.succeed(5)
```

## From Chunks

We can create a stream from a `Chunk`:

```scala
val s1 = ZStream.fromChunk(Chunk(1, 2, 3))
// s1: ZStream[Any, Nothing, Int] = zio.stream.ZStream@115f3db6
```

Or from multiple `Chunks`:

```scala
val s2 = ZStream.fromChunks(Chunk(1, 2, 3), Chunk(4, 5, 6))
// s2: ZStream[Any, Nothing, Int] = zio.stream.ZStream@40b2f38e
```

## From ZIO

**ZStream.fromZIO** — We can create a stream from a ZIO workflow by using `ZStream.fromZIO` constructor. For example, the following stream is a stream that reads a line from a user:

```scala
val readline: ZStream[Any, IOException, String] = 
  ZStream.fromZIO(Console.readLine)
```

A stream that produces one random number:

```scala
val randomInt: ZStream[Any, Nothing, Int] = 
  ZStream.fromZIO(Random.nextInt)
```

**ZStream.fromZIOOption** — In some cases, depending on the result of the ZIO workflow, we should decide to emit an element or return an empty stream. In these cases, we can use `fromZIOOption` constructor:

```scala 
object ZStream {
  def fromZIOOption[R, E, A](fa: ZIO[R, Option[E], A]): ZStream[R, E, A] = ???
}
```

Let's see an example of using this constructor. In this example, we read a string from user input, and then decide to emit that or not; If the user enters an `EOF` string, we emit an empty stream, otherwise we emit the user input:

```scala
val userInput: ZStream[Any, IOException, String] = 
  ZStream.fromZIOOption(
    Console.readLine.mapError(Option(_)).flatMap {
      case "EOF" => ZIO.fail[Option[IOException]](None)
      case o     => ZIO.succeed(o)
    }
  ) 
```

## From Asynchronous Callback

Assume we have an asynchronous function that is based on callbacks. We would like to register a callbacks on that function and get back a stream of the results emitted by those callbacks. We have `ZStream.async` which can adapt functions that call their callbacks multiple times and emit the results over a stream:

```scala
// Asynchronous Callback-based API
def registerCallback(
    name: String,
    onEvent: Int => Unit,
    onError: Throwable => Unit
): Unit = ???

// Lifting an Asynchronous API to ZStream
val stream = ZStream.async[Any, Throwable, Int] { cb =>
  registerCallback(
    "foo",
    event => cb(ZIO.succeed(Chunk(event))),
    error => cb(ZIO.fail(error).mapError(Some(_)))
  )
}
```

The error type of the `register` function is optional, so by setting the error to the `None` we can use it to signal the end of the stream.

## From Iterators

Iterators are data structures that allow us to iterate over a sequence of elements. Similarly, we can think of ZIO Streams as effectual Iterators; every `ZStream` represents a collection of one or more, but effectful values.

**ZStream.fromIteratorSucceed** — We can convert an iterator that does not throw exception to `ZStream` by using `ZStream.fromIteratorSucceed`:

```scala
val s1: ZStream[Any, Throwable, Int] = ZStream.fromIterator(Iterator(1, 2, 3))
val s2: ZStream[Any, Throwable, Int] = ZStream.fromIterator(Iterator.range(1, 4))
val s3: ZStream[Any, Throwable, Int] = ZStream.fromIterator(Iterator.continually(0))
```

Also, there is another constructor called **`ZStream.fromIterator`** that creates a stream from an iterator which may throw an exception.

**ZStream.fromIteratorZIO** — If we have an effectful Iterator that may throw Exception, we can use `fromIteratorZIO` to convert that to the ZIO Stream:

```scala

val lines: ZStream[Any, Throwable, String] = 
  ZStream.fromIteratorZIO(ZIO.attempt(Source.fromFile("file.txt").getLines()))
```

Using this method is not good for resourceful effects like above, so it's better to rewrite that using `ZStream.fromIteratorScoped` function.

**ZStream.fromIteratorScoped** — Using this constructor we can convert a scoped iterator to ZIO Stream:

```scala
val lines: ZStream[Any, Throwable, String] = 
  ZStream.fromIteratorScoped(
    ZIO.fromAutoCloseable(
      ZIO.attempt(scala.io.Source.fromFile("file.txt"))
    ).map(_.getLines())
  )
```

**ZStream.fromJavaIterator** — It is the Java version of these constructors which create a stream from Java iterator that may throw an exception. We can convert any Java collection to an iterator and then lift them to the ZIO Stream.

For example, to convert the Java Stream to the ZIO Stream, `ZStream` has a `fromJavaStream` constructor which convert the Java Stream to the Java Iterator and then convert that to the ZIO Stream using `ZStream.fromJavaIterator` constructor:

```scala
def fromJavaStream[A](stream: => java.util.stream.Stream[A]): ZStream[Any, Throwable, A] =
  ZStream.fromJavaIterator(stream.iterator())
```

Similarly, `ZStream` has `ZStream.fromJavaIteratorSucceed`, `ZStream.fromJavaIteratorZIO` and `ZStream.fromJavaIteratorScoped` constructors.

## From Iterables

**ZStream.fromIterable** — We can create a stream from `Iterable` collection of values:

```scala
val list = ZStream.fromIterable(List(1, 2, 3))
```

**ZStream.fromIterableZIO** — If we have an effect producing a value of type `Iterable` we can use `fromIterableZIO` constructor to create a stream of that effect.

Assume we have a database that returns a list of users using `Task`:

```scala
trait Database {
  def getUsers: Task[List[User]]
}

object Database {
  def getUsers: ZIO[Database, Throwable, List[User]] = 
    ZIO.serviceWithZIO[Database](_.getUsers)
}
```

As this operation is effectful, we can use `ZStream.fromIterableZIO` to convert the result to the `ZStream`:

```scala
val users: ZStream[Database, Throwable, User] = 
  ZStream.fromIterableZIO(Database.getUsers)
```

## From Repetition

**ZStream.repeat** — Repeats the provided value infinitely:

```scala
val repeatZero: ZStream[Any, Nothing, Int] = ZStream.repeat(0)
```

**ZStream.repeatWith** — This is another variant of `repeat`, which repeats according to the provided schedule. For example, the following stream produce zero value every second:

```scala

val repeatZeroEverySecond: ZStream[Any, Nothing, Int] = 
  ZStream.repeatWithSchedule(0, Schedule.spaced(1.seconds))
```

**ZStream.repeatZIO** — Assume we have an effectful API, and we need to call that API and create a stream from the result of that. We can create a stream from that effect that repeats forever.

Let's see an example of creating a stream of random numbers:

```scala
val randomInts: ZStream[Any, Nothing, Int] =
  ZStream.repeatZIO(Random.nextInt)
```

**ZStream.repeatZIOOption** — We can repeatedly evaluate the given effect and terminate the stream based on some conditions.

Let's create a stream repeatedly from user inputs until user enter "EOF" string:

```scala
val userInputs: ZStream[Any, IOException, String] = 
  ZStream.repeatZIOOption(
    Console.readLine.mapError(Option(_)).flatMap {
      case "EOF" => ZIO.fail[Option[IOException]](None)
      case o     => ZIO.succeed(o)
    }
  )
```

Here is another interesting example of using `repeatZIOOption`; In this example, we are draining an `Iterator` to create a stream of that iterator:

```scala
def drainIterator[A](it: Iterator[A]): ZStream[Any, Throwable, A] =
  ZStream.repeatZIOOption {
    ZIO.attempt(it.hasNext).mapError(Some(_)).flatMap { hasNext =>
      if (hasNext) ZIO.attempt(it.next()).mapError(Some(_))
      else ZIO.fail(None)
    }
  }
```

**ZStream.tick** —  A stream that emits Unit values spaced by the specified duration:

```scala
val stream: ZStream[Any, Nothing, Unit] = 
  ZStream.tick(1.seconds)
```

There are some other variant of repetition API like `repeatZIOWith`, `repeatZIOOption`, `repeatZIOChunk` and `repeatZIOChunkOption`.

## From Unfolding/Pagination

In functional programming, `unfold` is dual to `fold`.

With `fold` we can process a data structure and build a return value. For example, we can process a `List[Int]` and return the sum of all its elements.

The `unfold` represents an operation that takes an initial value and generates a recursive data structure, one-piece element at a time by using a given state function. For example, we can create a natural number by using `one` as the initial element and the `inc` function as the state function.

### Unfold

**ZStream.unfold** — `ZStream` has `unfold` function, which is defined as follows:

```scala
object ZStream {
  def unfold[S, A](s: S)(f: S => Option[(A, S)]): ZStream[Any, Nothing, A] = ???
}
```

- **s** — An initial state value
- **f** — A state function `f` that will be applied to the initial state `s`. If the result of this application is `None` the stream will end, otherwise the result is `Some`, so the next element in the stream would be `A` and the current state of transformation changed to the new `S`, this new state is the basis of the next unfold process.

For example, we can a stream of natural numbers using `ZStream.unfold`:

```scala
val nats: ZStream[Any, Nothing, Int] = ZStream.unfold(1)(n => Some((n, n + 1)))
```

We can write `countdown` function using `unfold`:

```scala
def countdown(n: Int) = ZStream.unfold(n) {
  case 0 => None
  case s => Some((s, s - 1))
}
```

Running this function with an input value of 3 returns a `ZStream` which contains 3, 2, 1 values.

**ZStream.unfoldZIO** — `unfoldZIO` is an effectful version of `unfold`. It helps us to perform _effectful state transformation_ when doing unfold operation.

Let's write a stream of lines of input from a user until the user enters the `exit` command:

```scala
val inputs: ZStream[Any, IOException, String] = ZStream.unfoldZIO(()) { _ =>
  Console.readLine.map {
    case "exit"  => None
    case i => Some((i, ()))
  } 
}   
```

`ZStream.unfoldChunk`, and `ZStream.unfoldChunkZIO` are other variants of `unfold` operations but for `Chunk` data type.

### Pagination

**ZStream.paginate** — This is similar to `unfold`, but allows the emission of values to end one step further. For example the following stream emits `0, 1, 2, 3` elements:

```scala
val stream = ZStream.paginate(0) { s =>
  s -> (if (s < 3) Some(s + 1) else None)
}
```

Similar to `unfold` API, `ZStream` has various other forms as well as `ZStream.paginateZIO`, `ZStream.paginateChunk` and `ZStream.paginateChunkZIO`.

### Unfolding vs. Pagination

One might ask what is the difference between `unfold` and `paginate` combinators? When we should prefer one over another? So, let's find the answer to this question by doing another example.

Assume we have a paginated API that returns an enormous amount of data in a paginated fashion. When we call that API, it returns a data type `ResultPage` which contains the first-page result and, it also contains a flag indicating whether that result is the last one, or we have more data on the next page:

```scala
case class PageResult(results: Chunk[RowData], isLast: Boolean)

def listPaginated(pageNumber: Int): ZIO[Any, Throwable, PageResult] = ZIO.fail(???)
```

We want to convert this API to a stream of `RowData` events. For the first attempt, we might think we can do it by using `unfold` operation as below:

```scala
val firstAttempt: ZStream[Any, Throwable, RowData] = 
  ZStream.unfoldChunkZIO(0) { pageNumber =>
    for {
      page <- listPaginated(pageNumber)
    } yield
      if (page.isLast) None
      else Some((page.results, pageNumber + 1))
  }
```

But it doesn't work properly; it doesn't include the last page result. So let's do a trick and to perform another API call to include the last page results:

```scala
val secondAttempt: ZStream[Any, Throwable, RowData] = 
  ZStream.unfoldChunkZIO(Option[Int](0)) {
    case None => ZIO.none // We already hit the last page
    case Some(pageNumber) => // We did not hit the last page yet
     for {
        page <- listPaginated(pageNumber)
      } yield Some(page.results, if (page.isLast) None else Some(pageNumber + 1))
  }
```

This works and contains all the results of returned pages. It works but as we saw, `unfold` is not friendliness to retrieve data from paginated APIs.

We need to do some hacks and extra works to include results from the last page. This is where `ZStream.paginate` operation comes to play, it helps us to convert a paginated API to ZIO stream in a more ergonomic way. Let's rewrite this solution by using `paginate`:

```scala
val finalAttempt: ZStream[Any, Throwable, RowData] = 
  ZStream.paginateChunkZIO(0) { pageNumber =>
    for {
      page <- listPaginated(pageNumber)
    } yield page.results -> (if (!page.isLast) Some(pageNumber + 1) else None)
  }
```

## From Wrapped Streams

Sometimes we have an effect that contains a `ZStream`, we can unwrap the embedded stream and produce a stream from those effects. If the stream is wrapped with the `ZIO` effect, we use `unwrap`, and if it is wrapped with scoped `ZIO` we use `unwrapScoped`:

```scala
val wrappedWithZIO: UIO[ZStream[Any, Nothing, Int]] = 
  ZIO.succeed(ZStream(1, 2, 3))
val s1: ZStream[Any, Nothing, Int] = 
  ZStream.unwrap(wrappedWithZIO)

val wrappedWithZIOScoped = ZIO.succeed(ZStream(1, 2, 3))
val s2: ZStream[Any, Nothing, Int] = 
  ZStream.unwrapScoped(wrappedWithZIOScoped)
```

## From Java IO

**ZStream.fromPath** — Create ZIO Stream from a file:

```scala

val file: ZStream[Any, Throwable, Byte] = 
  ZStream.fromPath(Paths.get("file.txt"))
```

**ZStream.fromInputStream** — Creates a stream from a `java.io.InputStream`:

```scala
val stream: ZStream[Any, IOException, Byte] = 
  ZStream.fromInputStream(new FileInputStream("file.txt"))
```

Note that the InputStream will not be explicitly closed after it is exhausted. Use `ZStream.fromInputStreamZIO`, or `ZStream.fromInputStreamScoped` instead.

**ZStream.fromInputStreamZIO** — Creates a stream from a `java.io.InputStream`. Ensures that the InputStream is closed after it is exhausted:

```scala
val stream: ZStream[Any, IOException, Byte] = 
  ZStream.fromInputStreamZIO(
    ZIO.attempt(new FileInputStream("file.txt"))
      .refineToOrDie[IOException]
  )
```

**ZStream.fromInputStreamScoped** — Creates a stream from a scoped `java.io.InputStream` value:

```scala
val scoped: ZIO[Scope, IOException, FileInputStream] =
  ZIO.fromAutoCloseable(
    ZIO.attempt(new FileInputStream("file.txt"))
  ).refineToOrDie[IOException]

val stream: ZStream[Any, IOException, Byte] = 
  ZStream.fromInputStreamScoped(scoped)
```

**ZStream.fromResource** — Create a stream from resource file:
```scala
val stream: ZStream[Any, IOException, Byte] =
  ZStream.fromResource("file.txt")
```

**ZStream.fromReader** — Creates a stream from a `java.io.Reader`:

```scala
val stream: ZStream[Any, IOException, Char] = 
   ZStream.fromReader(new FileReader("file.txt"))
```

ZIO Stream also has `ZStream.fromReaderZIO` and `ZStream.fromReaderScoped` variants.

## From Java Stream

We can use `ZStream.fromJavaStreamTotal` to convert a Java Stream to ZIO Stream:

```scala
val stream: ZStream[Any, Throwable, Int] = 
  ZStream.fromJavaStream(java.util.stream.Stream.of(1, 2, 3))
```

ZIO Stream also has `ZStream.fromJavaStream`, `ZStream.fromJavaStreamZIO` and `ZStream.fromJavaStreamScoped` variants.

## From Queue and Hub

`Queue` and `Hub` are two asynchronous messaging data types in ZIO that can be converted into the ZIO Stream:

```scala
object ZStream {
  def fromQueue[O](
    queue: Dequeue[O],
    maxChunkSize: Int = DefaultChunkSize
  ): ZStream[Any, Nothing, O] = ???

  def fromHub[A](
    hub: Hub[A]
  ): ZStream[Any, Nothing, A] = ???
}
```

If they contain `Chunk` of elements, we can use `ZStream.fromChunk...` constructors to create a stream from those elements (e.g. `ZStream.fromChunkQueue`):

```scala
for {
  promise <- Promise.make[Nothing, Unit]
  hub     <- Hub.unbounded[Chunk[Int]]
  scoped = ZStream.fromChunkHubScoped(hub).tap(_ => promise.succeed(()))
  stream  = ZStream.unwrapScoped(scoped)
  fiber   <- stream.foreach(printLine(_)).fork
  _       <- promise.await
  _       <- hub.publish(Chunk(1, 2, 3))
  _       <- fiber.join
} yield ()
```

Also, If we need to shutdown a `Queue` or `Hub`, once the stream is closed, we should use `ZStream.from..Shutdown` constructors (e.g. `ZStream.fromQueueWithShutdown`).

Also, we can lift a `TQueue` to the ZIO Stream:

```scala
for {
  q <- STM.atomically(TQueue.unbounded[Int])
  stream = ZStream.fromTQueue(q)
  fiber <- stream.foreach(printLine(_)).fork
  _     <- STM.atomically(q.offer(1))
  _     <- STM.atomically(q.offer(2))
  _     <- fiber.join
} yield ()
```

## From Schedule

We can create a stream from a `Schedule` that does not require any further input. The stream will emit an element for each value output from the schedule, continuing for as long as the schedule continues:

```scala
val stream: ZStream[Any, Nothing, Long] =
  ZStream.fromSchedule(Schedule.spaced(1.second) >>> Schedule.recurs(10))
```

---

## Error Handling

## Recovering from Failure

If we have a stream that may fail, we might need to recover from the failure and run another stream, the `ZStream#orElse` takes another stream, so when the failure occurs it will switch over to the provided stream:

```scala

val s1 = ZStream(1, 2, 3) ++ ZStream.fail("Oh! Error!") ++ ZStream(4, 5)
val s2 = ZStream(6, 7, 8)

val stream = s1.orElse(s2)
// Output: 1, 2, 3, 6, 7, 8
```

Another variant of `orElse` is `ZStream#orElseEither`, which distinguishes elements of the two streams using the `Either` data type. Using this operator, the result of the previous example should be `Left(1), Left(2), Left(3), Right(6), Right(7), Right(8)`.

ZIO stream has `ZStream#catchAll` which is powerful version of `ZStream#orElse`. By using `catchAll` we can decide what to do based on the type and value of the failure:

```scala
val first =
  ZStream(1, 2, 3) ++
    ZStream.fail("Uh Oh!") ++
    ZStream(4, 5) ++
    ZStream.fail("Ouch")

val second = ZStream(6, 7, 8)
val third = ZStream(9, 10, 11)

val stream = first.catchAll {
  case "Uh Oh!" => second
  case "Ouch"   => third
}
// Output: 1, 2, 3, 6, 7, 8
```

## Recovering from Defects

If we need to recover from all causes of failures including defects we should use the `ZStream#catchAllCause` method:

```scala
val s1 = ZStream(1, 2, 3) ++ ZStream.dieMessage("Oh! Boom!") ++ ZStream(4, 5)
val s2 = ZStream(7, 8, 9)

val stream = s1.catchAllCause(_ => s2)
// Output: 1, 2, 3, 7, 8, 9
```

## Recovery from Some Errors

If we need to recover from specific failure we should use `ZStream#catchSome`:

```scala
val s1 = ZStream(1, 2, 3) ++ ZStream.fail("Oh! Error!") ++ ZStream(4, 5)
val s2 = ZStream(7, 8, 9)
val stream = s1.catchSome {
  case "Oh! Error!" => s2
}
// Output: 1, 2, 3, 7, 8, 9
```

And, to recover from a specific cause, we should use `ZStream#catchSomeCause` method:

```scala

val s1 = ZStream(1, 2, 3) ++ ZStream.dieMessage("Oh! Boom!") ++ ZStream(4, 5)
val s2 = ZStream(7, 8, 9)
val stream = s1.catchSomeCause { case Die(value, _) => s2 }
```

## Recovering to ZIO Effect

If our stream encounters an error, we can provide some cleanup task as ZIO effect to our stream by using the `ZStream#onError` method:

```scala

val stream = 
  (ZStream(1, 2, 3) ++ ZStream.dieMessage("Oh! Boom!") ++ ZStream(4, 5))
    .onError(_ => Console.printLine("Stream application closed! We are doing some cleanup jobs.").orDie)
```

## Retry a Failing Stream

When a stream fails, it can be retried according to the given schedule to the `ZStream#retry` operator:

```scala
val numbers = ZStream(1, 2, 3) ++ 
  ZStream
    .fromZIO(
      Console.print("Enter a number: ") *> Console.readLine
        .flatMap(x =>
          x.toIntOption match {
            case Some(value) => ZIO.succeed(value)
            case None        => ZIO.fail("NaN")
          }
        )
    )
    .retry(Schedule.exponential(1.second))
```

## From/To Either

Sometimes, we might be working with legacy API which does error handling with the `Either` data type. We can _absolve_ their error types into the ZStream effect using `ZStream.absolve`:

```scala
def legacyFetchUrlAPI(url: URL): Either[Throwable, String] = ???

def fetchUrl(
    url: URL
): ZStream[Any, Throwable, String] = 
  ZStream.fromZIO(
    ZIO.attemptBlocking(legacyFetchUrlAPI(url))
  ).absolve
```

The type of this stream before absolving is `ZStream[Any, Throwable, Either[Throwable, String]]`, this operation let us submerge the error case of an `Either` into the `ZStream` error type.

We can do the opposite by exposing an error of type `ZStream[R, E, A]` as a part of the `Either` by using `ZStream#either`:

```scala
val inputs: ZStream[Any, Nothing, Either[IOException, String]] = 
  ZStream.fromZIO(Console.readLine).either
```

When we are working with streams of `Either` values, we might want to fail the stream as soon as the emission of the first `Left` value:

```scala
// Stream of Either values that cannot fail
val eitherStream: ZStream[Any, Nothing, Either[String, Int]] =
  ZStream(Right(1), Right(2), Left("failed to parse"), Right(4))

// A Fails with the first emission of the left value
val stream: ZStream[Any, String, Int] = eitherStream.rightOrFail("fail")
```

## Refining Errors

We can keep one or some errors and terminate the fiber with the rest by using `ZStream#refineOrDie`:

```scala
val stream: ZStream[Any, Throwable, Int] =
  ZStream.fail(new Throwable)

val res: ZStream[Any, IllegalArgumentException, Int] =
  stream.refineOrDie { case e: IllegalArgumentException => e }
```

## Timing Out

We can timeout a stream if it does not produce a value after some duration using `ZStream#timeout`, `ZStream#timeoutFail` and `timeoutFailCause` operators:

```scala
stream.timeoutFail(new TimeoutException)(10.seconds)
```

Or we can switch to another stream if the first stream does not produce a value after some duration:

```scala
val alternative = ZStream.fromZIO(ZIO.attempt(???))
stream.timeoutTo(10.seconds)(alternative)
```

---

## Introduction to ZStream

A `ZStream[R, E, O]` is a description of a program that, when evaluated, may emit zero or more values of type `O`, may fail with errors of type `E`, and uses an environment of type `R`.

One way to think of `ZStream` is as a `ZIO` program that could emit multiple values. As we know, a `ZIO[R, E, A]` data type, is a functional effect which is a description of a program that needs an environment of type `R`, it may end with an error of type `E`, and in case of success, it returns a value of type `A`. The important note about `ZIO` effects is that in the case of success they always end with exactly one value. There is no optionality here, no multiple infinite values, we always get exact value:

```scala
val failedEffect: ZIO[Any, String, Nothing]       = ZIO.fail("fail!")
val oneIntValue : ZIO[Any, Nothing, Int]          = ZIO.succeed(3)
val oneListValue: ZIO[Any, Nothing, List[Int]]    = ZIO.succeed(List(1, 2, 3))
val oneOption   : ZIO[Any, Nothing , Option[Int]] = ZIO.succeed(None)
```

A functional stream is pretty similar, it is a description of a program that requires an environment of type `R` and it may signal with errors of type `E` and it yields `O`, but the difference is that it will yield zero or more values.

So a `ZStream` represents one of the following cases in terms of its elements:
- **An Empty Stream** — It might end up empty; which represent an empty stream, e.g. `ZStream.empty`.
- **One Element Stream** — It can represent a stream with just one value, e.g. `ZStream.succeed(3)`.
- **Multiple Finite Element Stream** — It can represent a stream of finite values, e.g. `ZStream.range(1, 10)`
- **Multiple Infinite Element Stream** — It can even represent a stream that _never ends_ as an infinite stream, e.g. `ZStream.iterate(1)(_ + 1)`.

```scala

val emptyStream         : ZStream[Any, Nothing, Nothing]   = ZStream.empty
val oneIntValueStream   : ZStream[Any, Nothing, Int]       = ZStream.succeed(4)
val oneListValueStream  : ZStream[Any, Nothing, List[Int]] = ZStream.succeed(List(1, 2, 3))
val finiteIntStream     : ZStream[Any, Nothing, Int]       = ZStream.range(1, 10)
val infiniteIntStream   : ZStream[Any, Nothing, Int]       = ZStream.iterate(1)(_ + 1)
```

Another example of a stream is when we're pulling a Kafka topic or reading from a socket. There is no inherent definition of an end there. Stream elements arrive at some point, or even they might never arrive at any point.

Based on type parameters of `ZStream`, there are 4 types of streams:

1. `ZStream[Any, Nothing, O]` — A stream that emits `O` values and cannot fail.
2. `ZStream[Any, Throwable, O]` — A stream that emits `O` values and can fail with `Throwable`.
3. `ZStream[Any, Nothing, Nothing]` — A stream that emits no elements.
4. `ZStream[R, E, O]` — A stream that requires access to the `R` service, can fail with error of type `E` and emits `O` values.

---

## Operations

## Tapping

Tapping is an operation of running an effect on each emission of the ZIO Stream. We can think of `ZStream#tap` as an operation that allows us to observe each element of the stream, do some effectful operation and discard the result of this observation. The `tap` operation does not change elements of the stream, it does not affect the return type of the stream.

For example, we can print each element of a stream by using the `tap` operation:

```scala
val stream: ZStream[Any, IOException, Int] =
  ZStream(1, 2, 3)
    .tap(x => printLine(s"before mapping: $x"))
    .map(_ * 2)
    .tap(x => printLine(s"after mapping: $x"))
```

## Taking Elements

We can take a certain number of elements from a stream:

```scala
val stream = ZStream.iterate(0)(_ + 1)
val s1 = stream.take(5)
// Output: 0, 1, 2, 3, 4

val s2 = stream.takeWhile(_ < 5)
// Output: 0, 1, 2, 3, 4

val s3 = stream.takeUntil(_ == 5)
// Output: 0, 1, 2, 3, 4, 5

val s4 = s3.takeRight(3)
// Output: 3, 4, 5
```

## Mapping

**map** — Applies a given function to all element of this stream to produce another stream:
```scala

val intStream: UStream[Int] = ZStream.fromIterable(0 to 100)
val stringStream: UStream[String] = intStream.map(_.toString)
```

If our transformation is effectful, we can use `ZStream#mapZIO` instead.

**mapZIOPar** —  It is similar to `mapZIO`, but will evaluate effects in parallel. It will emit the results downstream in the original order. The `n` argument specifies the number of concurrent running effects.

Let's write a simple page downloader, which download URLs concurrently:

```scala
def fetchUrl(url: URL): Task[String] = ZIO.succeed(???)
def getUrls: Task[List[URL]] = ZIO.succeed(???)

val pages = ZStream.fromIterableZIO(getUrls).mapZIOPar(8)(fetchUrl)  
```

**mapChunk** — Each stream is backed by some `Chunk`s. By using `mapChunk` we can batch the underlying stream and map every `Chunk` at once:

```scala
val chunked = 
  ZStream
    .fromChunks(Chunk(1, 2, 3), Chunk(4, 5), Chunk(6, 7, 8, 9))

val stream = chunked.mapChunks(x => x.tail)

// Input:  1, 2, 3, 4, 5, 6, 7, 8, 9
// Output:    2, 3,    5,    7, 8, 9
```

If our transformation is effectful we can use `mapChunksZIO` combinator.

**mapAccum** — It is similar to a `map`, but it **transforms elements statefully**. `mapAccum` allows us to _map_ and _accumulate_ in the same operation.

```scala
abstract class ZStream[-R, +E, +O] {
  def mapAccum[S, O1](s: S)(f: (S, O) => (S, O1)): ZStream[R, E, O1]
}
```

Let's write a transformation, which calculate _running total_ of input stream:

```scala
def runningTotal(stream: UStream[Int]): UStream[Int] =
  stream.mapAccum(0)((acc, next) => (acc + next, acc + next))

// input:  0, 1, 2, 3,  4,  5
// output: 0, 1, 3, 6, 10, 15
```

**mapConcat** — It is similar to `map`, but maps each element to zero or more elements with the type of `Iterable` and then flattens the whole stream:

```scala
val numbers: UStream[Int] = 
  ZStream("1-2-3", "4-5", "6")
    .mapConcat(_.split("-"))
    .map(_.toInt)

// Input:  "1-2-3", "4-5", "6"
// Output: 1, 2, 3, 4, 5, 6
```

The effectful version of `mapConcat` is `mapConcatZIO`.

`ZStream` also has chunked versions of that which are `mapConcatChunk` and `mapConcatChunkZIO`.

**as** — The `ZStream#as` method maps the success values of this stream to the specified constant value.

For example, we can map all element to the unit value:

```scala
val unitStream: ZStream[Any, Nothing, Unit] = 
  ZStream.range(1, 5).as(())
```

## Filtering

The `ZStream#filter` allows us to filter emitted elements:

```scala
val s1 = ZStream.range(1, 11).filter(_ % 2 == 0)
// Output: 2, 4, 6, 8, 10

// The `ZStream#withFilter` operator enables us to write filter in for-comprehension style
val s2 = for {
  i <- ZStream.range(1, 11).take(10)
  if i % 2 == 0
} yield i
// Output: 2, 4, 6, 8, 10

val s3 = ZStream.range(1, 11).filterNot(_ % 2 == 0)
// Output: 1, 3, 5, 7, 9
```

## Scanning

Scans are like folds, but with a history. Like folds, they take a binary operator with an initial value. A scan combines elements of a stream and emits every intermediary result as an output of the stream:

```scala
val scan = ZStream(1, 2, 3, 4, 5).scan(0)(_ + _)
// Output: 0, 1, 3, 6, 10, 15
// Iterations:
//        =>  0 (initial value)
//  0 + 1 =>  1
//  1 + 2 =>  3
//  3 + 3 =>  6
//  6 + 4 => 10
// 10 + 5 => 15

val fold = ZStream(1, 2, 3, 4, 5).runFold(0)(_ + _)
// Output: 15 (ZIO effect containing 15)
```

## Draining

Assume we have an effectful stream, which contains a sequence of effects; sometimes we might want to execute its effect without emitting any element, in these situations to discard the results we should use the `ZStream#drain` method. It removes all output values from the stream:

```scala
val s1: ZStream[Any, Nothing, Nothing] = ZStream(1, 2, 3, 4, 5).drain
// Emitted Elements: <empty stream, it doesn't emit any element>

val s2: ZStream[Any, IOException, Int] =
  ZStream
    .repeatZIO {
      for {
        nextInt <- Random.nextInt
        number = Math.abs(nextInt % 10)
        _ <- Console.printLine(s"random number: $number")
      } yield (number)
    }
    .take(3)
// Emitted Elements: 1, 4, 7
// Result of Stream Effect on the Console:
// random number: 1
// random number: 4
// random number: 7

val s3: ZStream[Any, IOException, Nothing] = s2.drain
// Emitted Elements: <empty stream, it doesn't emit any element>
// Result of Stream Effect on the Console:
// random number: 4
// random number: 8
// random number: 2
```

The `ZStream#drain` often used with `ZStream#merge` to run one side of the merge for its effect while getting outputs from the opposite side of the merge:

```scala
val logging = ZStream.fromZIO(
  printLine("Starting to merge with the next stream")
)
val stream = ZStream(1, 2, 3) ++ logging.drain ++ ZStream(4, 5, 6)

// Emitted Elements: 1, 2, 3, 4, 5, 6
// Result of Stream Effect on the Console:
// Starting to merge with the next stream
```

Note that if we do not drain the `logging` stream, the emitted elements would be contained unit value:

```scala
val stream = ZStream(1, 2, 3) ++ logging ++ ZStream(4, 5, 6)

// Emitted Elements: 1, 2, 3, (), 4, 5, 6
// Result of Stream Effect on the Console:
// Starting to merge with the next stream
```

## Changes

The `ZStream#changes` emits elements that are not equal to the previous element:

```scala
val changes = ZStream(1, 1, 1, 2, 2, 3, 4).changes
// Output: 1, 2, 3, 4
```

The `ZStream#changes` operator, uses natural equality to determine whether two elements are equal. If we prefer the specialized equality checking, we can provide a function of type `(O, O) => Boolean` to the `ZStream#changesWith` operator.

Assume we have a stream of events with a composite key of _partition_ and _offset_ attributes, and we know that the offset is monotonic in each partition. So, we can use the `changesWith` operator to create a stream of unique elements:

```scala
case class Event(partition: Long, offset: Long, metadata: String) 
val events: ZStream[Any, Nothing, Event] = ZStream.fromIterable(???)

val uniques = events.changesWith((e1, e2) => (e1.partition == e2.partition && e1.offset == e2.offset))
```

## Collecting

We can perform `filter` and `map` operations in a single step using the `ZStream#collect` operation:

```scala
val source1 = ZStream(1, 2, 3, 4, 0, 5, 6, 7, 8)
  
val s1 = source1.collect { case x if x < 6 => x * 2 }
// Output: 2, 4, 6, 8, 0, 10

val s2 = source1.collectWhile { case x if x != 0 => x * 2 }
// Output: 2, 4, 6, 8

val source2 = ZStream(Left(1), Right(2), Right(3), Left(4), Right(5))

val s3 = source2.collectLeft
// Output: 1, 4

val s4 = source2.collectWhileLeft
// Output: 1

val s5 = source2.collectRight
// Output: 2, 3, 5

val s6 = source2.drop(1).collectWhileRight
// Output: 2, 3

val s7 = source2.map(_.toOption).collectSome
// Output: 2, 3, 5

val s8 = source2.map(_.toOption).collectWhileSome
// Output: empty stream
```

We can also do effectful collect using `ZStream#collectZIO` and `ZStream#collectWhileZIO`.

ZIO stream has `ZStream#collectSuccess` which helps us to perform effectful operations and just collect the success values:

```scala
val urls = ZStream(
  "dotty.epfl.ch",
  "zio.dev",
  "zio.github.io/zio-json",
  "zio.github.io/zio-nio/"
)

def fetch(url: String): ZIO[Any, Throwable, String] = 
  ZIO.attemptBlocking(???)

val pages = urls
  .mapZIO(url => fetch(url).exit)
  .collectSuccess
```

## Zipping

We can zip two stream by using `ZStream.zip` or `ZStream#zipWith` operator:

```scala
val s1: UStream[(Int, String)] =
  ZStream(1, 2, 3, 4, 5, 6).zipWith(ZStream("a", "b", "c"))((a, b) => (a, b))

val s2: UStream[(Int, String)] = 
  ZStream(1, 2, 3, 4, 5, 6).zip(ZStream("a", "b", "c"))
  
// Output: (1, "a"), (2, "b"), (3, "c")
``` 

The new stream will end when one of the streams ends.

In case of ending one stream before another, we might need to zip with default values; the `ZStream#zipAll` or `ZStream#zipAllWith` takes default values of both sides to perform such mechanism for us:

```scala
val s1 = ZStream(1, 2, 3)
  .zipAll(ZStream("a", "b", "c", "d", "e"))(0, "x")
val s2 = ZStream(1, 2, 3).zipAllWith(
  ZStream("a", "b", "c", "d", "e")
)(_ => 0, _ => "x")((a, b) => (a, b))

// Output: (1, a), (2, b), (3, c), (0, d), (0, e)
```

Sometimes we want to zip streams, but do not want to zip two elements one by one. For example, we may have two streams producing elements at different speeds, and do not want to wait for the slower one when zipping elements. When we need to zip elements with the latest element of the slower stream, `ZStream#zipLatest` or `ZStream#zipLatestWith` will do this for us. It zips two streams so that when a value is emitted by either of the two streams, it is combined with the latest value from the other stream to produce a result:

```scala
val s1 = ZStream(1, 2, 3)
  .schedule(Schedule.spaced(1.second))

val s2 = ZStream("a", "b", "c", "d")
  .schedule(Schedule.spaced(500.milliseconds))
  .rechunk(3)

s1.zipLatest(s2)

// Output: (1, a), (1, b), (1, c), (1, d), (2, d), (3, d)
```

ZIO Stream also has three useful operators for zipping element of a stream with their previous/next elements and also both of them:

```scala
val stream: UStream[Int] = ZStream.fromIterable(1 to 5)

val s1: UStream[(Option[Int], Int)]              = stream.zipWithPrevious
val s2: UStream[(Int, Option[Int])]              = stream.zipWithNext
val s3: UStream[(Option[Int], Int, Option[Int])] = stream.zipWithPreviousAndNext
```

By using `ZStream#zipWithIndex` we can index elements of a stream:

```scala
val indexedStream: ZStream[Any, Nothing, (String, Long)] = 
  ZStream("Mary", "James", "Robert", "Patricia").zipWithIndex

// Output: ("Mary", 0L), ("James", 1L), ("Robert", 2L), ("Patricia", 3L)
```

## Cross Product

ZIO stream has `ZStream#cross` and its variants to compute _Cartesian Product_ of two streams:

```scala
val first = ZStream(1, 2, 3)
val second = ZStream("a", "b")

val s1 = first cross second
val s2 = first <*> second
val s3 = first.crossWith(second)((a, b) => (a, b))
// Output: (1,a), (1,b), (2,a), (2,b), (3,a), (3,b)

val s4 = first crossLeft second 
val s5 = first <* second
// Keep only elements from the left stream
// Output: 1, 1, 2, 2, 3, 3 

val s6 = first crossRight second
val s7 = first *> second
// Keep only elements from the right stream
// Output: a, b, a, b, a, b
```

Note that the right-hand side stream would be run multiple times, for every element in the left stream.

ZIO stream also has `ZStream.crossN` which takes streams up to four one.

## Partitioning

### partition
`ZStream#partition` function splits the stream into tuple of streams based on the predicate. The first stream contains all element evaluated to true, and the second one contains all element evaluated to false.

The faster stream may advance by up to `buffer` elements further than the slower one. Two streams are wrapped by `Scope` type.

In the example below, left stream consists of even numbers only:

```scala
val partitionResult: ZIO[Scope, Nothing, (ZStream[Any, Nothing, Int], ZStream[Any, Nothing, Int])] =
  ZStream
    .fromIterable(0 to 100)
    .partition(_ % 2 == 0, buffer = 50)
```

### partitionEither
If we need to partition a stream using an effectful predicate we can use `ZStream.partitionEither`.

```scala
abstract class ZStream[-R, +E, +O] {
  final def partitionEither[R1 <: R, E1 >: E, O2, O3](
    p: O => ZIO[R1, E1, Either[O2, O3]],
    buffer: Int = 16
  ): ZIO[R1 with Scope, E1, (ZStream[Any, E1, O2], ZStream[Any, E1, O3])]
}
```

Here is a simple example of using this function:

```scala
val partitioned: ZIO[Scope, Nothing, (ZStream[Any, Nothing, Int], ZStream[Any, Nothing, Int])] =
  ZStream
    .fromIterable(1 to 10)
    .partitionEither(x => ZIO.succeed(if (x < 5) Left(x) else Right(x)))
```

## GroupBy

### groupByKey

To partition the stream by function result we can use `groupBy` by providing a function of type `O => K` which determines by which keys the stream should be partitioned.

```scala
abstract class ZStream[-R, +E, +O] {
  final def groupByKey[K](
    f: O => K,
    buffer: Int = 16
  ): ZStream.GroupBy[R, E, K, O]
}
```

In the example below, exam results are grouped into buckets and counted:

```scala

  case class Exam(person: String, score: Int)

  val examResults = Seq(
    Exam("Alex", 64),
    Exam("Michael", 97),
    Exam("Bill", 77),
    Exam("John", 78),
    Exam("Bobby", 71)
  )

  val groupByKeyResult: ZStream[Any, Nothing, (Int, Int)] =
    ZStream
      .fromIterable(examResults)
      .groupByKey(exam => exam.score / 10 * 10) {
        case (k, s) => ZStream.fromZIO(s.runCollect.map(l => k -> l.size))
      }
```

:::note
`groupByKey` partition the stream by a simple function of type `O => K`; It is not an effectful function. In some cases we need to partition the stream by using an _effectful function_ of type `O => ZIO[R1, E1, (K, V)]`; So we can use `groupBy` which is the powerful version of `groupByKey` function.
:::

### groupBy
It takes an effectful function of type `O => ZIO[R1, E1, (K, V)]`; ZIO Stream uses this function to partition the stream and gives us a new data type called `ZStream.GroupBy` which represent a grouped stream. `GroupBy` has an `apply` method, that takes a function of type `(K, ZStream[Any, E, V]) => ZStream[R1, E1, A]`; ZIO Runtime runs this function across all groups and then merges them in a non-deterministic fashion as a result.

```scala
abstract class ZStream[-R, +E, +O] {
  final def groupBy[R1 <: R, E1 >: E, K, V](
    f: O => ZIO[R1, E1, (K, V)],
    buffer: Int = 16
  ): ZStream.GroupBy[R1, E1, K, V]
}
```

In the example below, we are going `groupBy` given names by their first character and then count the number of names in each group:

```scala
val counted: UStream[(Char, Long)] =
  ZStream("Mary", "James", "Robert", "Patricia", "John", "Jennifer", "Rebecca", "Peter")
    .groupBy(x => ZIO.succeed((x.head, x))) { case (char, stream) =>
      ZStream.fromZIO(stream.runCount.map(count => char -> count))
    }
// Input:  Mary, James, Robert, Patricia, John, Jennifer, Rebecca, Peter
// Output: (P, 2), (R, 2), (M, 1), (J, 3)
```

Let's change the above example a bit into an example of classifying students. The teacher assigns the student to a specific class based on the student's talent. Note that the partitioning operation is an effectful:

```scala
val classifyStudents: ZStream[Any, IOException, (String, Seq[String])] =
  ZStream.fromZIO(
    printLine("Please assign each student to one of the A, B, or C classrooms.")
  ) *> ZStream("Mary", "James", "Robert", "Patricia", "John", "Jennifer", "Rebecca", "Peter")
    .groupBy(student =>
      printLine(s"What is the classroom of $student? ") *>
        readLine.map(classroom => (classroom, student))
    ) { case (classroom, students) =>
      ZStream.fromZIO(
        students
          .runFold(Seq.empty[String])((s, e) => s :+ e)
          .map(students => classroom -> students)
      )
    }

// Input: 
// Please assign each student to one of the A, B, or C classrooms.
// What is the classroom of Mary? A
// What is the classroom of James? B
// What is the classroom of Robert? A
// What is the classroom of Patricia? C
// What is the classroom of John? B
// What is the classroom of Jennifer? A
// What is the classroom of Rebecca? C
// What is the classroom of Peter? A
//
// Output: 
// (B,List(James, John))
// (A,List(Mary, Robert, Jennifer, Peter))
// (C,List(Patricia, Rebecca))
```

## Grouping

### grouped
To partition the stream results with the specified chunk size, we can use the `grouped` function.

```scala
val groupedResult: ZStream[Any, Nothing, Chunk[Int]] =
  ZStream.fromIterable(0 to 8).grouped(3)

// Input:  0, 1, 2, 3, 4, 5, 6, 7, 8
// Output: Chunk(0, 1, 2), Chunk(3, 4, 5), Chunk(6, 7, 8)
```

### groupedWithin
It allows grouping events by time or chunk size, whichever is satisfied first. In the example below every chunk consists of 30 elements and is produced every 3 seconds.

```scala

val groupedWithinResult: ZStream[Any, Nothing, Chunk[Int]] =
  ZStream.fromIterable(0 to 10)
    .repeat(Schedule.spaced(1.seconds))
    .groupedWithin(30, 10.seconds)
```

## Concatenation

We can concatenate two streams by using `ZStream#++` or `ZStream#concat` operator which returns a stream that emits the elements from the left-hand stream and then emits the elements from the right stream:

```scala silent:nest
val a = ZStream(1, 2, 3)
val b = ZStream(4, 5)
val c1 = a ++ b
val c2 = a concat b
```

Also, we can use `ZStream.concatAll` constructor to concatenate given streams together:

```scala
val c3 = ZStream.concatAll(Chunk(a, b))
```

There is also the `ZStream#flatMap` combinator which create a stream which elements are generated by applying a function of type `O => ZStream[R1, E1, O2]` to each output of the source stream and concatenated all of the results:

```scala
val stream = ZStream(1, 2, 3).flatMap(x => ZStream.repeat(x).take(4))
// Input:  1, 2, 3
// Output: 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3
```

Assume we have an API that takes an author name and returns all its book:

```scala
def getAuthorBooks(author: String): ZStream[Any, Throwable, Book] = ZStream(???)
```

If we have a stream of author's names, we can use `ZStream#flatMap` to concatenate the results of all API calls:

```scala
val authors: ZStream[Any, Throwable, String] = 
  ZStream("Mary", "James", "Robert", "Patricia", "John")
val allBooks: ZStream[Any, Throwable, Book]  = 
  authors.flatMap(getAuthorBooks _)
```

If we need to do the `flatMap` concurrently, we can use `ZStream#flatMapPar`, and also if the order of concatenation is not important for us, we can use the `ZStream#flatMapParSwitch` operator.

## Merging

Sometimes we need to interleave the emission of two streams and create another stream. In these cases, we can't use the `ZStream.concat` operation because the `concat` operation waits for the first stream to finish and then consumes the second stream. So we need a non-deterministic way of picking elements from different sources. ZIO Stream's `merge` operations does this for us. Let's discuss some variants of this operation:

### merge

The `ZSstream#merge` picks elements randomly from specified streams:

```scala
val s1 = ZStream(1, 2, 3).rechunk(1)
val s2 = ZStream(4, 5, 6).rechunk(1)

val merged = s1 merge s2
// As the merge operation is not deterministic, it may output the following stream of numbers:
// Output: 4, 1, 2, 5, 6, 3
```

Merge operation always try to pull one chunk from each stream, if we chunk our streams equal or over 3 elements in the last example, we encounter a new stream containing one of the `1, 2, 3, 4, 5, 6` or `4, 5, 6, 1, 2, 3` elements.

### Termination Strategy

When we merge two streams, we should think about the _termination strategy_ of this operation. Each stream has a specific lifetime. One stream may emit all its elements and finish its job, another stream may end after one hour of emission, one another may have a long-running lifetime and never end. So when we merge two streams with different lifetimes, what is the termination strategy of the resulting stream?

By default, when we merge two streams using `ZStream#merge` operation, the newly produced stream will terminate when both specified streams terminate. We can also define the _termination strategy_ corresponding to our requirement. ZIO Stream supports four different termination strategies:

- **Left** — The resulting stream will terminate when the left-hand side stream terminates.
- **Right** — The resulting stream will terminate when the right-hand side stream finishes.
- **Both** — The resulting stream will terminate when both streams finish.
- **Either** — The resulting stream will terminate when one of the streams finishes.

Here is an example of specifying termination strategy when merging two streams:

```scala

val s1 = ZStream.iterate(1)(_+1).take(5).rechunk(1)
val s2 = ZStream.repeat(0).rechunk(1)

val merged = s1.merge(s2, HaltStrategy.Left)
```

We can also use `ZStream#mergeTerminateLeft`, `ZStream#mergeTerminateRight` or `ZStream#mergeTerminateEither` operations instead of specifying manually the termination strategy.

### mergeAll

Usually, micro-services or long-running applications are composed of multiple components that need to run infinitely in the background and if something happens to them, or they terminate abruptly we should crash the entire application.

So our main fiber should perform these three things:

* **Launch and wait** — It should launch all of those background components and wait infinitely. It should not exit prematurely, because then our application won't be running.
* **Interrupt everything** — It should interrupt all those components whenever we receive a termination signal from the operating system.
* **Watch all fibers** — It should watch all those fibers (background components), and quickly exit if something goes wrong.

So how should we do that with our main fiber? Let's try to create a long-running application:

```scala 
val main = 
  kafkaConsumer.runDrain.fork *>
  httpServer.fork *>
  scheduledJobRunner.fork *>
  ZIO.never
```

We can launch the Kafka consumer, the HTTP server, and our job runner and fork them, and then wait using `ZIO.never`. This will indeed wait, but if something happens to any of them and if they crash, nothing happens. So our application just hangs and remains up without anything working in the background. So this approach does not work properly.

So another idea is to watch background components. By using `ZIO.raceFirst` as soon as one of those fibers terminates with either success or failure, it will interrupt all the rest of the components:

```scala
val main =
  ZIO.raceFirst(kafkaConsumer.runDrain, List(httpServer, scheduledJobRunner))
```

We can also do this with streams:

```scala
val main =
  for {
  //_ <- other resources
    _ <- ZStream
      .mergeAllUnbounded(16)(
        kafkaConsumer.drain,
        ZStream.fromZIO(httpServer),
        ZStream.fromZIO(scheduledJobRunner)
      )
      .runDrain
  } yield ()
```

Using `ZStream.mergeAll` we can combine all these streaming components concurrently into one application.

### mergeWith

Sometimes we need to merge two streams and after that, unify them and convert them to new element types. We can do this by using the `ZStream#mergeWith` operation:

```scala
val s1 = ZStream("1", "2", "3")
val s2 = ZStream(4.1, 5.3, 6.2)

val merged = s1.mergeWith(s2)(_.toInt, _.toInt)
```

## Interleaving

When we `merge` two streams, the ZIO Stream picks elements from two streams randomly. But how to merge two streams deterministically? The answer is the `ZStream#interleave` operation.

The `ZStream#interleave` operator pulls an element from each stream, one by one, and then returns an interleaved stream. When one stream is exhausted, all remaining values in the other stream will be pulled:

```scala
val s1 = ZStream(1, 2, 3)
val s2 = ZStream(4, 5, 6, 7, 8)

val interleaved = s1 interleave s2

// Output: 1, 4, 2, 5, 3, 6, 7, 8
```

ZIO Stream also has the `interleaveWith` operator, which is a more powerful version of `interleave`. By using `ZStream#interleaveWith`, we can specify the logic of interleaving:

```scala
val s1 = ZStream(1, 3, 5, 7, 9)
val s2 = ZStream(2, 4, 6, 8, 10)

val interleaved = s1.interleaveWith(s2)(ZStream(true, false, false).forever)
// Output: 1, 2, 4, 3, 6, 8, 5, 10, 7, 9
```

`ZStream#interleaveWith` uses a stream of boolean to decide which stream to choose. If it reaches a true value, it will pick a value from the left-hand side stream, otherwise, it will pick from the right-hand side.

## Interspersing

We can intersperse any stream by using `ZStream#intersperse` operator:

```scala
val s1 = ZStream(1, 2, 3, 4, 5).intersperse(0)
// Output: 1, 0, 2, 0, 3, 0, 4, 0, 5

val s2 = ZStream("a", "b", "c", "d").intersperse("[", "-", "]")
// Output: [, -, a, -, b, -, c, -, d]
```

## Broadcasting

We can broadcast a stream by using `ZStream#broadcast`, it returns a scoped list of streams that have the same elements as the source stream. The `broadcast` operation emits each element to the inputs of returning streams. The upstream stream can emit events as much as `maximumLag`, then it decreases its speed by the slowest downstream stream.

In the following example, we are broadcasting stream of random numbers to the two downstream streams. One of them is responsible to compute the maximum number, and the other one does some logging job with additional delay. The upstream stream decreases its speed by the logging stream:

```scala
val stream: ZIO[Any, IOException, Unit] =
  ZIO.scoped {
    ZStream
      .fromIterable(1 to 20)
      .mapZIO(_ => Random.nextInt)
      .map(Math.abs)
      .map(_ % 100)
      .tap(e => printLine(s"Emit $e element before broadcasting"))
      .broadcast(2, 5)
      .flatMap { streams =>
          for {
            out1 <- streams(0).runFold(0)((acc, e) => Math.max(acc, e))
                      .flatMap(x => printLine(s"Maximum: $x"))
                      .fork
            out2 <- streams(1).schedule(Schedule.spaced(1.second))
                      .foreach(x => printLine(s"Logging to the Console: $x"))
                      .fork
            _    <- out1.join.zipPar(out2.join)
          } yield ()
      }
  }
```

## Distribution

The `ZStream#distributedWith` operator is a more powerful version of `ZStream#broadcast`. It takes a `decide` function, and based on that decide how to distribute incoming elements into the downstream streams:

```scala
abstract class ZStream[-R, +E, +O] {
  final def distributedWith[E1 >: E](
    n: Int,
    maximumLag: Int,
    decide: O => UIO[Int => Boolean]
  ): ZIO[R with Scope, Nothing, List[Dequeue[Exit[Option[E1], O]]]] = ???
}
```

In the example below, we are partitioning incoming elements into three streams using `ZStream#distributedWith` operator:

```scala
val partitioned: ZIO[Scope, Nothing, (UStream[Int], UStream[Int], UStream[Int])] =
  ZStream
    .iterate(1)(_ + 1)
    .schedule(Schedule.fixed(1.seconds))
    .distributedWith(3, 10, x => ZIO.succeed(q => x % 3 == q))
    .flatMap { 
      case q1 :: q2 :: q3 :: Nil =>
        ZIO.succeed(
          ZStream.fromQueue(q1).flattenExitOption,
          ZStream.fromQueue(q2).flattenExitOption,
          ZStream.fromQueue(q3).flattenExitOption
        )
      case _ => ZIO.dieMessage("Impossible!")
    }
```

## Buffering

Since the ZIO streams are pull-based, it means the consumers do not need to message the upstream to slow down. Whenever a downstream stream pulls a new element, the upstream produces a new element. So, the upstream stream is as fast as the slowest downstream stream. Sometimes we need to run producer and consumer independently, in such a situation we can use an asynchronous non-blocking queue for communication between faster producer and slower consumer; the queue can buffer elements between two streams. ZIO stream also has a built-in `ZStream#buffer` operator which does the same thing for us.

The `ZStream#buffer` allows a faster producer to progress independently of a slower consumer by buffering up to `capacity` chunks in a queue.

In the following example, we are going to buffer a stream. We print each element to the console as they are emitting before and after the buffering:

```scala
ZStream
  .fromIterable(1 to 10)
  .rechunk(1)
  .tap(x => Console.printLine(s"before buffering: $x"))
  .buffer(4)
  .tap(x => Console.printLine(s"after buffering: $x"))
  .schedule(Schedule.spaced(5.second))  
```

We spaced 5 seconds between each emission to show the lag between producing and consuming messages.

Based on the type of underlying queue we can use one the buffering operators:
- **Bounded Queue** — `ZStream#buffer(capacity: Int)`
- **Unbounded Queue** — `ZStream#bufferUnbounded`
- **Sliding Queue** — `ZStream#bufferSliding(capacity: Int)`
- **Dropping Queue** `ZStream#bufferDropping(capacity: Int)`

## Debouncing

The `ZStream#debounce` method debounces the stream with a minimum period of `d` between each element:

```scala
val stream = (
  ZStream(1, 2, 3) ++
    ZStream.fromZIO(ZIO.sleep(500.millis)) ++ ZStream(4, 5) ++
    ZStream.fromZIO(ZIO.sleep(10.millis)) ++
    ZStream(6)
).debounce(100.millis) // emit only after a pause of at least 100 ms
// Output: 3, 6
```

## Aggregation

Aggregation is the process of converting one or more elements of type `A` into elements of type `B`. This operation takes a transducer as an aggregation unit and returns another stream that is aggregated. We have two types of aggregation:

### Synchronous Aggregation

They are synchronous because the upstream emits an element when the _transducer_ emits one. To apply a synchronous aggregation to the stream we can use `ZStream#aggregate` or `ZStream#transduce` operations.

Let's see an example of synchronous aggregation:

```scala
val stream = ZStream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
val s1 = stream.transduce(ZSink.collectAllN[Int](3))
// Output Chunk(1,2,3), Chunk(4,5,6), Chunk(7,8,9), Chunk(10)
```

Sometimes stream processing element by element is not efficient, specially when we are working with files or doing I/O works; so we might need to aggregate them and process them in a batch way:

```scala
val source =
  ZStream
    .iterate(1)(_ + 1)
    .take(200)
    .tap(x =>
      printLine(s"Producing Element $x")
        .schedule(Schedule.duration(1.second).jittered)
    )

val sink = 
  ZSink.foreach((e: Chunk[Int]) =>
    printLine(s"Processing batch of events: $e")
      .schedule(Schedule.duration(3.seconds).jittered)
  )
  
val myApp = 
  source.transduce(ZSink.collectAllN[Int](5)).run(sink)
```

Let's see one output of running this program:

```
Producing element 1
Producing element 2
Producing element 3
Producing element 4
Producing element 5
Processing batch of events: Chunk(1,2,3,4,5)
Producing element 6
Producing element 7
Producing element 8
Producing element 9
Producing element 10
Processing batch of events: Chunk(6,7,8,9,10)
Producing element 11
Producing element 12
Processing batch of events: Chunk(11,12)
```

Elements are grouped into Chunks of 5 elements and then processed in a batch way.

### Asynchronous Aggregation

Asynchronous aggregations, aggregate elements of upstream as long as the downstream operators are busy. To apply an asynchronous aggregation to the stream, we can use `ZStream#aggregateAsync`, `ZStream#aggregateAsyncWithin`, and `ZStream#aggregateAsyncWithinEither` operations.

For example, consider `source.aggregateAsync(ZSink.collectAllN[Nothing, Int](5)).mapZIO(processChunks)`. Whenever the downstream (`mapZIO(processChunks)`) is ready for consumption and pulls the upstream, the transducer `(ZTransducer.collectAllN(5))` will flush out its buffer, regardless of whether the `collectAllN` buffered all its 5 elements or not. So the `ZStream#aggregateAsync` will emit when downstream pulls:

```scala
val myApp = 
  source.aggregateAsync(ZSink.collectAllN[Int](5)).run(sink)
```

Let's see one output of running this program:

```
Producing element 1
Producing element 2
Producing element 3
Producing element 4
Processing batch of events: Chunk(1,2)
Processing batch of events: Chunk(3,4)
Producing element 5
Processing batch of events: Chunk(5)
Producing element 6
Processing batch of events: Chunk(6)
Producing element 7
Producing element 8
Producing element 9
Processing batch of events: Chunk(7)
Producing element 10
Producing element 11
Processing batch of events: Chunk(8,9)
Producing element 12
Processing batch of events: Chunk(10,11)
Processing batch of events: Chunk(12)
```

The `ZStream#aggregateAsyncWithin` is another aggregator which takes a scheduler. This scheduler will consume all events produced by the given transducer. So the `aggregateAsyncWithin` will emit when the transducer emits or when the scheduler expires:

```scala
abstract class ZStream[-R, +E, +O] {
  final def aggregateAsyncWithin[R1 <: R, E1 >: E, E2, A1 >: A, B](
    sink: ZSink[R1, E1, A1, E2, A1, B],
    schedule: Schedule[R1, Option[B], Any]
  )(implicit trace: Trace): ZStream[R1, E2, B] = ???
}
```

When we are doing I/O, batching is very important. With ZIO streams, we can create user-defined batches. It is pretty easy to do that with the `ZStream#aggregateAsyncWithin` operator. Let's see the below snippet code:

```scala
dataStream.aggregateAsyncWithin(
   ZSink.collectAllN[Record](2000),
   Schedule.fixed(30.seconds)
 )
```

So it will collect elements into a chunk up to 2000 elements and if we have got less than 2000 elements and 30 seconds have passed, it will pass currently collected elements down the stream whether it has collected zero, one, or 2000 elements. So this is a sort of timeout for aggregation operation. This approach aggressively favors **throughput** over **latency**. It will introduce a fixed amount of latency into a stream. We will always wait for up to 30 seconds if we haven't reached this sort of boundary value.

Instead, thanks to `Schedule` we can create a much smarter **adaptive batching algorithm** that can balance between **throughput** and **latency*. So what we are doing here is that we are creating a schedule that operates on chunks of records. What the `Schedule` does is that it starts off with 30-second timeouts for as long as its input has a size that is lower than 1000, now once we see an input that has a size look higher than 1000, we will switch to a second schedule with some jittery, and we will remain with this schedule for as long as the batch size is over 1000:

```scala
val schedule: Schedule[Any, Option[Chunk[Record]], Long] =
// Start off with 30-second timeouts as long as the batch size is < 1000
  Schedule.fixed(30.seconds).whileInput[Option[Chunk[Record]]](_.getOrElse(Chunk.empty).length < 100) andThen
    // and then, switch to a shorter jittered schedule for as long as batches remain over 1000
    Schedule.fixed(5.seconds).jittered.whileInput[Option[Chunk[Record]]](_.getOrElse(Chunk.empty).length >= 1000)
    
dataStream
  .aggregateAsyncWithin(ZSink.collectAllN[Record](2000), schedule)
```

---

## Resourceful Streams

Most of the constructors of `ZStream` have a special variant to lift a scoped resource to a Stream (e.g. `ZStream.fromReaderScoped`). By using these constructors, we are creating streams that are resource-safe. Before creating a stream, they acquire the resource, and after usage; they close the stream.

ZIO Stream also has `acquireRelease` and `finalizer` constructors which are similar to `ZIO.acquireRelease`. They allow us to clean up or finalizing before the stream ends:

## Acquire Release

We can provide `acquire` and `release` actions to `ZStream.acquireReleaseWith` to create a resourceful stream:

```scala
object ZStream {
  def acquireReleaseWith[R, E, A](
    acquire: ZIO[R, E, A]
  )(
    release: A => URIO[R, Any]
  ): ZStream[R, E, A] = ???
```

Let's see an example of using an acquire release when reading a file. In this example, by providing `acquire` and `release` actions to `ZStream.acquireReleaseWith`, it gives us a scoped stream of `BufferedSource`. As this stream is scoped, we can convert that `BufferedSource` to a stream of its lines and then run it, without worrying about resource leakage:

```scala
val lines: ZStream[Any, Throwable, String] =
  ZStream
    .acquireReleaseWith(
      ZIO.attempt(Source.fromFile("file.txt")) <* printLine("The file was opened.")
    )(x => ZIO.succeed(x.close()) <* printLine("The file was closed.").orDie)
    .flatMap { is =>
      ZStream.fromIterator(is.getLines())
    }
```

## Finalization

We can also create a stream that never fails and define a finalizer for it, so that finalizer will be executed before that stream ends.

```scala
object ZStream {
  def finalizer[R](
    finalizer: URIO[R, Any]
  ): ZStream[R, Nothing, Any] = ???
}
```

It is useful when need to add a finalizer to an existing stream. Assume we need to clean up the temporary directory after our streaming application ends:

```scala

def application: ZStream[Any, IOException, Unit] = ZStream.fromZIO(printLine("Application Logic."))
def deleteDir(dir: Path): ZIO[Any, IOException, Unit] = printLine("Deleting file.")

val myApp: ZStream[Any, IOException, Any] =
  application ++ ZStream.finalizer(
    (deleteDir(Paths.get("tmp")) *>
      printLine("Temporary directory was deleted.")).orDie
  )
```

## Ensuring

We might want to run some code after the execution of the stream's finalization. To do so, we can use the `ZStream#ensuring` operator:

```scala
ZStream
  .finalizer(Console.printLine("Finalizing the stream").orDie)
  .ensuring(
    printLine("Doing some other works after stream's finalization").orDie
  )
  
// Output:
// Finalizing the stream
// Doing some other works after stream's finalization
```

---

## Scheduling

To schedule the output of a stream we use `ZStream#schedule` combinator.

Let's space between each emission of the given stream:

```scala

val stream = ZStream(1, 2, 3, 4, 5).schedule(Schedule.spaced(1.second))
```

---

## Streams Are Chunked by Default

Every time we are working with streams, we are always working with chunks. There are no streams with individual elements, these streams have always chunks in their underlying implementation. So every time we evaluate a stream, when we pull an element out of a stream, we are actually pulling out a chunk of elements.

So why streams are designed in this way? This is because of the **efficiency and performance** issues. Every I/O operation in the programming world works with batches. We never work with a single element. 

For example, whenever we are reading or writing from/to a file descriptor, or a socket we are reading or writing multiple elements at a time. This is also true when we are working with an HTTP server or even JDBC drivers. We always read and write multiple bytes to be more performant.

So let's talk a bit about Chunk. Chunk is a ZIOs immutable array-backed collection. It is initially written for ZIO stream, but later it has been evolved into a very attractive general collection type which is also useful for other purposes. 

The `Chunk` data type is an immutable array-backed collection. Most importantly it tries to keep primitives unboxed. This is super important for the efficient processing of files and sockets. They are also very useful and efficient for encoding and decoding and writing transducers. 

To learn more about this data type, we have introduced that at the [Chunk](../chunk.md) section.

---

## Type Aliases

The `ZStream` data type, has two type aliases:

```scala
type Stream[+E, +A] = ZStream[Any, E, A]
type UStream[+A]    = ZStream[Any, Nothing, A]
```

1. `Stream[E, A]` is a type alias for `ZStream[Any, E, A]`, which represents a ZIO stream that does not require any services, and may fail with an `E`, or produce elements with an `A`.

2. `UStream[A]` is a type alias for `ZStream[Any, Nothing, A]`, which represents a ZIO stream that does not require any services, it cannot fail, and after evaluation, it may emit zero or more values of type `A`.

---

## ConcurrentMap

A `ConcurrentMap` is a wrapper over `java.util.concurrent.ConcurrentHashMap`.

## Motivation

The `HashMap` in the Scala standard library is not thread-safe. This means that if multiple fibers are accessing the same key, and trying to modify the value, this can lead to inconsistent results.

For example, assume we have a `HashMap` with a key `foo` and a value of `0`. Let's see what happens if we perform the `inc` workflow 100 times concurrently:

```scala

object MainApp extends ZIOAppDefault {

  def inc(ref: Ref[mutable.HashMap[String, Int]], key: String) =
    for {
      _ <- ref.get
      _ <- ref.update { map =>
        map.updateWith(key)(_.map(_ + 1))
        map
      }
    } yield ()

  def run =
    for {
      ref <- Ref.make(mutable.HashMap(("foo", 0)))
      _ <- ZIO.foreachParDiscard(1 to 100)(_ => inc(ref, "foo"))
      _ <- ref.get.map(_.get("foo")).debug("The final value of foo is")
    } yield ()

}
// Different outputs on different executions:
// The final value of foo is Some(72)
// The final value of foo is Some(84)
// The final value of foo is Some(78)
// ...
```

Since the `HashMap` is not thread-safe, every time we run this program, we might get different results, which is not desirable.

So we need a concurrent data structure that can be used safely in concurrent environments, which the `ConcurrentMap` does for us:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      map <- ConcurrentMap.make(("foo", 0), ("bar", 1), ("baz", 2))
      _ <- ZIO.foreachParDiscard(1 to 100)(_ =>
        map.computeIfPresent("foo", (_, v) => v + 1)
      )
      _ <- map.get("foo").debug("The final value of foo is")
    } yield ()
}
// Output:
// The final value of foo is Some(100)
```

## Creation

To make an empty `ConcurrentMap` we use `ConcurrentMap.empty`:

```scala

val empty = ConcurrentMap.empty[String, Int]
```

And to make a `ConcurrentMap` with some initial values we use `ConcurrentMap.make` or `ConcurrentMap.fromIterable`:

```scala

val map1 = ConcurrentMap.make(("foo", 0), ("bar", 1), ("baz", 2))
val map2 = ConcurrentMap.fromIterable(List(("foo", 0), ("bar", 1), ("baz", 2)))
```

## Update Operations

Basic operations are provided to manipulate the values in the `ConcurrentMap`:

### Putting values

| Method                                          | Definition                                                                                                 |
|-------------------------------------------------|------------------------------------------------------------------------------------------------------------|
| `put(key: K, value: V): UIO[Option[V]]`         | Adds a new key-value pair and optionally returns previously bound value.                                   |
| `putAll(keyValues: (K, V)*): UIO[Unit]`         | Adds all new key-value pairs.                                                                              |                                                                              |
| `putIfAbsent(key: K, value: V): UIO[Option[V]]` | Adds a new key-value pair, unless the key is already bound to some other value.                            |                            

### Removing values

| Method                                      | Definition                                                                                                 |
|---------------------------------------------|------------------------------------------------------------------------------------------------------------|
| `remove(key: K): UIO[Option[V]]`            | Removes the entry for the given key, optionally returning value associated with it.                        |                        
| `remove(key: K, value: V): UIO[Boolean]`    | Removes the entry for the given key if it is mapped to a given value.                                      |                                      
| `removeIf(p: (K, V) => Boolean): UIO[Unit]` | Removes all elements which do not satisfy the given predicate.                                             |
| `retainIf(p: (K, V) => Boolean): UIO[Unit]` | Removes all elements which do not satisfy the given predicate.                                             |
| `clear: UIO[Unit]`                          | Removes all elements.                                                                                      |

### Replacing values

| Method                                                    | Definition                                                                                                 |
|-----------------------------------------------------------|------------------------------------------------------------------------------------------------------------|
| `replace(key: K, value: V): UIO[Option[V]]`               | Replaces the entry for the given key only if it is mapped to some value.                                   |                                   
| `replace(key: K, oldValue: V, newValue: V): UIO[Boolean]` | Replaces the entry for the given key only if it was previously mapped to a given value.                    |

### Remapping Values

| Method                                                         | Definition                                                                                                 |
|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|
| `compute(key: K, remap: (K, V) => V): UIO[Option[V]]`          | Attempts to compute a mapping for the given key and its current mapped value.                              |
| `def computeIfAbsent(key: K, map: K => V): UIO[V]`             | Computes a value of a non-existing key.                                                                    |
| `computeIfPresent(key: K, remap: (K, V) => V): UIO[Option[V]]` | Attempts to compute a new mapping of an existing key.                                                      |

## Retrieval Operations

| Method                                                            | Definition                                                                                                 |
|-------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|
| `get(key: K): UIO[Option[V]]`                                     | Retrieves the value associated with the given key.                                                         |
| `exists(p: (K, V) => Boolean): UIO[Boolean]`                      | Tests whether a given predicate holds true for at least one element in a map.                              |
| `collectFirst[B](pf: PartialFunction[(K, V), B]): UIO[Option[B]]` | Finds the first element of a map for which the partial function is defined and applies the function to it. | 
| `fold[S](zero: S)(f: (S, (K, V)) => S): UIO[S]`                   | Folds the elements of a map using the given binary operator.                                               |
| `forall(p: (K, V) => Boolean): UIO[Boolean]`                      | Tests whether a predicate is satisfied by all elements of a map.                                           |
| `isEmpty: UIO[Boolean]`                                           | True if there are no elements in this map.                                                                 |
| `toChunk: UIO[Chunk[(K, V)]]`                                     | Collects all entries into a chunk.                                                                         |                                                                         
| `toList: UIO[List[(K, V)]]`                                       | Collects all entries into a list.                                                                          |                                                                          

## Example Usage

Given:

```scala

for {
  emptyMap <- ConcurrentMap.empty[Int, String]
  data     <- ZIO.succeed(Chunk(1 -> "A", 2 -> "B", 3 -> "C"))
  mapA     <- ConcurrentMap.fromIterable(data)
  map100   <- ConcurrentMap.make(1 -> 100)
  mapB     <- ConcurrentMap.make(("A", 1), ("B", 2), ("C", 3))
} yield ()
```

| Operation                                                | Result  |
|----------------------------------------------------------|---------|
| `mapA.collectFirst { case (3, _) => "Three" }`           | "Three" |
| `mapA.collectFirst { case (4, _) => "Four" }`            | Empty   |
| `map100.compute(1, _+_).get(1)`                          | 101     |
| `emptyMap.computeIfAbsent("abc", _.length).get("abc")`   | 3       |
| `map100.computeIfPresent(1, _+_).get(1)`                 | 101     |
| `mapA.exists((k, _) => k % 2 == 0)`                      | true    |
| `mapA.exists((k, _) => k == 4)`                          | false   |
| `mapB.fold(0) { case (acc, (_, value)) => acc + value }` | 6       |
| `mapB.forall((_, v) => v < 4)`                           | true    |
| `emptyMap.get(1)`                                        | None    |
| `emptyMap.put(1, "b").get(1)`                            | "b"     |
| `mapA.putIfAbsent(2, "b").get(2)`                        | "B"     |
| `emptyMap.putAll((1, "A"), (2, "B"), (3, "C")).get(1)`   | "A"     |
| `mapA.remove(1).get(1)`                                  | None    |
| `mapA.remove(1,"b").get(1)`                              | "A"     |
| `mapA.removeIf((k, _) => k != 1).get(1)`                 | "A"     |
| `mapA.removeIf((k, _) => k != 1).get(2)`                 | None    |
| `mapA.retainIf((k, _) => k == 1).get(1)`                 | "A"     |
| `mapA.retainIf((k, _) => k == 1).get(2)`                 | None    |
| `mapA.clear.isEmpty`                                     | true    |

---

## ConcurrentSet

A `ConcurrentSet` is a Set wrapper over `java.util.concurrent.ConcurrentHashMap`.

## Creation

| Method                                                      | Definition                                                           |
|-------------------------------------------------------------|----------------------------------------------------------------------|
| `empty[A]: UIO[ConcurrentSet[A]]`                           | Makes an empty `ConcurrentSet`                                       |
| `empty[A](initialCapacity: Int): UIO[ConcurrentSet[A]]`     | Makes an empty `ConcurrentSet` with initial capacity                 |
| `fromIterable[A](as: Iterable[(A)]): UIO[ConcurrentSet[A]]` | Makes a new `ConcurrentSet` initialized with the provided collection |
| `make[A](as: A*): UIO[ConcurrentSet[A]]`                    | Makes a new `ConcurrentSet` initialized with the provided elements   |

## Update Operations

Basic operations are provided to manipulate the values in the `ConcurrentSet`:

### Adding Values

| Method                                  | Definition          |
|-----------------------------------------|---------------------|
| `add(x: A): UIO[Boolean]`               | Adds a new value.   |
| `addAll(xs: Iterable[A]): UIO[Boolean]` | Adds all new values.|

### Updating Values

| Method                            | Definition                                                            |
|-----------------------------------|-----------------------------------------------------------------------|
| `transform(f: A => A): UIO[Unit]` | Transform all elements of the ConcurrentSet using the given function. |

### Removing Values

| Method                                     | Definition                                                                              |
|--------------------------------------------|-----------------------------------------------------------------------------------------|
| `remove(x: A): UIO[Boolean]`               | Removes the entry for the given value if it is mapped to an existing element.           |
| `removeAll(xs: Iterable[A]): UIO[Boolean]` | Removes all the entries for the given values if they are mapped to an existing element. |
| `removeIf(p: A => Boolean): UIO[Boolean]`  | Removes all elements which satisfy the given predicate.                                 |
| `retainAll(xs: Iterable[A]): UIO[Boolean]` | Retain all the entries for the given values if they are mapped to an existing element.  |
| `retainIf(p: A => Boolean): UIO[Boolean]`  | Removes all elements which do not satisfy the given predicate.                          |
| `clear: UIO[Unit]`                         | Removes all elements.                                                                   |

## Retrieval Operations

| Method                                                        | Definition                                                                                                 |
|---------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|
| `collectFirst[B](pf: PartialFunction[(A, B)): UIO[Option[B]]` | Finds the first element of a set for which the partial function is defined and applies the function to it. |
| `exists(p: A => Boolean): UIO[Boolean]`                       | Tests whether a given predicate holds true for at least one element in the set.                            |
| `fold[R, E, S](zero: S)(f: (S, A) => S): UIO[S]`              | Folds the elements of a set using the given binary operator.                                               |
| `forall(p: A => Boolean): UIO[Boolean]`                       | Tests whether a predicate is satisfied by all elements of a set.                                           |
| `find[B](p: A => Boolean): UIO[Option[A]]`                    | Retrieves the elements in which predicate is satisfied.                                                    |
| `contains(x: A): UIO[Boolean]`                                | Tests whether if the element is in the set.                                                                |
| `containsAll(xs: Iterable[A]): UIO[Boolean]`                  | Tests if the elements in the collection are a subset of the set.                                           |
| `size: UIO[Int]`                                              | Number of elements in the set.                                                                             |
| `isEmpty: UIO[Boolean]`                                       | True if there are no elements in the set.                                                                  |
| `toSet: UIO[Set[A]]`                                          | Convert the ConcurrentSet to Set.                                                                          |

## Example Usage

Given:

```scala

for {
  emptySet  <- ConcurrentSet.empty[Int] 
  setA      <- ConcurrentSet.make[Int](1, 2, 3, 4)
} yield ()
```

| Operation                                 | Result                |
|-------------------------------------------|-----------------------|
| `emptySet.add(1).toSet`                   | Set(1)                |
| `setA.addAll(Chunk(5, 6).toSet)`          | Set(1, 2, 3, 4, 5, 6) |
| `setA.remove(1).toSet`                    | Set(2, 3, 4)          |
| `setA.removeAll(1, 3).toSet`              | Set(2, 4)             |
| `setA.retainAll(List(1, 3, 5, 6)).toSet`  | Set(1, 3)             |
| `setA.clear.isEmpty`                      | true                  |
| `setA.contains(5)`                        | false                 |
| `setA.containsAll(Chunk(1, 2, 3))`        | true                  |
| `setA.exists(_ > 4)`                      | false                 |
| `setA.forAll(_ < 5)`                      | true                  |
| `setA.removeIf(_ % 2 == 0)`               | Set(1, 3)             |
| `setA.retainIf(_ % 2 == 0)`               | Set(2, 4)             |
| `setA.find(_ > 2)`                        | Set(3, 4)             |
| `setA.collectFirst { case 3 => "Three" }` | Set(3)                |
| `setA.size`                               | 4                     |
| `setA.transform(_ + 10)`                  | Set(11, 12, 13, 14)   |
| `setA.fold(0)(_ + _)`                     | 10                    |

---

## CountdownLatch

A synchronization aid that allows one or more fibers to wait until a set of operations being performed in other fibers completes.

A `CountDownLatch` is initialized with a given count. The `await` method block until the current count reaches zero due to invocations of the `countDown` method, after which all waiting fibers are released and any subsequent invocations of `await` return immediately. This is a one-shot phenomenon -- the count cannot be reset. If you need a version that
resets the count, consider using a [`CyclicBarrier`](cyclicbarrier.md).

A `CountDownLatch` is a versatile synchronization tool and can be used for a number of purposes. A `CountDownLatch` initialized with a count of one serves as a simple on/off latch, or gate: all fibers invoking `await` wait at the gate until it is opened by a fiber invoking `countDown`. A `CountDownLatch`initialized to N can be used to make one fiber wait until N fibers have completed some action, or some action has been completed N times.

A useful property of a `CountDownLatch` is that it doesn't require that fibers calling `countDown` wait for the count to reach zero before proceeding, it simply prevents any fiber from proceeding past an `await`until all fibers could pass.

## Creation

To create a `CountDownLatch` we can simply use the `make` constructor. It takes an initial number, for the countdown counter:

```scala
object CountdownLatch {
  def make(n: Int): IO[Option[Nothing], CountdownLatch]
}
```

## Operations

There are two important operations defined on `CountdownLatch`:

```scala
class CountdownLatch {
  val countDown: UIO[Unit]
  val await: UIO[Unit]
}
```

The **`countDown`** operation decrements the count of the latch, releasing all waiting fibers if the count reaches zero, and the **`await`** operation causes the current fiber to wait until the latch has counted down to zero.

## Examples

### Simple on/off Latch

We can simply create an on/off latch using `Promise`. In the following example, we don't want to start the `consume` process before the first `50` number appears in the queue. As it requires a simple on/of latch we can implement that using the `Promise` data type:

```scala

object MainApp extends ZIOAppDefault {

  def consume(queue: Queue[Int]): UIO[Nothing] =
    queue.take
      .flatMap(i => ZIO.debug(s"consumed: $i"))
      .forever

  def produce(queue: Queue[Int], latch: Promise[Nothing, Unit]): UIO[Nothing] =
    (Random
      .nextIntBounded(100)
      .tap(i => queue.offer(i))
      .tap(i => ZIO.when(i == 50)(latch.succeed(()))) *> ZIO.sleep(500.millis)).forever

  def run =
    for {
      latch <- Promise.make[Nothing, Unit]
      queue <- Queue.unbounded[Int]
      _     <- produce(queue, latch) <&> (latch.await *> consume(queue))
    } yield ()
}
```

Alternatively, we can have an on/off latch using `CountDownLatch` with an initial count of _one_:

```scala

object MainApp extends ZIOAppDefault {

  def consume(queue: Queue[Int]): UIO[Nothing] =
    queue.take
      .flatMap(i => ZIO.debug(s"consumed: $i"))
      .forever

  def produce(queue: Queue[Int], latch: CountdownLatch): UIO[Nothing] =
    (Random
      .nextIntBounded(100)
      .tap(i => queue.offer(i))
      .tap(i => ZIO.when(i == 50)(latch.countDown)) *> ZIO.sleep(500.millis)).forever

  def run =
    for {
      latch <- CountdownLatch.make(1)
      queue <- Queue.unbounded[Int]
      _     <- produce(queue, latch) <&> (latch.await *> consume(queue))
    } yield ()
}
```

### Advanced Latches

We can solve more advanced problems by increasing the initial count of `CountdownLatch`.

Assume we had several producers concurrently in the previous example and the consumer was required to wait until at least five 50 numbers were added to the queue before they were allowed to consume. We can do this as follows:

```scala

object MainApp extends ZIOAppDefault {

  def consume(queue: Queue[Int]): UIO[Nothing] =
    queue.take
      .flatMap(i => ZIO.debug(s"consumed: $i"))
      .forever

  def produce(queue: Queue[Int], latch: CountdownLatch): UIO[Nothing] =
    (Random
      .nextIntBounded(100)
      .tap(i => queue.offer(i))
      .tap(i => ZIO.when(i == 50)(latch.countDown)) *> ZIO.sleep(500.millis)).forever

  def run =
    for {
      latch <- CountdownLatch.make(5)
      queue <- Queue.unbounded[Int]
      p = ZIO.collectAllParDiscard(ZIO.replicate(10)(produce(queue, latch)))
      c = latch.await *> consume(queue)
      _     <-  p <&> c
    } yield ()
}
```

In this example, 10 producers are producing numbers concurrently, and the consumer is waiting for its condition to be fulfilled to start the consumption process.

---

## CyclicBarrier

A synchronization aid that allows a set of fibers to all wait for each other to reach a common barrier point.

CyclicBarriers are useful in programs involving a fixed sized party of fibers that must occasionally wait for each other. The barrier is called cyclic because it can be re-used after the waiting fibers are released.

## Creation

To create a `CyclicBarrier` we must provide the number of parties, and we can also provide an optional action:

1. **number of parties**— The fibers that need to synchronize their execution are called _parties_. This number denotes how many parties must occasionally wait for each other. In other words, it specifies the number of parties required to trip the barrier.
2. **action**— An optional command that is run once per barrier point, after the last fiber in the party arrives, but before any fibers are resumed. This action is useful for updating the shared state before any of the parties continue.

```scala
object CyclicBarrier {
  def make(parties: Int)                  : UIO[CyclicBarrier] = ???
  def make(parties: Int, action: UIO[Any]): UIO[CyclicBarrier] = ???
}
```

If we create a barrier and don't call `await` on that, the barrier is not going to be released and the number of `waiting` fibers remains zero:

```scala

for {
  barrier  <- CyclicBarrier.make(100, ZIO.debug("This is a release action!"))
  isBroken <- barrier.isBroken  
  waiting  <- barrier.waiting
} yield assert(!isBroken && waiting == 0)
```

## Simple Example

In the following example, we started three tasks, each one has a different working time, but they won't return until the other parties finished their jobs:

```scala

object MainApp extends ZIOAppDefault {
  def task(name: String) =
    for {
      b <- ZIO.service[CyclicBarrier]
      _ <- ZIO.debug(s"task-$name: started my job right now!")
      d <- Random.nextLongBetween(1000, 10000)
      _ <- ZIO.sleep(Duration.fromMillis(d))
      _ <- ZIO.debug(s"task-$name: finished my job and waiting for other parties to finish their jobs")
      _ <- b.await 
      _ <- ZIO.debug(s"task-$name: the barrier is now broken, so I'm going to exit immediately!")
    } yield ()

  def run =
    for {
      b    <- CyclicBarrier.make(3)
      tasks = task("1") <&> task("2") <&> task("3")
      _    <- tasks.provide(ZLayer.succeed(b))
    } yield ()
}
```

## Cyclic Example

ّIf we change the previous example and add more than three tasks, the first three arriving tasks will be blocked and wait for synchronization. After the barrier is broken, the next three tasks will be blocked on the next barrier. **This process will be executed again and again for further tasks. This is why we say that the barrier is cyclic**:

```scala

object MainApp extends ZIOAppDefault {

  def task(name: String) =
    for {
      b <- ZIO.service[CyclicBarrier]
      _ <- ZIO.debug(s"task-$name: started my job right now!")
      d <- Random.nextLongBetween(1000, 10000)
      _ <- ZIO.sleep(Duration.fromMillis(d))
      _ <- ZIO.debug(s"task-$name: finished my job and waiting for other parties to finish their jobs")
      _ <- b.await
      _ <- ZIO.debug(s"task-$name: the barrier is now broken, so I'm going to exit immediately!")
    } yield ()

  def run =
    for {
      b <- CyclicBarrier.make(
             parties = 3,
             action = ZIO.debug(
               "The barrier is released right now!" +
                 "I can do some effectful actions on release of barrier."
             )
           )
      tasks = task("1") <&>
                task("2") <&>
                task("3") <&>
                task("4") <&>
                task("5")
      _ <- tasks.provide(ZLayer.succeed(b))
    } yield ()
}
```

In this example after breakage of the barrier by proceeding with `task 1`, `task 2`, and `task 3`, the `CyclicBarrier` will be reset to the initial state, so other tasks can come in and `await` on the barrier. So here, `task 4` and `task 5`, proceed with their job and finally wait for all parties to come into the barrier point, but in this example, as we didn't provide `task 6`, the remaining tasks will block the execution of the whole program, infinitely; because the number of waiting fibers are not equal to `parties`.

If we add another concurrent task (e.g. `task("6")`) to our list of tasks, finally the next group of jobs that are waiting for each other will trip the barrier.

## Internals

Each `CyclicBarrier` has the following internal _private_ properties, knowing them helps us to have a deep understanding of how `CyclicBarrier` works:

```scala
class CyclicBarrier private (
  private val _parties: Int,
  private val _waiting: Ref[Int],
  private val _lock: Ref[Promise[Unit, Unit]],
  private val _action: UIO[Any],
  private val _broken: Ref[Boolean]
)
```

Let's introduce each one:

1. `_parties`— The fibers that need to synchronize their execution are called _parties_. It is an immutable property and will be assigned when we create a `CyclicBarrier` using one of the `make` constructors of the `CyclicBarrier`.
2. `_waiting`— This is a mutable property that denotes the number of already fibers waiting for the release of the barrier. These fibers are waiting together for synchronization purpose. To access this property, we can use the `waiting` member of a `CyclicBarrier` which returns `UIO[Int]`.
3. `_lock`— This is a mutable property that contains a `Promise[Unit, Unit]`:
  - When a barrier is _released_, the value of this promise internally will be succeeded with a `Unit` value.
  - When a barrier is _broken_, the value of this promise internally will be failed with a `Unit` value.
  - There is no public API for changing the value of this property.
4. `_action`— When we create a `CyclicBarrier` we can provide an effectful _action_ of type `UIO[Any]` which will be executed when the barrier is released before any of the parties continue.
5. `_broken`— This is a mutable property which denotes that whether the barrier is broken or not:
  - The default value of `_broken` is `false`.
  - When one of the `_waiting` fibers is interrupted, the barrier will be broken and the value of `_broken` will be changed to `true`.
  - We can access this value using `isBroken` method on a `CyclicBarrier`.

## Operations

Let's take a look at the operations defined on a `CyclicBarrier`, then we'll drill down to the important ones:

| Method                   | Definition                                                                                  |
|--------------------------|---------------------------------------------------------------------------------------------|
| `parties: Int`           | The number of parties required to trip this barrier.                                        |
| `waiting: UIO[Int]`      | The number of parties currently waiting at the barrier.                                     |
| `await: IO[Unit, Int]`   | Waits until all parties have invoked await on this barrier. Fails if the barrier is broken. |
| `reset: UIO[Unit]`       | Resets the barrier to its initial state. Breaks any waiting party.                          |
| `isBroken: UIO[Boolean]` | Queries if this barrier is in a broken state.                                               |

### reset

When we reset a barrier, the barrier will be reset to its _initial state_ through the following uninterruptible steps:
- It breaks any waiting party. So all _waiting_ fibers will be interrupted correspondingly.
- The barrier will be ready to synchronize the next groups of parties. So further `await` calls will be accepted for synchronization. This is why we say that the barrier is cyclic.
- Number of _waiting_ fibers will be reset to zero, so there is no fiber in a _waiting_ state.
- If the barrier is broken, it will set its _broken status_ to `false`.

Here is an example shows the mechanism of `reset` method:

```scala

object MainApp extends ZIOAppDefault {
  def task(name: String, b: CyclicBarrier) =
    for {
      _ <- ZIO.debug(s"task-$name: started my job right now!")
      _ <- b.await
      _ <- ZIO.debug(
             s"task-$name: the barrier is now released, " +
               s"so I'm going to exit immediately!"
           )
    } yield ()

  def run =
    for {
      b  <- CyclicBarrier.make(3)
      f1 <- task("1", b).fork
      f2 <- task("2", b).fork
      f3 <-
        (ZIO.sleep(1.second) *> task("3", b))
          .onInterrupt(
            ZIO.debug(
              "task-3: I started my job with some delay! " +
                "so before getting the chance to await on the barrier, " +
                "the reset operation interrupted me!"
            )
          )
          .fork
      _ <- f1.status.repeatWhile(!_.isInstanceOf[Fiber.Status.Suspended])
      _ <- f2.status.repeatWhile(!_.isInstanceOf[Fiber.Status.Suspended])
      _ <- b.waiting.debug("waiting fibers before reset")
      _ <- ZIO.whenZIO(f3.status.map(_.isInstanceOf[Fiber.Status.Running]))(b.reset)
      _ <- b.waiting.debug("waiting fibers after reset")
      _ <- f1.join
      _ <- f2.join
      _ <- f3.join
    } yield ()
}
```

### await

When we call `await` on a `CyclicBarrier`, it will return a value of type `IO[Unit, Int]` through the following uninterruptible steps:
- If the barrier is broken, it will fail with the type of `Unit`.
- Then, it will wait until all parties have invoked `await` on this barrier:
  - If the number of _waiting_ fibers reaches the number of _parties_:
    - First, the optional _action_ effect will be performed.
    - Before resuming all _waiting_ fibers, the barrier will be reset to its _initial state_ using the `reset` method.
    - Accordingly, all parties that are in _waiting_ state due to the call to `await` method will resume and continue processing.
  - If the number of _waiting_ fibers is not reached the number of _parties_, it will suspend the fiber (and that fiber will become one of the _waiting_ fibers) until all parties have invoked `await` on this barrier. During this process, if any waiting fibers are interrupted, [the barrier will be broken](#barrier-breakage-model).

### Barrier Breakage Model

A barrier can be broken in one of the following cases:
1. The `CyclicBarrier` uses an _all-or-none breakage model_ for failed synchronization attempts: If a fiber leaves a barrier point prematurely because of interruption, failure, or timeout, all other fibers waiting at that barrier point will break other parties.
2. Manual reset of a barrier will break all waiting parties.

An example:

```scala

for {
  barrier   <- CyclicBarrier.make(100)
  f1        <- barrier.await.timeout(1.second).fork
  f2        <- barrier.await.fork
  _         <- f1.status.repeatWhile(!_.isInstanceOf[Fiber.Status.Suspended])
  _         <- f2.status.repeatWhile(!_.isInstanceOf[Fiber.Status.Suspended])
  isBroken1 <- barrier.isBroken
  _         <- TestClock.adjust(1.second)
  isBroken2 <- barrier.isBroken
  res1      <- f1.await
  res2      <- f2.await
} yield assert(!isBroken1 && isBroken2)
```

---

## Introduction to ZIO's Synchronization Primitives

When we access shared resources in a concurrent environment, we should choose a proper synchronization mechanism to avoid incorrect results and data inconsistencies. ZIO provides a set of synchronization primitives and concurrent data structures in the `zio-concurrent` module that helps us to achieve the desired synchronization.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-concurrent" % "2.x.x"
```

## Synchronization

ZIO has several synchronization tools:

- **[`ReentrantLock`](reentrantlock.md)**— The `ReentrantLock` is a synchronization tool that is useful for synchronizing blocks of code.
- **[`CountDownLatch`](countdownlatch.md)**— The `CountDownLatch` is a synchronization tool that allows one or more fibers to wait for the finalization of multiple operations.
- **[`CyclicBarrier`](cyclicbarrier.md)**— The `CyclicBarrier` is a synchronization tool that allows a set of fibers to all wait for each other to reach a common barrier point.

## Concurrent Data Structures

It also has some concurrent data structure:

- **[`ConcurrentMap`](concurrentmap.md)**— A `ConcurrentMap` is a Map wrapper over `java.util.concurrent.ConcurrentHashMap`
- **[`ConcurrentSet`](concurrentset.md)**— A `ConcurrentSet` is a Set wrapper over `java.util.concurrent.ConcurrentHashMap`.

---

## MVar

An `MVar[A]` is a mutable location that is either empty or contains a value of type `A`. So the `MVar` acts like a _single-element buffer_.

`MVar` can be used in multiple different ways:
- As a simple on/off latch
- As a binary semaphore `MVar[Unit]`, with `take` and `put` as `acquire` and `release`
- As a synchronized mutable variable
- As a channel, with `take` and `put` as `receive` and `send`

They were introduced in the paper [Concurrent Haskell](http://research.microsoft.com/~simonpj/papers/concurrent-haskell.ps.gz) by Simon Peyton Jones, Andrew Gordon and, Sigbjorn Finne.

## Creation

There are two ways to create an `MVar`:

1. **`MVar.empty[A]`**— To create an `MVar` of type `A` that is _initially empty_, for example:

```scala

val empty = MVar.empty[Int]
```

2. **`MVar.make[A]`**— To create an `MVar` of type `A` that is _initially full_, for example:

```scala

val full = MVar.make(42)
```

## Operations

### Blocking `put` and `take`

`MVar` has two fundamental operations:

- `MVar#put` which fills an `MVar` if it is empty and blocks otherwise.
- `MVar#take` which empties an `MVar` if it is full and blocks otherwise.

```scala
class MVar[A] {
  def put(a: A): UIO[Unit] = ???
  def take: UIO[A] = ???
}
```

So we can put something into it, making it full, or take something out, making it empty, and in two cases, it will block the calling fiber:
- If it is full and the calling fiber tries to put something in it.
- If it is empty and the calling fiber tries to take something out of it.

These two features of `MVar` make it possible to synchronize multiple fibers.

### Nonblocking `tryPut` and `tryTake`

While `put` and `take` are blocking operations, there are also non-blocking versions of these operations:
- `MVar#tryPut` which tries to fill an `MVar` and returns `true` if successful or `false` if it is full.
- `MVar#tryTake` which tries to empty an `MVar` and returns `Some(x)` if it is full of `x` or `None` if it is empty.

```scala
class MVar[A] {
  def tryPut(x: A): UIO[Boolean] = ???
  def tryTake: UIO[Option[A]] = ???
}
```

### Blocking `update`, and `modify`

Using `update` and `modify` we can update the value of an `MVar`. The `update` doesn't return the updated value, but the `modify` does:

```scala
class MVar[A] {
  def update(f: A => A): UIO[Unit] = ???
  def modify[B](f: A => (B, A)): UIO[B] = ???
}
```

Like the `put` and `take` operations, the `update` and `modify` operations are blocking, this means if the `MVar` is empty, they will block the calling fiber until the `MVar` becomes full.

## Use Cases

### Simple On/Off Latch

We can use an `MVar` to implement a simple on/off latch:

```scala

object MainApp extends ZIOAppDefault {

  def job1(latch: MVar[Unit]) =
    for {
      _ <- ZIO.debug("Job 1: I started my work")
      _ <- ZIO.sleep(5.second)
      _ <- ZIO.debug("Job 1: I finished my work")
      _ <- latch.put(())
    } yield ()

  def job2(latch: MVar[Unit]) = for {
    _ <- ZIO.debug("Job 2: I'm waiting for job 1 to finish its work")
    _ <- latch.take
    _ <- ZIO.debug("Job 2: I'm starting my work")
    _ <- ZIO.sleep(4.second)
    _ <- ZIO.debug("Job 2: I finished my work")
  } yield ()

  def run =
    MVar.empty[Unit].flatMap { latch =>
      job1(latch) <&> job2(latch)
    }
}
```

In the above example, we created an empty `MVar`, and then we created two `ZIO` workflows that will be executed concurrently. The first one will wait for the second one to finish its work. But the second one at some point in its execution will need to synchronize with the first one. It needs to make sure that the first one has finished its work before it continues its own work.

### Binary Semaphore

Assume we have a function `inc` that takes a `Ref[Int]` and increments its value by one as below:

```scala

object MainApp extends ZIOAppDefault {

  def inc(ref: Ref[Int]) =
    for {
      v <- ref.get
      result = v + 1
      _ <- ref.set(result)
    } yield ()
    
  def run =
    for {
      ref <- Ref.make(0)
      _ <- ZIO.foreachParDiscard(1 to 100)(_ => inc(ref))
      _ <- ref.get.debug("result")
    } yield ()

}
```

When we perform the `inc` function, 100 times, we expect the final value of the `ref` to be 100. But if we run the program multiple times, we will get different results. This is because the `inc` function is not atomic, and the `ref` may be updated by another thread between the time we read it and the time we write it.

So we need a way to ensure that between the time we read the ref and the time we write to it, no other threads will be able to make changes to it.

We know that `Ref` has the `update` operation that is atomic. So if we rewrite the `inc` as below, our program will work as expected:

```scala
def inc(ref: Ref[Int]) =
  ref.update(_ + 1)
```

Although the solution to this problem is `Ref#update`, we want to use `MVar` to implement the same functionality for pedagogical purposes. So let's see how we can do that using `MVar`:

```scala

object MainApp extends ZIOAppDefault {

  def inc(ref: Ref[Int]) =
    for {
      v <- ref.get
      result = v + 1
      _ <- ref.set(result)
    } yield ()
    
  def run =
    for {
      semaphore <- MVar.make[Unit](())
      ref <- Ref.make(0)
      _ <- ZIO.foreachParDiscard(1 to 100) { _ =>
          for {
            _ <- semaphore.take     // acquire
            _ <- inc(ref)
            _ <- semaphore.put(())  // release
          } yield ()
      }
      _ <- ref.get.debug("result")
    } yield ()

}
```

So we used the `take` as `acquire` and the `put` as the `release` operation of the binary semaphore.

Note that, in the above solution, if any interruption occurs while we have acquired the semaphore (between `acquire` and `release` operations), the semaphore will not be released. So to prevent such a situation, we need to make sure that we always release the semaphore whether the critical section runs successfully or not. Let's model the whole solution in a new data type called `BinarySemaphore`:

```scala

class BinarySemaphore private (mvar: MVar[Unit]) {
  def acquire: ZIO[Any, Nothing, Unit] = mvar.take

  def release: ZIO[Any, Nothing, Unit] = mvar.put(())

  def guard[R, E, A](
      region: ZIO[R, E, A]
  ): ZIO[R, E, A] =
    ZIO.acquireReleaseWith(acquire)(_ => release)(_ => region)
}

object BinarySemaphore {
  def make(): ZIO[Any, Nothing, BinarySemaphore] =
    MVar.make(()).map(new BinarySemaphore(_))
}
```

Now we can apply the `guard` function to the `inc` function of the previous example:

```scala

object MainApp extends ZIOAppDefault {

  def inc(ref: Ref[Int]) =
    for {
      v <- ref.get
      result = v + 1
      _ <- ref.set(result)
    } yield ()

  def run =
    for {
      semaphore <- BinarySemaphore.make()
      ref <- Ref.make(0)
      _ <- ZIO.foreachParDiscard(1 to 100) { _ =>
        semaphore.guard(inc(ref))
      }
      _ <- ref.get.debug("result")
    } yield ()

}
```

### Synchronized Mutable Variable

We can have synchronized mutable variables using the `MVar` data type:

```scala

object MainApp extends ZIOAppDefault {
  def inc(state: MVar[Int]) =
    state.update(_ + 1)

  def run =
    MVar
      .make(0)
      .flatMap(s => ZIO.foreachParDiscard(1 to 100)(_ => inc(s)) *> s.take)
      .debug("result")
}
```

In this case, we executed the same `inc` workflow 100 times concurrently. All the concurrent fibers access the same shared mutable variable called `state` in a synchronized way. In this case, we used the `update`, a safe operation that will atomically update the value of `MVar`.

A question that may be raised is that can we compose `take` and `update` to implement the same functionality for the `inc` workflow as below?

```scala
def inc(state: MVar[Int]) =
  state.take.flatMap(s => state.put(s + 1))
```

Can we say this is the same as the previous `inc` function? No, because although the `take` and `put` are atomic by themselves, their composition is not. So in a real-world scenario, in a concurrent environment it is possible that in between the `take` and `put` operations, the `state` is modified by another fiber. So this is why we used the `update` operation instead, which is an atomic operation.

### Producer/Consumer Channel

We can use an `MVar` to implement a producer/consumer channel:

```scala

object MainApp extends ZIOAppDefault {
  def producer(state: MVar[Int]) =
    Random.nextIntBounded(100)
      .flatMap(state.put)
      .forever
 
  def consumer(state: MVar[Int]) =
    state.take
      .flatMap(i => ZIO.debug(s"$i consumed!"))
      .delay(1.second)
      .forever

  def run =
    MVar.empty[Int].flatMap { s =>
      producer(s) <&> consumer(s)
    }
}
```

In such a case we want to model a producer/consumer channel to make sure the producer doesn't produce any value unless the consumer is ready to consume it. So in this example, `MVar` acts as one element size channel that handles backpressure. 

If we add more consumers, the speed of consuming elements will be increased. Note that, by having multiple consumers, the data will not be duplicated through the consumers. If we have three consumers, each piece of data will be consumed only by one of the consumers:

```scala

object MainApp extends ZIOAppDefault {
  def producer(state: MVar[Int]) =
    ZIO.foreachDiscard(1 to Int.MaxValue)(state.put)

  def consumer(state: MVar[Int])(name: String) =
    state.take
      .flatMap(i => ZIO.debug(s"Consumer $name: $i consumed!"))
      .delay(1.second)
      .forever

  def run =
    MVar.empty[Int].flatMap { s =>
      producer(s) <&>
        consumer(s)("A") <&> consumer(s)("B") <&> consumer(s)("C")
    }
}
```

---

## ReentrantLock

A `ReentrantLock` is a lock which can be acquired multiple times by the same fiber. When a fiber acquires (`lock`) a reentrant lock, it will become the owner of that lock. Other fibers cannot obtain the lock unless the lock owner releases (`unlock`) the lock. As the lock is reentrant, the lock owner can call the `lock` again, multiple times.

## Reentrancy

In reentrancy, only the current working fiber can access a shared resource, preventing any other fibers from doing so. Reentrant locks allow their owner (the fiber that owns the lock) to re-enter them multiple times.

Therefore, in reentrancy locks are acquired per-fiber instead of per-invocation. In other words, if a fiber is not reentrant, and tries to acquire a lock that it already holds, the request won’t succeed.

## Creating ReentrantLocks

Using `ReentrantLocks.make` we can create a reentrant lock in the _unlocked state_:

```scala
object ReentrantLock {
  def make(fairness: Boolean = false): UIO[ReentrantLock] = ???
}
```

By default, it creates a reentrant lock with an unfair policy, so waiters will be picked randomly. If we set the `fairness` parameter to `true`, the reentrant lock will pick the longest waiting fiber.

## Locking and Unlocking

The two basic operations on reentrant locks are `lock` and `unlock`. They acquire and release the lock, respectively:

```scala
trait ReentrantLock {
  lazy val lock:   UIO[Unit]
  lazy val unlock: UIO[Unit]
}
```

1. **`ReentrantLock#lock`**— When a fiber attempt to acquire the lock one of the following cases will happen:

   - When the state is _unlocked_ and in another word if the lock is not held by another fiber, it will acquire the lock and returns immediately and the _hold count_ increased by one.

   - When the state is _locked_ and the current fiber already holds the lock, then the _hold count_ is incremented by one, and the method returns immediately. 

   - When the state is _locked_ and the lock is held by another fiber, then the current fiber will be put to sleep until the lock has been acquired, at which point the lock hold count will be reset to one.

2. **`ReentrantLock#unlock`**— When a fiber attempt to release the lock, one of the following cases will happen:
    - If the current fiber is the holder of this lock then the hold count is decremented. If the hold count is now zero then the lock is released. So if there are any fibers blocked on acquire, one fiber will be picked using (fairness or unfairness policy) and woken up.
    - If the current fiber is not the holder of this lock then nothing happens.

## Fairness Policy

The ReentrantLock constructor offers two fairness policies:
  - unfair policy (the default)
  - fair policy

When a fiber fails to acquire the lock, it is placed in the waiting queue. So when the owning fiber releases the lock, the next waiting fiber chosen by the fairness policy is allowed to try acquiring the lock:
  - In the case of a fairness policy, fibers always acquire a lock in the order in which they requested it. So the reentrant lock will pick the longest waiting fiber from the waiting queue.
  - In case of unfair policy, the reentrant lock will pick a random fiber from the waiting queue.

## Convenience Operations

1. **`ReentrantLock#tryLock`**— Acquires the lock only if it is not held by another fiber at the time of invocation otherwise it will return immediately, so it is a non-blocking operation.

- When the state is _unlocked_ `tryLock` changes the state to _locked_ (with the current fiber as owner and a hold count of 1) and returns `true`.
- When the state is _locked_ `tryLock` leaves the state _unchanged_ and returns `false`.

```scala
trait ReentrantLock {
  lazy val tryLock: UIO[Boolean]
}
```

2. **`ReentrantLock#withLock`**— Acquires and releases the lock as a scoped effect. By using this method, the unlock method will be called automatically at the end of the scope.

```scala
trait ReentrantLock {
  lazy val withLock: URIO[Scope, Int]
}
```

## Querying ReentrantLocks

A reentrant lock has two states: _locked_ or _unlocked_. When the reentrant lock is in _locked_ state it has these properties:
- **Owner** indicates which fiber has acquired the lock. This can be queried by calling the `ReentrantLock#owner` method.
- **Hold Count** indicates how many times its owner acquired the lock. This can be queried using by calling the `ReentrantLock#holdCount` method.
- **Waiters** is a collection of fibers that are waiting to acquire this lock. We can query all of them using the `ReentrantLock#queuedFibers` method.

## Examples

### Example of Simple Locking Mechanism

In the following example, the main fiber acquires the lock, and then we try to acquire the lock from its child fiber. We will see that the child fiber will be blocked when it attempts to acquire the lock until the parent fiber releases it:

```scala

object MainApp extends ZIOAppDefault {

  def run =
    for {
      l  <- ReentrantLock.make()
      fn <- ZIO.fiberId.map(_.threadName)
      _  <- l.lock
      _  <- ZIO.debug(s"$fn acquired the lock.")
      task =
        for {
          fn <- ZIO.fiberId.map(_.threadName)
          _  <- ZIO.debug(s"$fn attempted to acquire the lock.")
          _  <- l.lock
          _  <- ZIO.debug(s"$fn acquired the lock.")
          _  <- ZIO.debug(s"$fn will release the lock after 5 second.")
          _  <- ZIO.sleep(5.second)
          _  <- l.unlock
          _  <- ZIO.debug(s"$fn released the lock.")
        } yield ()
      f <- task.fork
      _ <- ZIO.debug(s"$fn will release the lock after 10 second.")
      _ <- ZIO.sleep(10.second)
      _ <- (l.unlock *> ZIO.debug(s"$fn released the lock.")).uninterruptible
      _ <- f.join
    } yield ()
    
}
// Output:
// zio-fiber-2 acquired the lock.
// zio-fiber-2 will release the lock after 10 second.
// zio-fiber-7 attempted to acquire the lock.
// zio-fiber-2 released the lock.
// zio-fiber-7 acquired the lock.
// zio-fiber-7 will release the lock after 5 second.
// zio-fiber-7 released the lock.
```

Parent fiber (`zio-fiber-2`) acquires the lock and then releases it after 10 seconds. Meanwhile, the child fiber (`zio-fiber-7`) tries to acquire the lock, but it cannot. The attempt to acquire the lock in the child fiber causes the fiber to go into sleep mode. Following the release of the lock by the parent fiber, the child fiber will awaken and acquire the lock.

### Example of Reentrancy

In the previous example, we used the simplest use-case of a locking mechanism that doesn't involve reentrancy. To illustrate how reentrancy works, let's look at another example:

```scala

object MainApp extends ZIOAppDefault {

  def task(l: ReentrantLock, i: Int): ZIO[Any, Nothing, Unit] = for {
    fn <- ZIO.fiberId.map(_.threadName)
    _  <- l.lock
    hc <- l.holdCount
    _  <- ZIO.debug(s"$fn (re)entered the critical section and now the hold count is $hc")
    _  <- ZIO.when(i > 0)(task(l, i - 1))
    _  <- l.unlock
    hc <- l.holdCount
    _  <- ZIO.debug(s"$fn exited the critical section and now the hold count is $hc")
  } yield ()

  def run =
    for {
      l <- ReentrantLock.make()
      _ <- task(l, 2) zipPar task(l, 3)
    } yield ()
}
// One possible output:
// zio-fiber-8 (re)entered the critical section and now the hold count is 1
// zio-fiber-8 (re)entered the critical section and now the hold count is 2
// zio-fiber-8 (re)entered the critical section and now the hold count is 3
// zio-fiber-8 (re)entered the critical section and now the hold count is 4
// zio-fiber-8 exited the critical section and now the hold count is 3
// zio-fiber-8 exited the critical section and now the hold count is 2
// zio-fiber-8 exited the critical section and now the hold count is 1
// zio-fiber-8 exited the critical section and now the hold count is 0
// zio-fiber-7 (re)entered the critical section and now the hold count is 1
// zio-fiber-7 (re)entered the critical section and now the hold count is 2
// zio-fiber-7 (re)entered the critical section and now the hold count is 3
// zio-fiber-7 exited the critical section and now the hold count is 2
// zio-fiber-7 exited the critical section and now the hold count is 1
// zio-fiber-7 exited the critical section and now the hold count is 0
```

In this example, inside the `task` function, we have a critical section. Also, the `task` itself is recursive and inside the critical section, it will call itself. When a fiber tries to enter the critical section and that fiber is the owner of that critical section, the `ReentrantLock` allows that fiber to reenter, and it will increment the `holdCount` by one.

### Example of Producing Deadlock

When two or more fibers wait forever for a lock held by another fiber, they have reached a deadlock. So when we are working with locks, we should be careful of avoiding deadlocks.

In this example, we are just trying to show a simple possible deadlock example:

```scala

object MainApp extends ZIOAppDefault {
  def workflow1(l1: ReentrantLock, l2: ReentrantLock) =
    for {
      f <- ZIO.fiberId.map(_.threadName)
      _ <- l1.lock *> ZIO.debug(s"$f locked the l1")
      o <- l2.owner.map(_.map(_.threadName))
      _ <- ZIO.debug(s"$f trying to lock the l2 while the $o is its owner") *>
        l2.lock *>
        ZIO.debug(s"$f locked the l2")
      _ <- l2.unlock
      _ <- l1.unlock
    } yield ()

  def workflow2(l1: ReentrantLock, l2: ReentrantLock) =
    for {
      f <- ZIO.fiberId.map(_.threadName)
      _ <- l2.lock *> ZIO.debug(s"$f locked the l2")
      o <- l1.owner.map(_.map(_.threadName))
      _ <- ZIO.debug(s"$f trying to lock the l1 while the $o is its owner") *>
        l1.lock *>
        ZIO.debug(s"$f locked the l1")
      _ <- l1.unlock
      _ <- l2.unlock
    } yield ()

  def run =
    for {
      l1 <- ReentrantLock.make()
      l2 <- ReentrantLock.make()
      _ <- workflow1(l1, l2) <&> workflow2(l1, l2)
    } yield ()
}
```

In we run this program, we have a possible deadlock situation, and it might print the following messages and lock forever:

```
zio-fiber-7 locked the l1
zio-fiber-8 locked the l2
zio-fiber-7 trying to lock the l2 while the Some(zio-fiber-8) is its owner
zio-fiber-8 trying to lock the l1 while the Some(zio-fiber-7) is its owner
```

When we run two workflows concurrently, it can cause a deadlock when the first workflow obtains `l1` and in the meantime, the second workflow obtains `l2`, now:
  - When the first workflow tries to obtain `l2` while `l2` is being obtained by `l1`.
  - When the second workflow tries to obtain the `l1` while the `l2` is being obtained by `l1`.
Eventually, both fibers will enter a waiting state, and there will be a deadlock.

---

## Annotating Tests

## Measuring Execution Time

We can annotate the execution time of each test using the `timed` test aspect:

```scala

suite("a timed suite")(
  test("A")(Live.live(ZIO.sleep(100.millis)).map(_ => assertTrue(true))),
  test("B")(assertTrue(true)),
  test("C")(assertTrue(true))
) @@ TestAspect.timed
```

After running the test suite, the output should be something like this:

```
+ a timed suite - 178 ms (100.00%)
  + A - 108 ms (60.95%)
  + B - 34 ms (19.39%)
  + C - 35 ms (19.66%)
Ran 3 tests in 346 ms: 3 succeeded, 0 ignored, 0 failed
```

## Tagging

ZIO Test allows us to define some arbitrary tags. By labeling tests with one or more tags, we can categorize them, and then, when running tests, we can filter tests according to their tags.

Let's tag all slow tests and run them separately:

```scala

object TaggedSpecsExample extends ZIOSpecDefault {
  def spec =
    suite("a suite containing tagged tests")(
      test("a slow test") {
        longRunningAssertion
      } @@ TestAspect.tag("slow", "math"),
      test("a simple test") {
        assertTrue(1 + 1 == 2)
      } @@ TestAspect.tag("math"),
      test("another slow test") {
        anotherLongRunningAssertion
      } @@ TestAspect.tag("slow")
    )
}
```

By adding the `-tags slow` argument to the command line, we will only run the slow tests:

```
sbt> testOnly TaggedSpecsExample -- -tags slow
```

The output would be:

```
[info] running (fork) TaggedSpecsExample -tags slow
[info] + a suite containing tagged tests - tagged: "slow", "math"
[info]   + a slow test - tagged: "slow", "math"
[info]   + another slow test - tagged: "slow"
[info] Ran 2 tests in 162 ms: 2 succeeded, 0 ignored, 0 failed
[success] Total time: 1 s, completed Nov 2, 2021, 12:36:36 PM
```

Run tests tagged with `slow` for the entire project:

```
sbt> testOnly -- -tags slow
```

---

## Before, After, and Around Test Aspects

1. We can run an effect _before_, _after_, or _around_ every test:
- `TestAspect.before`
- `TestAspect.after`
- `TestAspect.afterFailure`
- `TestAspect.afterSuccess`
- `TestAspect.around`

```scala

test("before and after") {
  for {
    tmp <- System.env("TEMP_DIR")
  } yield assertTrue(tmp.contains("/tmp/test"))
} @@ TestAspect.before(
  TestSystem.putEnv("TEMP_DIR", s"/tmp/test")
) @@ TestAspect.after(
  System.env("TEMP_DIR").flatMap(deleteDir)
)
```

2. The `TestAspect.aroundTest` takes a scoped resource and evaluates every test within the context of the scoped function.

3. There are also `TestAspect.beforeAll`, `TestAspect.afterAll`, `afterAllFailure`, `afterAllSuccess`, and `TestAspect.aroundAll` variants.

4. Using `TestAspect.aroundWith` and `TestAspect.aroundAllWith` we can evaluate every test or all test between two given effects, `before` and `after`, where the result of the `before` effect can be used in the `after` effect.

---

## Conditional Aspects

When we apply a conditional aspect, it will run the spec only if the specified predicate is satisfied.

- **`ifEnv`** — Only runs a test if the specified environment variable satisfies the specified assertion.
- **`ifEnvSet`** — Only runs a test if the specified environment variable is set.
- **`ifProp`** — Only runs a test if the specified Java property satisfies the specified assertion.
- **`ifPropSet`** — Only runs a test if the specified Java property is set.

```scala

test("a test that will run if the product is deployed in the testing environment") {
  ???
} @@ ifEnv("ENV")(_ == "testing")

test("a test that will run if the java.io.tmpdir property is available") {
  ???
} @@ ifEnvSet("java.io.tmpdir")
```

---

## Configuring Tests

To run cases, there are some [default configuration settings](../services/test-config.md) which are used by test runner, such as _repeats_, _retries_, _samples_ and _shrinks_. We can change these settings using test aspects:

## Number of Repeats

The `repeats(n: Int)` test aspect runs each test with the number of times to repeat tests to ensure they are stable set to the specified value:

```scala

test("repeating a test") {
  ZIO.attempt("Repeating a test to ensure its stability")
    .debug
    .map(_ => assertTrue(true))
} @@ TestAspect.nonFlaky @@ TestAspect.repeats(5)
```

## Number of Retries

The `retries(n: Int)` test aspect runs each test with the number of times to retry flaky tests set to the specified value.

## Number of Samples

The `samples(n: Int)` test aspect runs each test with the number of sufficient samples to check for a random variable set to the specified value.

Let's change the number of default samples in the following example:

```scala

test("customized number of samples") {
  for {
    ref <- Ref.make(0)
    _ <- check(Gen.int)(_ => assertZIO(ref.update(_ + 1))(Assertion.anything))
    value <- ref.get
  } yield assertTrue(value == 50)
} @@ TestAspect.samples(50)
```

## Maximum Number of Shrinks

The `shrinks(n: Int)` test aspect runs each test with the maximum number of shrinkings to minimize large failures set to the specified value.

---

## Debugging and Diagnostics

## Debugging
The `TestConsole` service has two modes debug and silent state. ZIO Test has two corresponding test aspects to switch the debug state on and off:

1. `TestAspect.debug` — When the `TestConsole` is in the debug state, the console output is rendered to the standard output in addition to being written to the output buffer. We can manually enable this mode by using `TestAspect.debug` test aspect.

2. `TestAspect.silent` — This test aspect turns off the debug mode and turns on the silent mode. So the console output is only written to the output buffer and not rendered to the standard output.

## Diagnostics

The `diagnose` is an aspect that runs each test on a separate fiber and prints a fiber dump if the test fails or has not terminated within the specified duration.

---

## Environment-specific Tests

## OS-specific Tests

To run a test on a specific operating system, we can use one of the `unix`, `mac` or `windows` test aspects or a combination of them. Additionally, we can use the `os` test aspect directly:

```scala

suite("os")(
  test("unix test") {
    ZIO.attempt("running on unix/linux os")
      .debug
      .map(_ => assertTrue(true))
  } @@ TestAspect.unix,
  test("macos test") {
    ZIO.attempt("running on macos")
      .debug
      .map(_ => assertTrue(true))
  } @@ TestAspect.os(_.isMac)
)
```

## Platform-specific Tests

Sometimes we have platform-specific tests. Instead of creating separate sources for each platform to test those tests, we can use a proper aspect to run those tests on a specific platform.

To run a test on a specific platform, we can use one of the `jvm`, `js`, or `native` test aspects or a combination of them. If we want to run our test only on one of these platforms, we can use one of the `jvmOnly`, `jsOnly`, or `nativeOnly` test aspects. To exclude one of these platforms, we can use the `exceptJs`, `exceptJVM`, or `exceptNative` test aspects:

```scala

test("Java virtual machine name can be accessed") {
  for {
    vm <- live(System.property("java.vm.name"))
  } yield
    assertTrue(vm.get.contains("VM"))
} @@ TestAspect.jvmOnly
```

## Version-specific Tests

Various test aspects can be used to run tests for specific versions of Scala, including `scala2`, `scala212`, `scala213`, and `dotty`. As in the previous section, these test aspects have corresponding `*only` and `except*` versions.

---

## Execution Strategy

ZIO Test has two different strategies to run members of a test suite: _sequential_ and _parallel_. Accordingly, there are two test aspects for specifying the execution strategy:

## Parallel

The default strategy is parallel. We can explicitly enable it using `TestAspect.parallel`:

```scala

suite("Parallel")(
  test("A")(Live.live(ZIO.attempt("Running Test A").delay(1.second)).debug.map(_ => assertTrue(true))),
  test("B")(ZIO.attempt("Running Test B").debug.map(_ => assertTrue(true))),
  test("C")(Live.live(ZIO.attempt("Running Test C").delay(500.millis)).debug.map(_ => assertTrue(true)))
) @@ TestAspect.parallel
```

After running this suite, we have the following output:

```
Running Test B
Running Test C
Running Test A
+ Parallel
  + A
  + B
  + C
```

To change the degree of the parallelism, we can use the `parallelN` test aspect. It takes the number of fibers and executes the members of a suite in parallel up to the specified number of concurrent fibers.

## Sequential

To execute them sequentially, we can use the `sequential` test aspect:

```scala

suite("Sequential")(
  test("A")(Live.live(ZIO.attempt("Running Test A").delay(1.second)).debug.map(_ => assertTrue(true))),
  test("B")(ZIO.attempt("Running Test B").debug.map(_ => assertTrue(true))),
  test("C")(Live.live(ZIO.attempt("Running Test C").delay(500.millis)).debug.map(_ => assertTrue(true)))
) @@ TestAspect.sequential
```

And here is the output:

```
Running Test A
Running Test B
Running Test C
+ Sequential
  + A
  + B
  + C
```

---

## Flaky and Non-flaky Tests

Whenever we deal with concurrency issues or race conditions, we should ensure that our tests pass consistently. The `nonFlaky` is a test aspect to do that.

It will run a test several times, by default 100 times, and if all those times pass, it will pass, otherwise, it will fail:

```scala

test("random value is always greater than zero") {
  for {
    random <- Random.nextIntBounded(100)
  } yield assertTrue(random > 0)
} @@ nonFlaky
```

Additionally, there is a `TestAspect.flaky` test aspect which retries a test until it succeeds.

---

## Ignoring Tests

To ignore running a test, we can use the `ignore` test aspect:

```scala

test("an ignored test") {
  assertTrue(false)
} @@ TestAspect.ignore
```

To fail all ignored tests, we can use the `success` test aspect:

```scala

suite("sample tests")(
  test("an ignored test") {
    assertTrue(false)
  } @@ TestAspect.ignore,
  test("another ignored test") {
    assertTrue(true)
  } @@ TestAspect.ignore
) @@ TestAspect.success
```

---

## Introduction to Test Aspects

A `TestAspect` is an aspect that can be weaved into specs. We can think of an aspect as a polymorphic function, capable of transforming one test into another, possibly enlarging the environment or error type. We use them to change existing tests or even entire suites or specs that we have already created.

We can think of a test aspect as a Spec transformer. It takes one spec, transforms it, and produces another spec (`Spec => Spec`). Test aspects are applied to a test or suite using the `@@` operator:

```scala

test("a single test") {
  ???
} @@ testAspect

suite("suite of multiple tests") {
  ???
} @@ testAspect
```

Test aspects encapsulate cross-cutting concerns and increase the modularity of our tests. So we can focus on the primary concerns of our tests and at the end of the day, we can apply required aspects to our tests.

The great thing about test aspects is that they are very composable. So we can chain them one after another. We can even have test aspects that modify other test aspects.

Let's say we have the following test:

```scala

test("test") {
  assertTrue(true)
}
```

We can pass this test to whatever test aspect we want. For example, to run this test only on the JVM and repeat it five times, we can write the test as below:

```scala

repeat(Schedule.recurs(5))(
  jvmOnly(
    test("test") {
      assertTrue(true)
    }
  )
)
```

To compose the aspects, we have a very nice `@@` syntax, which helps us to write tests concisely. So the previous example can be written as follows:

```scala

test("test") {
  assertTrue(true)
} @@ jvmOnly @@ repeat(Schedule.recurs(5))
```

When composing test aspects, **the order of test aspects is important**. So if we change the order, their behavior may change. For example, the following test will repeat the test 2 times:

```scala

suite("suite")(
  test("A") {
    ZIO.debug("executing test")
      .map(_ => assertTrue(true))
  },
) @@ nonFlaky @@ repeats(2)
```

The output:

```
executing test
executing test
executing test
+ suite - repeated: 2
  + A - repeated: 2
Ran 1 test in 343 ms: 1 succeeded, 0 ignored, 0 failed
```

But the following test aspect repeats the test 100 times:

```scala

suite("suite")(
  test("A") {
    ZIO.debug("executing test")
      .map(_ => assertTrue(true))
  },
) @@ repeats(2) @@ nonFlaky
```

The output:

```
executing test
executing test
executing test
executing test
executing test
...
executing test
+ suite - repeated: 100
  + A - repeated: 100
Ran 1 test in 478 ms: 1 succeeded, 0 ignored, 0 failed
```

## Examples

So let's say we have a challenge that we need to run a test, and we want to make sure there is no flaky on the JVM, and then we want to make sure it doesn't take more than 60 seconds:

```scala

test("a test with two aspects composed together") {
  ???
} @@ jvm(nonFlaky) @@ timeout(60.seconds)
```

This is another example of a test suite showing the use of aspects to modify test behavior:

```scala

object MySpec extends ZIOSpecDefault {
  def spec = suite("A Suite")(
    test("A passing test") {
      assertTrue(true)
    },
    test("A passing test run for JVM only") {
      assertTrue(true)
    } @@ jvmOnly, // @@ jvmOnly only runs tests on the JVM
    test("A passing test run for JS only") {
      assertTrue(true)
    } @@ jsOnly, // @@ jsOnly only runs tests on Scala.js
    test("A passing test with a timeout") {
      assertTrue(true)
    } @@ timeout(10.nanos), // @@ timeout will fail a test that doesn't pass within the specified time
    test("A failing test... that passes") {
      assertTrue(true)
    } @@ failing, //@@ failing turns a failing test into a passing test
    test("A ignored test") {
      assertTrue(false)
    } @@ ignore, //@@ ignore marks test as ignored
    test("A test using a live service instead of the test service") {
      for {
        _ <- TestClock.timeZone
      } yield assertCompletes
    } @@ withLiveClock, //@@ withLiveClock uses the live Clock service from the ZIO runtime in the test
    test("A flaky test that only works on the JVM and sometimes fails; let's compose some aspects!") {
      assertTrue(false)
    } @@ jvmOnly           // only run on the JVM
      @@ eventually        // @@ eventually retries a test indefinitely until it succeeds
      @@ timeout(20.nanos) // it's a good idea to compose `eventually` with `timeout`, or the test may never end
  ) @@ timeout(60.seconds) // apply a timeout to the whole suite
}
```

---

## Non-deterministic Test Data

The random process of the `TestRandom` is said to be deterministic since, with the initial seed, we can generate a sequence of predictable numbers. So with the same initial seed, it will generate the same sequence of numbers.

By default, the initial seed of the `TestRandom` is fixed. So repeating a generator more and more results in the same sequence:

```scala

test("pseudo-random number generator with fixed initial seed") {
  check(Gen.int(0, 100)) { n =>
    ZIO.attempt(n).debug.map(_ => assertTrue(true))
  }
} @@
  samples(5) @@
  after(Console.printLine("----").orDie) @@
  repeat(Schedule.recurs(1))
```

Regardless of how many times we repeat this test, the output would be the same:

```
99
51
81
48
51
----
99
51
81
48
51
----
+ pseudo-random numbers with fixed initial seed - repeated: 2
Ran 1 test in 522 ms: 1 succeeded, 0 ignored, 0 failed
```

The `nondeterministic` test aspect, will change the seed of the pseudo-random generator before each test repetition:

```scala

test("pseudo-random number generator with random initial seed on each repetition") {
  check(Gen.int(0, 100)) { n =>
    ZIO.attempt(n).debug.map(_ => assertTrue(true))
  }
} @@
  nondeterministic @@
  samples(5) @@
  after(Console.printLine("----").orDie) @@
  repeat(Schedule.recurs(1))
```

Here is a sample output, which we have different sequences of numbers on each run:

```
73
9
17
33
10
----
42
85
38
2
73
----
+ pseudo-random number generator with random initial seed on each repetition - repeated: 2
Ran 1 test in 733 ms: 1 succeeded, 0 ignored, 0 failed
```

---

## Passing Failed Tests

The `failing` aspect makes a test that failed for any reason pass.

```scala

test("passing a failing test") {
  assertTrue(false)
} @@ TestAspect.failing
```

If the test passes this aspect will make it fail:

```scala

test("failing a passing test") {
  assertTrue(true)
} @@ TestAspect.failing
```

It is also possible to pass a failing test on a specified failure:

```scala

test("a test that will only pass on a specified failure") {
  ZIO.fail("Boom!").map(_ => assertTrue(true))
} @@ TestAspect.failing[String] {
  case TestFailure.Assertion(_, _) => true
  case TestFailure.Runtime(cause: Cause[String], _) => cause match {
    case Cause.Fail(value, _)
      if value == "Boom!" => true
    case _ => false
  }
}
```

---

## Repeat and Retry

There are some situations where we need to repeat a test with a specific schedule, or our tests might fail, and we need to retry them until we make sure that our tests pass. ZIO Test has the following test aspects for these scenarios:

## Repeat 

The `repeat` test aspect takes a `schedule` and repeats a test based on it. The test passes if it passes every time:

```scala

test("repeating a test based on the scheduler to ensure it passes every time") {
  ZIO.debug("repeating successful tests")
    .map(_ => assertTrue(true))
} @@ TestAspect.repeat(Schedule.recurs(5))
```

## Retry

If our test fails occasionally, we can retry failed tests by providing a `schedule` to the `retry` test aspect:

For example, the following test retries a maximum of five times. Once a successful assertion is made, the test passes:

```scala

test("retrying a failing test based on the schedule until it succeeds") {
  ZIO.debug("retrying a failing test")
    .map(_ => assertTrue(true))
} @@ TestAspect.retry(Schedule.recurs(5))
```

## Eventually

3. The `eventually` test aspect keeps retrying a test until it passes, regardless of how many times it fails:

```scala

test("retrying a failing test until it succeeds") {
  ZIO.debug("retrying a failing test")
    .map(_ => assertTrue(true))
} @@ TestAspect.eventually
```

---

## Restoring State of Test Services

ZIO Test has some test aspects which restore the state of given restorable test services, such as `TestClock`, `TestConsole`, `TestRandom` and `TestSystem`, to their starting state after the test is run. Note that these test aspects are only useful when we are repeating tests.

Here is a list of restore methods:

- `TestAspect.restore`
- `TestAspect.restoreTestClock`
- `TestAspect.restoreTestConsole`
- `TestAspect.restoreTestRandom`
- `TestAspect.restoreTestSystem`
- `TestAspect.restoreTestEnvironment`

Let's try an example. Assume we have written the following test aspect, which repeats the test 5 times:

When we run a test with this testing aspect, on each try, we have a polluted test environment:

```scala

suite("clock suite")(
  test("adjusting clock") {
    for {
      clock <- ZIO.clock
      _     <- TestClock.adjust(1.second)
      time  <- clock.currentTime(TimeUnit.SECONDS).debug("current time")
    } yield assertTrue(time == 1)
  } @@ repeat5
)
```

This test fails in the second retry:

```
current time: 1
current time: 2
- some suite
  - clock suite
    - adjusting clock
      ✗ 2 was not equal to 1
      time == 1
      time = 2
```

It failed because of the first run of the test changed the state of the `TestClock` service, so on the next run, the initial state of the test is not zero. In such a situation, when we are repeating a test, after each run we can restore the state of the test to its initial state, using `TestAspect.restore*` test aspects:

```scala

suite("clock suite")(
  test("adjusting clock") {
    for {
      clock <- ZIO.clock
      _     <- TestClock.adjust(1.second)
      time  <- clock.currentTime(TimeUnit.SECONDS).debug("current time")
    } yield assertTrue(time == 1)
  } @@ TestAspect.restoreTestClock @@ repeat5
)
```

The output of running this test would be as follows:

```
current time: 1
current time: 1
current time: 1
current time: 1
current time: 1
current time: 1
+ clock suite
  + adjusting clock
  Ran 1 test in 470 ms: 1 succeeded, 0 ignored, 0 failed
```

---

## Changing the Size of Sized Generators

To change the default _size_ used by [sized generators](../property-testing/built-in-generators.md#sized-generators) we can use `size` test aspect:

```scala

test("generating small list of characters") {
  check(Gen.small(Gen.listOfN(_)(Gen.alphaNumericChar))) { n =>
    ZIO.attempt(n).debug *> Sized.size.map(s => assertTrue(s == 50))
  }
} @@ TestAspect.size(50) @@ TestAspect.samples(5)
```

Sample output:

```
List(p, M)
List()
List(0, m, 5)
List(Y)
List(O, b, B, V)
+ generating small list of characters
Ran 1 test in 676 ms: 1 succeeded, 0 ignored, 0 failed
```

---

## Timing-out Tests

The `timeout` test aspect takes a duration and times out each test. If the test case runs longer than the time specified, it is immediately canceled and reported as a failure, with a message showing that the timeout was exceeded:

```scala

test("effects can be safely interrupted") {
  for {
    _ <- ZIO.attempt(println("Still going ...")).forever
  } yield assertTrue(true)
} @@ TestAspect.timeout(1.second)
```

By applying a `timeout(1.second)` test aspect, this will work with ZIO's interruption mechanism. So when we run this test, you can see a tone of print lines, and after a second, the `timeout` aspect will interrupt that.

---

## Built-in Assertions

To create `Assertion[A]` object one can use functions defined under `zio.test.Assertion`. There are already a number of useful assertions predefined like `equalTo`, `isFalse`, `isTrue`, `contains`, `throws` and more.

Using the `Assertion` type effectively often involves finding the best fitting function for the type of assumptions you would like to verify.

This list is intended to break up the available functions into groups based on the _Result type_. The types of the functions are included as well, to guide intuition.

For instance, if we wanted to assert that the fourth element of a `Vector[Int]` was a value equal to the number `5`, we would first look at assertions that operate on `Seq[A]`, with the type `Assertion[Seq[A]]`. For this example, I would select `hasAt`, as it accepts both the position into a sequence, as well as an `Assertion[A]` to apply at that position:

```scala
Assertion.hasAt[A](pos: Int)(assertion: Assertion[A]): Assertion[Seq[A]]
```

I could start by writing:

```scala

val xs = Vector(0, 1, 2, 3)

test("Fourth value is equal to 5") {
  assert(xs)(hasAt(3)(???))
}
```

The second parameter to `hasAt` is an `Assertion[A]` that applies to the third element of that sequence, so I would look for functions that operate on `A`, of the return type `Assertion[A]`.

I could select `equalTo`, as it accepts an `A` as a parameter, allowing me to supply `5`:

```scala

val xs = Vector(0, 1, 2, 3)

test("Fourth value is equal to 5") {
  assert(xs)(hasAt(3)(equalTo(5)))
}
```

Let's say this is too restrictive, and I would prefer to assert that a value is _near_ the number five, with a tolerance of two. This requires a little more knowledge of the type `A`, so I'll look for an assertion in the `Numeric` section. `approximatelyEquals` looks like what we want, as it permits the starting value `reference`, as well as a `tolerance`, for any `A` that is `Numeric`:

```scala
Assertion.approximatelyEquals[A: Numeric](reference: A, tolerance: A): Assertion[A]
```

Changing out `equalTo` with `approximatelyEquals` leaves us with:

```scala

val xs = Vector(0, 1, 2, 3)

test("Fourth value is approximately equal to 5") {
  assert(xs)(hasAt(3)(approximatelyEquals(5, 2)))
}
```

## Any

Assertions that apply to `Any` value.

| Function                                                         | Result type      | Description                                                          |
| --------                                                         | -----------      | -----------                                                          |
| `anything`                                                       | `Assertion[Any]` | Makes a new assertion that always succeeds.                          |
| `isNull`                                                         | `Assertion[Any]` | Makes a new assertion that requires a null value.                    |
| `isSubtype[A](assertion: Assertion[A])(implicit C: ClassTag[A])` | `Assertion[Any]` | Makes a new assertion that requires a value have the specified type. |
| `nothing`                                                        | `Assertion[Any]` | Makes a new assertion that always fails.                             |
| `throwsA[E: ClassTag]`                                           | `Assertion[Any]` | Makes a new assertion that requires the expression to throw.         |

## A

Assertions that apply to specific values.

| Function                                                              | Result type    | Description                                                                             |
| --------                                                              | -----------    | -----------                                                                             |
| `equalTo[A](expected: A)`                                             | `Assertion[A]` | Makes a new assertion that requires a value equal the specified value.                  |
| `hasField[A, B](name: String, proj: A => B, assertion: Assertion[B])` | `Assertion[A]` | Makes a new assertion that focuses in on a field in a case class.                       |
| `isOneOf[A](values: Iterable[A])`                                     | `Assertion[A]` | Makes a new assertion that requires a value to be equal to one of the specified values. |
| `not[A](assertion: Assertion[A])`                                     | `Assertion[A]` | Makes a new assertion that negates the specified assertion.                             |
| `throws[A](assertion: Assertion[Throwable])`                          | `Assertion[A]` | Makes a new assertion that requires the expression to throw.                            |

## Numeric

Assertions on `Numeric` types

| Function                                                      | Result type    | Description                                                                                     |
| --------                                                      | -----------    | -----------                                                                                     |
| `approximatelyEquals[A: Numeric](reference: A, tolerance: A)` | `Assertion[A]` | Makes a new assertion that requires a given numeric value to match a value with some tolerance. |
| `isNegative[A](implicit num: Numeric[A])`                     | `Assertion[A]` | Makes a new assertion that requires a numeric value is negative.                                |
| `isPositive[A](implicit num: Numeric[A])`                     | `Assertion[A]` | Makes a new assertion that requires a numeric value is positive.                                |
| `isZero[A](implicit num: Numeric[A])`                         | `Assertion[A]` | Makes a new assertion that requires a numeric value is zero.                                    |
| `nonNegative[A](implicit num: Numeric[A])`                    | `Assertion[A]` | Makes a new assertion that requires a numeric value is non negative.                            |
| `nonPositive[A](implicit num: Numeric[A])`                    | `Assertion[A]` | Makes a new assertion that requires a numeric value is non positive.                            |

## Ordering

Assertions on types that support `Ordering`

| Function                                                           | Result type    | Description                                                                                              |
| --------                                                           | -----------    | -----------                                                                                              |
| `isGreaterThan[A](reference: A)(implicit ord: Ordering[A])`        | `Assertion[A]` | Makes a new assertion that requires the value be greater than the specified reference value.             |
| `isGreaterThanEqualTo[A](reference: A)(implicit ord: Ordering[A])` | `Assertion[A]` | Makes a new assertion that requires the value be greater than or equal to the specified reference value. |
| `isLessThan[A](reference: A)(implicit ord: Ordering[A])`           | `Assertion[A]` | Makes a new assertion that requires the value be less than the specified reference value.                |
| `isLessThanEqualTo[A](reference: A)(implicit ord: Ordering[A])`    | `Assertion[A]` | Makes a new assertion that requires the value be less than or equal to the specified reference value.    |
| `isWithin[A](min: A, max: A)(implicit ord: Ordering[A])`           | `Assertion[A]` | Makes a new assertion that requires a value to fall within a specified min and max (inclusive).          |

## Iterable

Assertions on types that extend `Iterable`, like `List`, `Seq`, `Set`, `Map`, and many others.

| Function                                                                    | Result type                | Description                                                                                                                                                                           |
| --------                                                                    | -----------                | -----------                                                                                                                                                                           |
| `contains[A](element: A)`                                                   | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain the specified element. See Assertion.exists if you want to require an Iterable to contain an element satisfying an assertion. |
| `exists[A](assertion: Assertion[A])`                                        | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain an element satisfying the given assertion.                                                                                    |
| `forall[A](assertion: Assertion[A])`                                        | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain only elements satisfying the given assertion.                                                                                 |
| `hasFirst[A](assertion: Assertion[A])`                                      | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable to contain the first element satisfying the given assertion.                                                                          |
| `hasIntersection[A](other: Iterable[A])(assertion: Assertion[Iterable[A]])` | `Assertion[Iterable[A]]`   | Makes a new assertion that requires the intersection of two Iterables satisfy the given assertion.                                                                                    |
| `hasLast[A](assertion: Assertion[A])`                                       | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable to contain the last element satisfying the given assertion.                                                                           |
| `hasSize[A](assertion: Assertion[Int])`                                     | `Assertion[Iterable[A]]`   | Makes a new assertion that requires the size of an Iterable be satisfied by the specified assertion.                                                                                  |
| `hasAtLeastOneOf[A](other: Iterable[A])`                                    | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain at least one of the specified elements.                                                                                       |
| `hasAtMostOneOf[A](other: Iterable[A])`                                     | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain at most one of the specified elements.                                                                                        |
| `hasNoneOf[A](other: Iterable[A])`                                          | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain none of the specified elements.                                                                                               |
| `hasOneOf[A](other: Iterable[A])`                                           | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable contain exactly one of the specified elements.                                                                                        |
| `hasSameElements[A](other: Iterable[A])`                                    | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable to have the same elements as the specified Iterable, though not necessarily in the same order.                                        |
| `hasSameElementsDistinct[A](other: Iterable[A])`                            | `Assertion[Iterable[A]]`   | Makes a new assertion that requires an Iterable to have the same distinct elements as the other Iterable, though not necessarily in the same order.                                   |
| `hasSubset[A](other: Iterable[A])`                                          | `Assertion[Iterable[A]]`   | Makes a new assertion that requires the specified Iterable to be a subset of the other Iterable.                                                                                      |
| `isDistinct`                                                                | `Assertion[Iterable[Any]]` | Makes a new assertion that requires an Iterable is distinct.                                                                                                                          |
| `isEmpty`                                                                   | `Assertion[Iterable[Any]]` | Makes a new assertion that requires an Iterable to be empty.                                                                                                                          |
| `isNonEmpty`                                                                | `Assertion[Iterable[Any]]` | Makes a new assertion that requires an Iterable to be non empty.                                                                                                                      |

## Ordering

Assertions that apply to ordered `Iterable`s

| Function                                        | Result type              | Description                                                                 |
| --------                                        | -----------              | -----------                                                                 |
| `isSorted[A](implicit ord: Ordering[A])`        | `Assertion[Iterable[A]]` | Makes a new assertion that requires an Iterable is sorted.                  |
| `isSortedReverse[A](implicit ord: Ordering[A])` | `Assertion[Iterable[A]]` | Makes a new assertion that requires an Iterable is sorted in reverse order. |

## Seq

Assertions that operate on sequences (`List`, `Vector`, `Map`, and many others)

| Function                                      | Result type         | Description                                                                                                                |
| --------                                      | -----------         | -----------                                                                                                                |
| `endsWith[A](suffix: Seq[A])`                 | `Assertion[Seq[A]]` | Makes a new assertion that requires a given string to end with the specified suffix.                                       |
| `hasAt[A](pos: Int)(assertion: Assertion[A])` | `Assertion[Seq[A]]` | Makes a new assertion that requires a sequence to contain an element satisfying the given assertion on the given position. |
| `startsWith[A](prefix: Seq[A])`               | `Assertion[Seq[A]]` | Makes a new assertion that requires a given sequence to start with the specified prefix.                                   |

## Either

Assertions for `Either` values.

| Function                              | Result type                   | Description                                                                         |
| --------                              | -----------                   | -----------                                                                         |
| `isLeft[A](assertion: Assertion[A])`  | `Assertion[Either[A, Any]]`   | Makes a new assertion that requires a Left value satisfying a specified assertion.  |
| `isLeft`                              | `Assertion[Either[Any, Any]]` | Makes a new assertion that requires an Either is Left.                              |
| `isRight[A](assertion: Assertion[A])` | `Assertion[Either[Any, A]]`   | Makes a new assertion that requires a Right value satisfying a specified assertion. |
| `isRight`                             | `Assertion[Either[Any, Any]]` | Makes a new assertion that requires an Either is Right.                             |

## Exit/Cause/Throwable

Assertions for `Exit` or `Cause` results.

| Function                                         | Result type                 | Description                                                                                                |
| --------                                         | -----------                 | -----------                                                                                                |
| `containsCause[E](cause: Cause[E])`              | `Assertion[Cause[E]]`       | Makes a new assertion that requires a Cause contain the specified cause.                                   |
| `dies(assertion: Assertion[Throwable])`          | `Assertion[Exit[Any, Any]]` | Makes a new assertion that requires an exit value to die.                                                  |
| `failsCause[E](assertion: Assertion[Cause[E]])`  | `Assertion[Exit[E, Any]]`   | Makes a new assertion that requires an exit value to fail with a cause that meets the specified assertion. |
| `fails[E](assertion: Assertion[E])`              | `Assertion[Exit[E, Any]]`   | Makes a new assertion that requires an exit value to fail.                                                 |
| `isInterrupted`                                  | `Assertion[Exit[Any, Any]]` | Makes a new assertion that requires an exit value to be interrupted.                                       |
| `succeeds[A](assertion: Assertion[A])`           | `Assertion[Exit[Any, A]]`   | Makes a new assertion that requires an exit value to succeed.                                              |
| `hasMessage(message: Assertion[String])`         | `Assertion[Throwable]`      | Makes a new assertion that requires an exception to have a certain message.                                |
| `hasThrowableCause(cause: Assertion[Throwable])` | `Assertion[Throwable]`      | Makes a new assertion that requires an exception to have a certain cause.                                  |

## Try

| Function                                     | Result type           | Description                                                                             |
| --------                                     | -----------           | -----------                                                                             |
| `isFailure(assertion: Assertion[Throwable])` | `Assertion[Try[Any]]` | Makes a new assertion that requires a Failure value satisfying the specified assertion. |
| `isFailure`                                  | `Assertion[Try[Any]]` | Makes a new assertion that requires a Try value is Failure.                             |
| `isSuccess[A](assertion: Assertion[A])`      | `Assertion[Try[A]]`   | Makes a new assertion that requires a Success value satisfying the specified assertion. |
| `isSuccess`                                  | `Assertion[Try[Any]]` | Makes a new assertion that requires a Try value is Success.                             |

## Sum type

An assertion that applies to some type, giving a method to transform the source
type into another type, then assert a property on that projected type.

| Function                                                                                      | Result type      | Description                                                           |
| --------                                                                                      | -----------      | -----------                                                           |
| `isCase[Sum, Proj]( termName: String, term: Sum => Option[Proj], assertion: Assertion[Proj])` | `Assertion[Sum]` | Makes a new assertion that requires the sum type be a specified term. |

## Map

Assertions for `Map[K, V]`

| Function                                             | Result type            | Description                                                                                                        |
| --------                                             | -----------            | -----------                                                                                                        |
| `hasKey[K, V](key: K)`                               | `Assertion[Map[K, V]]` | Makes a new assertion that requires a Map to have the specified key.                                               |
| `hasKey[K, V](key: K, assertion: Assertion[V])`      | `Assertion[Map[K, V]]` | Makes a new assertion that requires a Map to have the specified key with value satisfying the specified assertion. |
| `hasKeys[K, V](assertion: Assertion[Iterable[K]])`   | `Assertion[Map[K, V]]` | Makes a new assertion that requires a Map have keys satisfying the specified assertion.                            |
| `hasValues[K, V](assertion: Assertion[Iterable[V]])` | `Assertion[Map[K, V]]` | Makes a new assertion that requires a Map have values satisfying the specified assertion.                          |

## String

Assertions for Strings

| Function                                   | Result type         | Description                                                                                       |
| --------                                   | -----------         | -----------                                                                                       |
| `containsString(element: String)`          | `Assertion[String]` | Makes a new assertion that requires a substring to be present.                                    |
| `endsWithString(suffix: String)`           | `Assertion[String]` | Makes a new assertion that requires a given string to end with the specified suffix.              |
| `equalsIgnoreCase(other: String)`          | `Assertion[String]` | Makes a new assertion that requires a given string to equal another ignoring case.                |
| `hasSizeString(assertion: Assertion[Int])` | `Assertion[String]` | Makes a new assertion that requires the size of a string be satisfied by the specified assertion. |
| `isEmptyString`                            | `Assertion[String]` | Makes a new assertion that requires a given string to be empty.                                   |
| `isNonEmptyString`                         | `Assertion[String]` | Makes a new assertion that requires a given string to be non empty.                               |
| `matchesRegex(regex: String)`              | `Assertion[String]` | Makes a new assertion that requires a given string to match the specified regular expression.     |
| `startsWithString(prefix: String)`         | `Assertion[String]` | Makes a new assertion that requires a given string to start with a specified prefix.              |

## Boolean

Assertions for Booleans

| Function  | Result type          | Description                                           |
| --------  | -----------          | -----------                                           |
| `isFalse` | `Assertion[Boolean]` | Makes a new assertion that requires a value be false. |
| `isTrue`  | `Assertion[Boolean]` | Makes a new assertion that requires a value be true.  |

## Option

Assertions for Optional values

| Function                             | Result type              | Description                                                                          |
| --------                             | -----------              | -----------                                                                          |
| `isNone`                             | `Assertion[Option[Any]]` | Makes a new assertion that requires a None value.                                    |
| `isSome[A](assertion: Assertion[A])` | `Assertion[Option[A]]`   | Makes a new assertion that requires a Some value satisfying the specified assertion. |
| `isSome`                             | `Assertion[Option[Any]]` | Makes a new assertion that requires an Option is Some.                               |

## Unit

Assertion for Unit

| Function | Result type       | Description                                            |
| -------- | -----------       | -----------                                            |
| `isUnit` | `Assertion[Unit]` | Makes a new assertion that requires the value be unit. |

---

## Classic Assertions

:::note
In almost all cases we encourage developers using _[smart assertions](smart-assertions.md)_ instead of [classic assertions](classic-assertions.md). They are more expressive and easier to use. So you can skip reading this section.

Only use _classic assertions_ when you know what you are doing. There are some rare cases where the smart assertions are not enough.
:::

The `assert` and its effectful counterpart `assertZIO` are the old way of asserting ordinary values and ZIO effects.

## Asserting Ordinary Values

In order to test ordinary values, we should use `assert`, like the example below:

```scala

test("sum") {
  assert(1 + 1)(Assertion.equalTo(2))
}
```

## Asserting ZIO Effects

If we are testing an effect, we should use the `assertZIO` function:

```scala

test("updating ref") {
  val value = for {
    r <- Ref.make(0)
    _ <- r.update(_ + 1)
    v <- r.get
  } yield v
  assertZIO(value)(Assertion.equalTo(1))
}
```

## The for-comprehension Style

Having this all in mind, probably the most common and also most readable way of structuring tests is to pass a for-comprehension to `test` function and yield a call to `assert` function.

```scala

test("updating ref") {
  for {
    r <- Ref.make(0)
    _ <- r.update(_ + 1)
    v <- r.get
  } yield assert(v)(Assertion.equalTo(v))
}
```

## Understanding the `test` Function

:::note
In this section we are going to learn about the internals of the `Assertion` data type. So feel free to skip this section if you are not interested.
:::

In order to understand the `Assertion` data type, let's first look at the `test` function:

```scala
def test[In](label: String)(assertion: => In)(implicit testConstructor: TestConstructor[Nothing, In]): testConstructor.Out
```

Its signature is a bit complicated and uses _path-dependent types_, but it doesn't matter. We can think of a `test` as a function from `TestResult` (or its effectful versions such as `ZIO[R, E, TestResult]` or `ZSTM[R, E, TestResult]`) to the `Spec[R, E]` data type:

```scala
def test(label: String)(assertion: => TestResult): Spec[Any, Nothing]
def test(label: String)(assertion: => ZIO[R, E, TestResult]): Spec[R, E]
```

Therefore, the function `test` needs a `TestResult`. The most common way to produce a `TestResult` is to resort to `assert` or its effectful counterpart `assertZIO`. The former one is for creating ordinary `TestResult` values and the latter one is for producing effectful `TestResult` values. Both of them accept a value of type `A` (effectful version wrapped in a `ZIO`) and an `Assertion[A]`.

## Understanding the `assert` Function

Let's look at the `assert` function:

```scala
def assert[A](expr: => A)(assertion: Assertion[A]): TestResult
``` 

It takes an expression of type `A` and an `Assertion[A]` and returns the `TestResult` which is the boolean algebra of the `AssertionResult`. Furthermore, we have an `Assertion[A]` which is capable of producing _assertion results_ on any value of type `A`. So the `assert` function can apply the expression to the assertion and produce the `TestResult`.

## Type-checker Macro

To check if the code compiles, we can use the `typeCheck` macro. It is useful when we want to test if the code compiles without running it. Here is an example of how to use it:

```scala

test("lazy list") {
  assertZIO(typeCheck(
    """
      |val lazyList: LazyList[Int] = LazyList(1, 2, 3, 4, 5)
      |lazyList.foreach(println)
      |""".stripMargin))(isRight)

} @@ TestAspect.exceptScala212
```

The `LazyCheck` introduced in Scala 2.13, so we excluded this test from Scala 2.12.

## Examples

### Example 1: Equality Assertion

Assume we have a function that concatenates two strings. One simple property of this function would be "the sum of the length of all inputs should be equal to the length of the output". Let's see an example of how we can make an assertion about this property:

```scala

test("The sum of the lengths of both inputs must equal the length of the output") {
  check(Gen.string, Gen.string) { (a, b) =>
    assert((a + b).length)(Assertion.equalTo(a.length + b.length))
  }
}
```

The syntax of assertion in the above code, is `assert(expression)(assertion)`. The first section is an expression of type `A` which is _result_ of our computation and the second one is the expected assertion of type `Assertion[A]`.

### Example 2: Field-level Assertion

There is also an easy way to test an object's data for certain assertions with `hasField` which accepts besides a name, a mapping function from object to its tested property, and `Assertion` object which will validate this property. Here our test checks if a person has at least 18 years and is not from the USA.

```scala

final case class Address(country:String, city:String)
final case class User(name:String, age:Int, address: Address)

test("Rich checking") {
  assert(
    User("Jonny", 26, Address("Denmark", "Copenhagen"))
  )(
    hasField("age", (u:User) => u.age, isGreaterThanEqualTo(18)) &&
    hasField("country", (u:User) => u.address.country, not(equalTo("USA")))
  )
}
```

What is nice about those tests is that test reporters will tell you exactly which assertion was broken. Let's say we would change `isGreaterThanEqualTo(18)` to `isGreaterThanEqualTo(40)` which will fail. Print out on the console will be a nice detailed text explaining what exactly went wrong:

```bash
[info]       User(Jonny,26,Address(Denmark,Copenhagen)) did not satisfy (hasField("age", _.age, isGreaterThanEqualTo(45)) && hasField("country", _.country, not(equalTo(USA))))
[info]       26 did not satisfy isGreaterThanEqualTo(45)
```

### Example 3: Test if a ZIO Effect Fails With a Particular Error Type

The following example shows how to test if a ZIO effect fails with a particular error type. To test if a ZIO effect fails with a particular error type, we can use the `ZIO#exit` to determine the exit type of that effect.

```scala

case class MyError(msg: String) extends Exception

val effect: ZIO[Any, MyError, Unit] = ZIO.fail(MyError("my error msg"))

test("test if a ZIO effect fails with a particular error type") {
  for {
    exit <- effect.exit
  } yield assertTrue(exit == Exit.fail(MyError("my error msg")))
}
```

The exit method on a ZIO effect returns an `Exit` value, which represents the outcome of the effect. The `Exit` value can be either `Exit.succeed` or `Exit.fail`. If the effect succeeded, the `Exit.succeed` value will contain the result of the effect. If the effect failed, the `Exit.fail` value will contain the error that caused the failure.

### Example 4: Test if a ZIO Effect Fails With a Subtype of a Particular Error Type

To test if a ZIO effect fails with a `subtype` of a particular error type, we can use the `assertZIO` function and the two `fails`, and `isSubtype` assertions from the zio-test library. The `assertZIO` function takes a ZIO effect and an assertion. The assertion is called with the result of the ZIO effect. If the assertion returns true, then the `assertZIO` will succeed, otherwise it will fail.

Assume we have these error types:

```scala
sealed trait MyError extends Exception
case class E1(msg: String) extends MyError
case class E2(msg: String) extends MyError
```

To assert if an error type is a subtype of a particular error type, we need to combine the `fails` and `isSubtype` assertions together:

```scala

Assertion.fails(isSubtype[MyError](anything))
```

Now let's look at an example:

```scala

val effect = ZIO.fail(E1("my error msg"))

test("Test if a ZIO effect fails with a MyError") {
  assertZIO(effect.exit)(fails(isSubtype[MyError](anything)))
}
```

---

## Introduction to ZIO Test Assertions

Assertions are used to make sure that the assumptions on computations are exactly what we expect them to be. They are _executable checks_ for a property that must be true in our code. Also, they can be seen as a _specification of a program_ and facilitate understanding of programs.

An `Assertion[A]` is a statement that can be used to assert the predicate of type `A => Boolean`. It is a piece of code that checks whether a value of type `A` satisfies some condition. If the condition is satisfied, the assertion passes; otherwise, it fails. We can think of the `Assertion[A]` as a function from `A` to `Boolean`:

```scala
case class Assertion[-A](arrow: TestArrow[A, Boolean]) {
  def test(value: A): Boolean = ???
  def run(value: => A): TestResult = ???
}
```

`Assertion` has a companion object with lots of predefined assertions that can be used to test values of different types. For example, the `Assertion.equalTo` takes a value of type `A` and returns an assertion that checks whether the value is equal to the given value:

```scala

def sut = 40 + 2
val assertion: Assertion[Int] = Assertion.equalTo[Int, Int](42)
assertion.test(sut) // true
```

:::note
Behind the scenes, the `Assertion` type uses a `TestArrow` type to represent the function from `A` to `Boolean`. For example, instead of using a predefined `equalTo` assertion, we can create our assertion directly from a `TestArrow`:

```scala

def sut = 40 + 2
val assertion: Assertion[Int] = Assertion(TestArrow.fromFunction(_ == 42))
assertion.test(sut) // true
```

Please note that the `TestArrow` is the fundamental building block of assertions specially the complex ones. Usually, as the end user, we do not require interacting with `TestArrow` directly. But it is good to know that it is there and how it works. We will see more about `TestArrow` in the next sections.
:::

## Built-in Assertions

The companion object of `Assertion` provides a comprehensive set of predefined assertions that can be used to test values of different types. We have a separate page for introducing the [built-in assertions](built-in-assertions.md) in ZIO Test.

## Logical Operations

As a proposition, assertions compose using logical conjunction and disjunction and can be negated:

```scala

val greaterThanZero: Assertion[Int] = Assertion.isPositive
val lessThanFive   : Assertion[Int] = Assertion.isLessThan(5)
val equalTo10      : Assertion[Int] = Assertion.equalTo(10)

val assertion: Assertion[Int] = greaterThanZero && lessThanFive || !equalTo10
```

After composing them, we can run it on any expression:

```scala

val result: TestResult = assertion.run(10)
```

## Composable Nested Assertions

Besides the logical operators, we can also combine assertions like the following to have assertions on more complex types like `Option[Int]`:

```scala
val assertion: Assertion[Option[Int]] = Assertion.isSome(Assertion.equalTo(5))

test("optional value is some(5)") {
  assert(Some(1 + 4))(assertion)
}
```

This nested assertion will pass only if the given value is `Some(5)`.

We can also combine assertions on more complex types like `Either[Int, Option[Int]]`:

```scala

test("either value is right(Some(5))") {
  assert(Right(Some(1 + 4)))(isRight(isSome(equalTo(5))))
}
```

Here we're checking deeply nested values inside an `Either` and `Option`. Because `Assertion`s compose this is not a problem. All layers are being peeled off tested for the condition until the final value is reached.

Here the expression `Right(Some(1 + 4))` is of type `Either[Any, Option[Int]]` and our assertion `isRight(isSome(equalTo(5)))` is of type `Assertion[Either[Any, Option[Int]]]`

:::note
Under the hood, the above assertion uses the `>>>` operator of `TestArrow` to make the composition of two assertions sequentially:

```scala

def isRight[A]: TestArrow[Either[Any, A], A] =
  TestArrow.fromFunction(_.toOption.get)

def isSome[A]: TestArrow[Option[A], A] =
  TestArrow.fromFunction(_.get)

def equalTo[A, B](expected: B): TestArrow[B, Boolean] =
  TestArrow.fromFunction((actual: B) => actual == expected)

val assertion: Assertion[Either[Any, Option[Int]]] = {
  val arrow: TestArrow[Either[Any, Option[Int]], Boolean] =
    isRight >>>        // Either[Any, Option[Int]] => Option[Int]
      isSome[Int] >>>  // Option[Int] => Int
      equalTo(5)       // Int => Boolean
  Assertion(arrow)
}
```

By composing an arrow of `TestArrow[Either[Any, Option[Int]], Option[Int]]` and `TestArrow[Option[Int], Boolean]` and `TestArrow[Int, Boolean]` we can create an arrow of `TestArrow[Either[Any, Option[Int]], Boolean]`. Using this technique, we can compose more arrows to create more and more complex assertions.

We can see that `TestArrow` has the same analogy as [`ZLayer`](../../contextual/zlayer.md). We are dealing with generalization of functions and composition of functions in a pure and declarative fashion, which is called "arrow" in functional programming. In other words, with `TestArrow`, we have reified the concept of a function and its composition, which allows us to manipulate functions as first-class values.

One of the benefits of reification of assertions into "arrows" is that we can write macros to generate assertions from pure Scala code. This is how the smart assertions work in ZIO Test.
:::

## Testing using Assertions

We have two types of methods for writing test assertions:
1. **[Classic Assertions](classic-assertions.md)**— This one is the classic way of asserting ordinary values (`assert`) and ZIO effects (`assertZIO`) without using macros.
2. **[Smart Assertions](smart-assertions.md)**— This is a unified syntax for asserting both ordinary values and ZIO effects using the `assertTrue` macro.

---

## Smart Assertions

The smart assertion is a simple way to assert both _ordinary values_ and _ZIO effects_. It uses the `assertTrue` function, which uses macro under the hood.

## Asserting Ordinary Values

In the following example, we assert simple ordinary values using the `assertTrue` method:

```scala

test("sum"){
  assertTrue(1 + 1 == 2)
}
```

We can assert multiple assertions inside a single `assertTrue`:

```scala
test("multiple assertions"){
  assertTrue(
    true,
    1 + 1 == 2,
    Some(1 + 1) == Some(2)
  )
}
```

## Asserting ZIO effects

The `assertTrue` method can also be used to assert ZIO effects:

```scala

test("updating ref") {
  for {
    r <- Ref.make(0)
    _ <- r.update(_ + 1)
    v <- r.get
  } yield assertTrue(v == 1)
}
```

Using `assertTrue` with for-comprehension style, we can think of testing as these three steps:

1. **Set up the test** — In this section we should setup the system under test (e.g. `Ref.make(0)`).
2. **Running the test** — Then we run the test scenario according to the test specification. (e.g `ref.update(_ + 1)`)
3. **Making assertions about the test** - Finally, we should assert the result with the right expectations (e.g. `assertTrue(v == 1)`)

## Assertion Operators

Each `assertTrue` returns a `AssertResult`, so they have the same operators as `AssertResult`. Here are some of the useful operators:

1. **`&&`** - This is the logical and operator to make sure that both assertions are true:

```scala

test("&&") {
  check(Gen.int <*> Gen.int) { case (x: Int, y: Int) =>
    assertTrue(x + y == y + x) && assertTrue(x * y == y * x)
  }
}
```

2. **||** - This is the logical or operator to make sure that at least one of the assertions is true:

```scala

suite("||")(
  test("false || true") {
    assertTrue(false) || assertTrue(true) // this will pass
  },
  test("true || false") {
    assertTrue(true) || assertTrue(false) // this will pass
  },
  test("true || true") {
    assertTrue(true) || assertTrue(true) // this will pass
  },
  test("false || false") {
    assertTrue(false) || assertTrue(false) // this will false
  },
)
```

3. **`!`** - This is the logical not operator to negate the assertion:

```scala

suite("unary !") (
    test("negate true") {
        !assertTrue(true) // this will fail
    },
    test("negate false") {
        !assertTrue(false) // this will pass
    }
)
```

4. **implies** - This is the logical implies operator to make sure that the first assertion implies the second assertion. It is equivalent to `!p || q` which is a conditional statement of the form "if p, then q" where p and q are propositions. The `==>` operator is an alias for `implies`.

```scala

suite("implies") (
  test("true implies true")(
    assertTrue(true) implies assertTrue(true) // this will pass
  ),
  test("true implies false")(
    assertTrue(true) implies assertTrue(false) // this will fail
  ),
  test("false implies true")(
    assertTrue(false) implies assertTrue(true) // this will pass
  ),
  test("false implies false")(
    assertTrue(false) implies assertTrue(false) // this will pass
  ),
)
```

The `implies` assertion is true if either the p is false or when both p and q are true:

| P     | Q     | P implies Q |
|-------|-------|-------------|
| true  | true  | true        |
| true  | false | false       |
| false | true  | true        |
| false | false | true        |

5. **iff** - This is the logical iff operator to make sure that the first assertion is true if and only if the second assertion is true. It is equivalent to `(p implies q) && (q implies p)`. The `<==>` operator is an alias for `iff`.

```scala

suite("iff") (
  test("true iff true")(
    assertTrue(true) iff assertTrue(true) // this will pass
  ),
  test("true iff false")(
    assertTrue(true) iff assertTrue(false) // this will fail
  ),
  test("false iff true")(
    assertTrue(false) iff assertTrue(true) // this will fail
  ),
  test("false iff false")(
    assertTrue(false) iff assertTrue(false) // this will pass
  )
)
```

Here is the truth table for the iff operator:

| P     | Q     | P iff Q |
|-------|-------|---------|
| true  | true  | true    |
| true  | false | false   |
| false | true  | false   |
| false | false | true    |

6. **??**- We can add a custom message to the assertion using the `??` operator. This will be useful when assertion fails, and we want to provide more information about the failure:

```scala

assertTrue(1 + 1 == 3) ?? "1 + 1 should be equal to 2"
```

## Asserting Nested Values

There are several operators designed specifically for use within the `assertTrue` macro, enhancing the ease and readability of assertions. These operators, intended exclusively for the `assertTrue` macro, leverage the `TestLens[A]` type-class to access the underlying value of the type `A`.

We use the `is` extension method inside the `assertTrue` macro to convert the given value to a `TestLens`. Now no matter how deeply nested the value is, we can access the underlying values using extension method defined for `TestLens` values:

### Testing Optional Values

There are two operators for testing optional values:

1. **`TestLens#some`** - This operator is used to peek into the `Some` value:

```scala

test("optional value is some(42)") {
  val sut: Option[Int] = Some(40 + 2)
  assertTrue(sut.is(_.some) == 42)
}
```

2. **`TestLens#anything`** - This operator is used to assert that the value is `Some`:

```scala

test("optional value is anything") {
  val sut: Option[Int] = Some(42)
  assertTrue(sut.is(_.anything))
}
```

### Testing Either Values

1. **`TestLens#right`** - This operator is used to peek into the `Right` value:

```scala

test("TestLens#right") {
  val sut: Either[Error, Int] = Right(40 + 2)
  assertTrue(sut.is(_.right) == 42)
}
```

2. **`TestLens#left`** - This operator is used to peek into the `Left` value:

```scala

case class Error(errorMessage: String)

test("TestLens#left") {
  val sut: Either[Error, Int] = Left(Error("Boom!"))
  assertTrue(sut.is(_.left).errorMessage == "Boom!")
}
```

3. **`TestLens#anything`** - This operator is used to assert that the value is `Right`:

```scala

test("TestLens#anything") {
  val sut: Either[Error, Int] = Right(42)
  assertTrue(sut.is(_.anything))
}
```

### Testing Exit Values

1. **`TestLens#success`** - This operator transforms the `Exit` value to its success type `A` if it is a `Exit.Success`, otherwise it will fail. So this can be used for asserting the success value of the `Exit`:

```scala

test("TestLens#success") {
  val sut: Exit[Error, Int] = Exit.succeed(42)
  assertTrue(sut.is(_.success) == 42)
}
```

2. **`TestLens#failure`** - This operator transforms the `Exit` value to its failure type `E` if it is a `Exit.Failure`, otherwise it will fail. So this can be used for asserting the failure value of the `Exit`:

```scala

case class Error(errorMessage: String)

test("TestLens#failure") {
  val sut: Exit[Error, Int] = Exit.fail(Error("Boom!"))
  assertTrue(sut.is(_.failure).errorMessage == "Boom!")
}
```

3. **`TestLens#die`** - This operator transforms the `Exit` value to its die type `E` if it is a `Exit.Die`, otherwise it will fail. So this can be used for asserting the die value of the `Exit`:

```scala

test("TestLens#die") {
  val sut: Exit[Error, Int] = Exit.die(new RuntimeException("Boom!"))
  assertTrue(sut.is(_.die).getMessage == "Boom!")
}
```

4. **`TestLens#cause`** - This operator transforms the `Exit` value to its underlying `Cause` value if it has one otherwise it will fail. So this can be used for asserting the cause of the `Exit`:

```scala

test("TestLens#cause") {
  for {
    exit <- ZIO.failCause(Cause.fail("Boom!")).exit
  } yield assertTrue(exit.is(_.cause) == Cause.fail("Boom!"))
}
// error: Error is already defined as case class Error
```

5. **`TestLens#interrupt`** - This operator transforms the `Exit` value to its interrupt value if it is a `Exit.Interrupt`, otherwise it will fail. So this can be used for asserting the interrupt value of the `Exit`:

```scala

test("TestLens#interrupt") {
  for {
    exit <- ZIO.sleep(5.seconds).fork.flatMap(_.interrupt)
  } yield assertTrue(exit.is(_.interrupted))
}
```

## Deeply Nested Values

Sometimes we need to test values with more than one level of nesting. There is no difference in the way we test nested values:

```scala

test("assertion of multiple nested values (TestLens#right.some)") {
  val sut: Either[Error, Option[Int]] = Right(Some(40 + 2))
  assertTrue(sut.is(_.right.some) == 42)
}
```

## Custom Assertions

Using `CustomAssertion` we can create our own custom assertions for use in `assertTrue`. We can define custom assertions using the `CustomAssertion.make` method. This method takes a partial function from the type `A` to `Either[String, B]`. If the partial function is defined for the given value, it returns `Right[B]`, otherwise it returns `Left[String]`. 

Here is an example of a custom assertion for a sealed trait and case classes:

```scala

// Define the sealed trait and case classes
sealed trait Book
case class Novel(pageCount: Int) extends Book
case class Comic(illustrations: Int) extends Book
case class Textbook(subject: String) extends Book

// Custom assertion for Book
val subject =
  CustomAssertion.make[Book] {
    case Textbook(subject) => Right(subject)
    case other => Left(s"Expected $$other to be Textbook")
  }

// Usage
suite("custom assertions")(
  test("subject assertion") {
    val book: Option[Book] = Some(Textbook("Mathematics"))
    assertTrue(book.is(_.some.custom(subject)) == "Mathematics")
  }
)
```

In the above example, we define a custom assertion for the `Book` sealed trait. The custom assertion `subject` is defined to extract the `subject` from the `Textbook` case class. So then we can assert the `subject` of the `Textbook` case class.

## More Examples

The `assertTrue` macro is designed to make it easy to write assertions in a more readable way. Most test cases can be written as when we're comparing ordinary values in Scala. However, we have a [`SmartAssertionSpec`](https://github.com/zio/zio/blob/series/2.x/test-tests/shared/src/test/scala/zio/test/SmartAssertionSpec.scala) which is a collection of examples to demonstrate the power of the `assertTrue` macro.

---

## zio.test.diff.Diff

When asserting two things are the same it's sometimes difficult to see the difference. Luckily there is a `zio.test.Diff` type-class. The purpose this type class is to output the difference between two things.

This can be one of the primitives types like `String`, `Int`, `Double`, etc. But also more complex structures like a `Map`, `List` and so-forth.

### Derive for case classes and algebraic data types

To _derive_ a type-class for a case class or a algebraic data type you can include the module `zio-test-magnolia` if it's not included already. Which includes `DeriveDiff` and `DeriveGen` as well.

To make it work you need to import the `DeriveDiff` object/trait:

```scala

```

An example of a difference output inside a test may look like this

```
     ✗ There was a difference
        Expected
        Person(
          name = "Bibi",
          nickname = Some("""Bibbo
          The
          Bibber
          Bobber"""),
          age = 300,
          pet = Pet(
            name = "The Beautiful Destroyer",
            hasBone = false,
            favoriteFoods = List("Alpha", "This is a wonderful way to live and die", "Potato", "Brucee Lee", "Potato", "Ziverge"),
            birthday = 2023-08-20T17:32:33.479852Z
          ),
          person = Some(Person(
            name = "Bibi",
            nickname = Some("""Bibbo
            The
            Bibber
            Bobber"""),
            age = 300,
            pet = Pet(
              name = "The Beautiful Destroyer",
              hasBone = false,
              favoriteFoods = List("Alpha", "This is a wonderful way to live and die", "Potato", "Brucee Lee", "Potato", "Ziverge"),
              birthday = 2023-08-20T17:32:33.479855Z
            ),
            person = None
          ))
        )
        Diff -expected +obtained
        Person(
          name = "Bibi" → "Boboo",
          nickname = Some(
            """Bibbo
            The
            Bibber
            Bobber""" → """Babbo
            The
            Bibber"""
          ),
          pet = Pet(
            name = "The Beautiful Destroyer" → "The Beautiful Crumb",
            favoriteFoods = List(
              1 = "This is a wonderful way to live and die" → "This is a wonderful \"way\" to dance and party",
              3 = "Brucee Lee",
              4 = "Potato",
              5 = "Ziverge"
            ),
            birthday = 2023-08-20T17:32:33.479852Z → -1000000000-01-01T00:00:00Z
          ),
          person = Some(
            Person(
              name = "Bibi" → "Boboo",
              nickname = Some(
                """Bibbo
                The
                Bibber
                Bobber""" → """Babbo
                The
                Bibber"""
              ),
              pet = Pet(
                name = "The Beautiful Destroyer" → "The Beautiful Crumb",
                favoriteFoods = List(
                  1 = "This is a wonderful way to live and die" → "This is a wonderful \"way\" to dance and party",
                  3 = "Brucee Lee",
                  4 = "Potato",
                  5 = "Ziverge"
                ),
                birthday = 2023-08-20T17:32:33.479855Z → -1000000000-01-01T00:00:00Z
              )
            )
          )
        )
      p1 == p2
      p1 = Person(
        name = "Boboo",
        nickname = Some("""Babbo
        The
        Bibber"""),
        age = 300,
        pet = Pet(
          name = "The Beautiful Crumb",
          hasBone = false,
          favoriteFoods = List("Alpha", "This is a wonderful \"way\" to dance and party", "Potato"),
          birthday = -1000000000-01-01T00:00:00Z
        ),
        person = Some(Person(
          name = "Boboo",
          nickname = Some("""Babbo
          The
          Bibber"""),
          age = 300,
          pet = Pet(
            name = "The Beautiful Crumb",
            hasBone = false,
            favoriteFoods = List("Alpha", "This is a wonderful \"way\" to dance and party", "Potato"),
            birthday = -1000000000-01-01T00:00:00Z
          ),
          person = None
        ))
      )
```

### Custom types

For more custom types you could provide type-class instances your self by implementing the `zio.test.diff.Diff` type-class.

```scala
// somewhere defined in your domain package
case class Percentage(repr: Int)

implicit val diffPercentage: Diff[Percentage] = Diff[Double].contramap(_.repr)
```

### Be wary of `LowPriDiff`

One thing to note that there is a trait `LowPriDiff` which is stacked on the companion object of `zio.test.diff.Diff`. There is lower priority type-class instance defined at `LowerPriDiff` which is a fallback for `AnyVal`. It's defined as `implicit def anyValDiff[A <: AnyVal]: Diff[A] = anyDiff[A]`, so if some custom types mess up your diff, you might want to check on this topic.

---

## Dynamic Test Generation

Tests in ZIO are dynamic. Meaning that they are not required to be statically defined at compile time. They can be generated at runtime effectfully.

Assume we have implemented the `add` operator which adds two numbers:

```scala
def add(a: Int, b: Int): Int = ???
```

We want to test this function using the following test data inside the `resources` directory:

```scala title="src/test/resources/test-data.csv"
0, 0, 0
1, 0, 1
0, 1, 1
0, -1, -1
-1, 0, -1
1, 1, 2
1, -1, 0
-1, 1, 0
```

Let's load it and create a bunch of tests using this test data:

```scala

def loadTestData: Task[List[((Int, Int), Int)]] =
  ZIO.attemptBlocking(
    scala.io.Source
      .fromResource("test-data.csv")
      .getLines()
      .toList
      .map(_.split(',').map(_.trim))
      .map(i => ((i(0).toInt, i(1).toInt), i(2).toInt))
  )
  
def makeTest(a: Int, b: Int)(expected: Int): Spec[Any, Nothing] =
  test(s"test add($a, $b) == $expected") {
    assertTrue(add(a, b) == expected)
  }

def makeTests: ZIO[Any, Throwable, List[Spec[Any, Nothing]]] =
  loadTestData.map { testData =>
    testData.map { case ((a, b), expected) =>
      makeTest(a, b)(expected)
    }
  }
```

Now we are ready to run all generated tests:

```scala

object AdditionSpec extends ZIOSpecDefault {
  override def spec = suite("add")(makeTests)
}
```

Here is the test runner's output:

```scala
+ add
  + test add(0, 0) == 0
  + test add(1, 0) == 1
  + test add(0, -1) == -1
  + test add(0, 1) == 1
  + test add(-1, 1) == 0
  + test add(1, -1) == 0
  + test add(1, 1) == 2
  + test add(-1, 0) == -1
8 tests passed. 0 tests failed. 0 tests ignored.
```

---

## Introduction to ZIO Test

**ZIO Test** is a zero dependency testing library that makes it easy to test effectual programs. In **ZIO Test**, all tests are immutable values and tests are tightly integrated with ZIO, so testing effectual programs is as natural as testing pure ones. 

## Motivation

We can easily assert ordinary values and data types to test them:

```scala

assert(1 + 2 == 2 + 1)
assert("Hi" == "H" + "i")

case class Point(x: Long, y: Long)
assert(Point(5L, 10L) == Point.apply(5L, 10L))
```

What about functional effects? Can we assert two effects using ordinary scala assertion to test whether they have the same functionality? As we know, a functional effect, like `ZIO`, describes a series of computations. Unfortunately, we can't assert functional effects without executing them. If we assert two `ZIO` effects, e.g. `assert(expectedEffect == actualEffect)`, the result says nothing about whether these two effects behave similarly and produce the same result or not. Instead, we should `unsafeRun` each one and assert their results.

Let's say we have a random generator effect, and we want to ensure that the output is bigger than zero, so we should `unsafeRun` the effect and assert the result:

```scala

val random = Unsafe.unsafe { implicit unsafe =>
  Runtime.default.unsafe.run(
    Random.nextIntBounded(10)
  ).getOrThrowFiberFailure()
}

assert(random >= 0)
```

Testing effectful programs is difficult since we should use many `unsafeRun` methods. Also, we might need to make sure that the test is non-flaky. In these cases, running `unsafeRun` multiple times is not straightforward. We need a testing framework that treats effects as _first-class values_. So this is the primary motivation for creating the ZIO Test library.

## How ZIO Test was designed

We designed ZIO Test around the idea of _making tests first-class objects_. This means that tests (and other concepts, like assertions) become ordinary values that can be passed around, transformed, and composed.

This approach allows for greater flexibility compared to some other testing frameworks, where tests and additional logic around tests had to be put into callbacks so that framework could make use of them.

As a result, this approach is also better suited to other `ZIO` concepts like `Scope`, which can only be used within a scoped block of code. This also created a mismatch between `BeforeAll`, `AfterAll` callback-like methods when there were resources that should be opened and closed during test suite execution.

Another thing worth pointing out is that tests being values are also effects. Implications of this design are far-reaching:

1. First, the well-known problem of testing asynchronous value is gone. Whereas in other frameworks we have to somehow "run" our effects and at best wrap them in `scala.util.Future` because blocking would eliminate running on ScalaJS, ZIO Test expects us to create `ZIO` objects. There is no need for indirect transformations from one wrapping object to another.

2. Second, because our tests are ordinary `ZIO` values, we don't need to turn to a testing framework for things like retries, timeouts, and resource management. We can solve all those problems with the full richness of functions that `ZIO` exposes.

---

## Installing ZIO Test

In order to use ZIO Test, we need to add the required configuration in our SBT settings:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-test"          % "2.1.26" % Test,
  "dev.zio" %% "zio-test-sbt"      % "2.1.26" % Test,
  "dev.zio" %% "zio-test-magnolia" % "2.1.26" % Test
)
```

If our SBT version is older than 1.8.0, we also need to add the test framework manually:

```scala
testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework")
```

**NOTE**: In order to use the live version of a service in our tests, we can use some new helpful test aspects e.g `withLiveClock`, `withLiveConsole`, `withLiveRandom`, `withLiveSystem`, etc.

---

## Integrating ZIO Test with JUnit

Unit testing is an essential practice in software development, enabling developers to validate the correctness and reliability of their code. JUnit, a widely adopted testing framework, has emerged as a standard choice for Java applications. With its robust features and extensive ecosystem, JUnit simplifies the process of writing and executing tests, empowering developers to deliver high-quality software.

In this section, we will explore the integration of ZIO Test, a powerful testing library for functional programming in Scala, with JUnit. By combining the strengths of both frameworks, developers can efficiently test ZIO-based applications under different build tools and IDEs.

To streamline the testing process, a custom JUnit runner is provided specifically for running ZIO Test specifications. Thus, we can conduct testing of ZIO specs within alternative build tools, such as Maven, Gradle, Bazel, and various integrated development environments (IDEs).

By adding the necessary dependency definition to the build tool, developers can effortlessly incorporate the ZIO Test JUnit runner:

```scala
libraryDependencies += "dev.zio" %% "zio-test-junit" % zioVersion % "test"
```

To make our spec appear as a JUnit test to build tools and IDEs, we can simply extend `zio.test.junit.JUnitRunnableSpec`:

```scala

object MySpec extends JUnitRunnableSpec {
  def spec = suite("MySpec")(
    test("test") {
      for {
        _ <- ZIO.unit
      } yield assertCompletes
    }
  )
}
```

Now, we can run our spec from the command line by running `sbt test`:

```bash
sbt:zio-quickstart-junit> test
+ MySpec
  + test
1 tests passed. 0 tests failed. 0 tests ignored.

Executed in 215 ms

[info] Completed tests
[success] Total time: 1 s, completed Jun 13, 2023, 4:39:27 PM
```

Or we can convert `MySpec` object to a scala `class` and annotate it with `@RunWith(classOf[ZTestJUnitRunner])`:

```scala

@RunWith(classOf[ZTestJUnitRunner])
class MySpec extends ZIOSpecDefault {
  def spec = suite("MySpec")(
    test("test") {
      for {
        _ <- ZIO.unit
      } yield assertCompletes
    }
  )
}
```

To run the above test using `sbt test` we also need to add the following line to our `build.sbt`:

```scala
libraryDependencies += "com.github.sbt" % "junit-interface" % "0.13.3" % Test
```

Now, we can run our spec by running `sbt test` from the command line:

```bash
sbt:zio-quickstart-junit> test
+ MySpec
  + test
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 1 s, completed Jun 13, 2023, 4:37:32 PM
```

To see practical examples in action, we encourage you to check out [the example code base](https://github.com/zio/zio-quickstarts/tree/master/zio-quickstart-junit-integration) provided in the [ZIO Quickstarts](https://github.com/zio/zio-quickstarts/) project on GitHub. This code base will help you dive deeper into testing ZIO specs using JUnit within different build tools and IDEs, enabling you to enhance the quality and stability of your ZIO applications. Happy testing!

---

## Built-in Generators

In the companion object of the `Gen` data type, there are tons of generators for various data types.

## Primitive Types Generators

ZIO Test provides generators for primitive types such as `Gen.int`, `Gen.string`, `Gen.boolean`, `Gen.float`, `Gen.double`, `Gen.bigInt`, `Gen.byte`, `Gen.bigDecimal`, `Gen.long`, `Gen.char`, and `Gen.short`.

Let's create an `Int` generator:

```scala

val intGen: Gen[Any, Int] = Gen.int
```

## Character Generators

In addition to `Gen.char`, ZIO Test offers a variety of specialized character generators:

* `Gen.alphaChar` — e.g. `Z, z, A, t, o, e, K, E, y, N`
* `Gen.alphaNumericChar` — e.g. `b, O, X, B, 4, M, k, 9, a, p`
* `Gen.asciiChar` — e.g. `, >, , , , 2, k, , , `
* `Gen.unicodeChar` — e.g. `了, , 옷, 嗀, , 뮲, ﹓, 癮, , ᜣ)`
* `Gen.numericChar` — e.g. `1, 0, 1, 5, 6, 9, 4, 4, 5, 2`
* `Gen.printableChar` — e.g. `H, J, (, Q, n, g, 4, G, 9, l`
* `Gen.whitespaceChars` — e.g. `, ,  , , ,  ,  ,  ,  , `
* `Gen.hexChar` — e.g. `3, F, b, 5, 9, e, 2, 8, b, e`
* `Gen.hexCharLower` — e.g. `f, c, 4, 4, c, 2, 5, 4, f, 3`
* `Gen.hexCharUpper` — e.g. `4, 8, 9, 8, C, 9, F, A, E, C`

## String Generators

Besides the primitive string generator, `Gen.string`, ZIO Test also provides the following specialized generators:

1. `Gen.stringBounded` — A generator of strings whose size falls within the specified bounds:

  ```scala mdoc:compile-only
  Gen.stringBounded(1, 5)(Gen.alphaChar)
    .runCollectN(10)
    .debug
  // Sample Output: List(b, YJXzY, Aro, y, WMPbj, Abxt, kJep, LKN, kUtr, xJ)
  ```

2. `Gen.stringN` — A generator of strings of fixed size:

  ```scala mdoc:compile-only
  Gen.stringN(5)(Gen.alphaChar)
    .runCollectN(10)
    .debug
  // Sample Output: List(BuywQ, tXCEy, twZli, ffLwI, BPEbz, OKYTi, xeDJW, iDUVn, cuMCr, keQAA)
  ```

3. `Gen.string1` — A generator of strings of at least one character.
4. `Gen.alphaNumericString` — A generator of alphanumeric characters.
5. `Gen.alphaNumericStringBounded` — A generator of alphanumeric strings whose size falls within the specified bounds.
6. `Gen.iso_8859_1` — A generator of strings that can be encoded in the ISO-8859-1 character set.
7. `Gen.asciiString` — A generator of US-ASCII characters.

## Generating Fixed Values

1. `Gen.const` — A constant generator of the specified value.

  ```scala mdoc:compile-only
  Gen.const(true).runCollectN(5)
  // Output: List(true, true, true, true, true)
  ```

2. `Gen.constSample` — A constant generator of the specified sample:

  ```scala mdoc:compile-only
   Gen.constSample(Sample.noShrink(false)).runCollectN(5)
  // Output: List(true, true, true, true, true)
  ```

3. `Gen.unit` — A constant generator of the unit value.

4. `Gen.throwable` — A generator of throwables.

Note that there is an empty generator called `Gen.empty`, which generates no values and returns nothing. We can think of that as a generator of empty stream, `Gen(Stream.empty)`.

## Generating from Fixed Values

1. `Gen.elements` — Constructs a non-deterministic generator that only generates randomly from the fixed values:

```scala

Gen.elements(
  DayOfWeek.MONDAY,
  DayOfWeek.TUESDAY,
  DayOfWeek.WEDNESDAY,
  DayOfWeek.THURSDAY,
  DayOfWeek.FRIDAY,
  DayOfWeek.SATURDAY,
  DayOfWeek.SUNDAY
).runCollectN(3).debug
// Sample Output: List(WEDNESDAY, THURSDAY, SUNDAY)
```

2. `Gen.fromIterable` — Constructs a deterministic generator that only generates the specified fixed values:

```scala
Gen.fromIterable(List("red", "green", "blue"))
  .runCollectN(10)
  .debug
```

## Collection Generators

ZIO Test has generators for collection data types such as _sets_, _lists_, _vectors_, _chunks_, and _maps_. These data types share similar APIs. The following example illustrates how the generator of sets works:

```scala
// A sized generator of sets
Gen.setOf(Gen.alphaChar)
// Sample Output: Set(Y, M, c), Set(), Set(g, x, Q), Set(s), Set(f, J, b, R)

// A sized generator of non-empty sets
Gen.setOf1(Gen.alphaChar)  
// Sample Output: Set(Y), Set(L, S), Set(i), Set(H), Set(r, Z, z)

// A generator of sets whose size falls within the specified bounds.
Gen.setOfBounded(1, 3)(Gen.alphaChar)
// Sample Output: Set(Q), Set(q, J), Set(V, t, h), Set(c), Set(X, O)

// A generator of sets of the specified size.
Gen.setOfN(2)(Gen.alphaChar)
// Sample Output: Set(J, u), Set(u, p), Set(i, m), Set(b, N), Set(B, Z)
```

## Bounded Generator

The `Gen.bounded` constructor is a generator whose size falls within the specified bounds:

```scala
Gen.bounded(2, 5)(Gen.stringN(_)(Gen.alphaChar))
  .runCollectN(5)
```

## Suspended Generator

The `Gen.suspend` constructs a generator lazily. This is useful to avoid infinite recursion when creating generators that refer to themselves.

## Unfold Generator

The `unfoldGen` takes the initial state and depending on the previous state, it determines what will be the next generated value:

```scala
def unfoldGen[R, S, A](s: S)(f: S => Gen[R, (S, A)]): Gen[R, List[A]]
```

Assume we want to test the built-in scala stack (`scala.collection.mutable.Stack`). One way to do that is to create an acceptable series of push and pop commands, and then check that the stack doesn't throw any exception by executing these commands:

```scala
sealed trait Command
case object Pop                    extends Command
final case class Push(value: Char) extends Command

val genPop:  Gen[Any, Command]    = Gen.const(Pop)
def genPush: Gen[Any, Command] = Gen.alphaChar.map(Push)

val genCommands: Gen[Any, List[Command]] =
  Gen.unfoldGen(0) { n =>
    if (n <= 0)
      genPush.map(command => (n + 1, command))
    else
      Gen.oneOf(
        genPop.map(command => (n - 1, command)),
        genPush.map(command => (n + 1, command))
      )
  }
```

We are now ready to test the generated list of commands:

```scala

test("unfoldGen") {
  check(genCommands) { commands =>
    val stack = scala.collection.mutable.Stack.empty[Int]
    commands.foreach {
      case Pop => stack.pop()
      case Push(value) => stack.push(value)
    }
    assertCompletes
  }
}
```

## From a ZIO Effect

1. `Gen.fromZIO`

  ```scala mdoc:compile-only
  val gen: Gen[Any, Int] = Gen.fromZIO(Random.nextInt) 
  ```

2. `Gen.fromZIOSample`

  ```scala mdoc:compile-only
  val gen: Gen[Any, Int] =
    Gen.fromZIOSample(
      Random.nextInt.map(Sample.shrinkIntegral(0))
    )
  ```

## From a Random Effect

3. `Gen.fromRandom` — Constructs a generator from a function that uses randomness:

  ```scala mdoc:compile-only
  val gen: Gen[Any, Int] = Gen.fromRandom(_.nextInt) 
  ```

4. `Gen.fromRandomSample` — Constructs a generator from a function that uses randomness to produce a sample:

  ```scala mdoc:compile-only
  val gen: Gen[Any, Int] =
    Gen.fromRandomSample(
      _.nextIntBounded(20).map(Sample.shrinkIntegral(0))
    )
  ```

## Uniform and Non-uniform Generators

1. `Gen.uniform` — A generator of uniformly distributed doubles between [0, 1].

2. `Gen.weighted` — A generator which chooses one of the given generators according to their weights. For example, the following generator will generate 90% true and 10% false values:

  ```scala mdoc:compile-only
  val trueFalse = Gen.weighted((Gen.const(true), 9), (Gen.const(false), 1))
  trueFalse.runCollectN(10).debug
  // Sample Output: List(false, false, false, false, false, false, false, false, true, false)
  ```

3. `Gen.exponential` — A generator of exponentially distributed doubles with mean `1`:

  ```scala mdoc:compile-only
  Gen.exponential.map(x => math.round(x * 100) / 100.0)
    .runCollectN(10)
    .debug
  // Sample Output: List(0.22, 3.02, 1.96, 1.13, 0.81, 0.92, 1.7, 1.47, 1.55, 0.46)
  ```

## Generating Date/Time Types

| Date/Time Types            | Generators           |
|----------------------------|----------------------|
| `java.time.DayOfWeek`      | `Gen.dayOfWeek`      |
| `java.time.Month`          | `Gen.month`          |
| `java.time.Year`           | `Gen.year`           |
| `java.time.Instant`        | `Gen.instant`        |
| `java.time.MonthDay`       | `Gen.monthDay`       |
| `java.time.YearMonth`      | `Gen.yearMonth`      |
| `java.time.ZoneId`         | `Gen.zoneId`         |
| `java.time.ZoneOffset`     | `Gen.zoneOffset`     |
| `java.time.ZonedDateTime`  | `Gen.zonedDateTime`  |
| `java.time.OffsetTime`     | `Gen.offsetTime`     |
| `java.time.OffsetDateTime` | `Gen.offsetDateTime` |
| `java.time.Period`         | `Gen.period`         |
| `java.time.LocalDate`      | `Gen.localDate`      |
| `java.time.LocalDateTime`  | `Gen.localDateTime`  |
| `java.time.LocalTime`      | `Gen.localTime`      |
| `zio.duration.Duration`    | `Gen.finiteDuration` |

## Function Generators

To test some properties, we need to generate functions. There are two types of function generators:

1. `Gen.function` — It takes a generator of type `B` and produces a generator of functions from `A` to `B`:

  ```scala
  def function[R, A, B](gen: Gen[R, B]): Gen[R, A => B]
  ```

Two `A` values will be considered to be equal, and thus will be guaranteed to generate the same `B` value, if they have the same
`hashCode`.

2. `Gen.functionWith` — It takes a generator of type `B` and also a hash function for `A` values, and produces a generator of functions from `A` to `B`:

  ```scala
  def functionWith[R, A, B](gen: Gen[R, B])(hash: A => Int): Gen[R, A => B]
  ```

Two `A` values will be considered to be equal, and thus will be guaranteed to generate the same `B` value, if they have the same hash. This is useful when `A` does not implement `hashCode` in a way that is consistent with equality.

Accordingly, ZIO Test provides a variety of function generators for `Function2`, `Function3`, ..., and also the `PartialFunction`:

* `Gen.function2` —  Gen[R, C] => Gen[R, (A, B) => C]
* `Gen.functionWith2` — Gen[R, B] => ((A, B) => Int) => Gen[R, (A, B) => C]
* `Gen.partialFunction` — Gen[R, B] => Gen[R, PartialFunction[A, B]]
* `Gen.partialFunctionWith` — Gen[R, B] => (A => Int) => Gen[R, PartialFunction[A, B]]

Let's write a test for `ZIO.foldLeft` operator. This operator has the following signature:

```scala
def foldLeft[R, E, S, A](in: => Iterable[A])(zero: => S)(f: (S, A) => ZIO[R, E, S]): ZIO[R, E, S]
```

We want to test the following property:

```scala
∀ (in, zero, f) => ZIO.foldLeft(in)(zero)(f) == ZIO(List.foldLeft(in)(zero)(f))
```

To test this property, we have an input of type `(Int, Int) => Int`. So we need a Function2 generator of integers:

```scala
val func2: Gen[Any, (Int, Int) => Int] = Gen.function2(Gen.int)
```

Now we can test this property:

```scala

test("ZIO.foldLeft should have the same result with List.foldLeft") {
  check(Gen.listOf(Gen.int), Gen.int, func2) { case (in, zero, f) =>
    assertZIO(
      ZIO.foldLeft(in)(zero)((s, a) => ZIO.attempt(f(s, a)))
    )(Assertion.equalTo(
      in.foldLeft(zero)((s, a) => f(s, a)))
    )
  }
}
```

## Generating ZIO Values

1. Successful effects (`Gen.successes`):

  ```scala mdoc:compile-only
  val gen: Gen[Any, UIO[Int]] = Gen.successes(Gen.int(-10, 10))
  ```

2. Failed effects (`Gen.failures`):

  ```scala mdoc:compile-only
  val gen: Gen[Any, IO[String, Nothing]] = Gen.failures(Gen.string)
  ```

3. Died effects (`Gen.died`):

  ```scala mdoc:compile-only
  val gen: Gen[Any, UIO[Nothing]] = Gen.died(Gen.throwable)
  ```

4. Cause values (`Gen.causes`):

  ```scala mdoc:compile-only
  val causes: Gen[Any, Cause[String]] = 
    Gen.causes(Gen.string, Gen.throwable)
  ```

5. Chained effects (`Gen.chained`, `Gen.chainedN`): A generator of effects that are the result of chaining the specified effect with itself a random number of times.

Let's see some example of chained ZIO effects:

  ```scala mdoc:compile-only

  val effect1 = ZIO(2).flatMap(x => ZIO(x * 2))
  val effect2 = ZIO(1) *> ZIO(2)
  ```

By using `Gen.chained` or `Gen.chainedN` generator, we can create generators of chained effects:

  ```scala mdoc:compile-only
  val chained : Gen[Any, ZIO[Any, Nothing, Int]] = 
    Gen.chained(Gen.successes(Gen.int))
    
  val chainedN: Gen[Any, ZIO[Any, Nothing, Int]] = 
    Gen.chainedN(5)(Gen.successes(Gen.int))
  ```

6. Concurrent effects (`Gen.concurrent`): A generator of effects that are the result of applying concurrency combinators to the specified effect that are guaranteed not to change its value.

  ```scala mdoc:compile-only
  val random  : Gen[Any, UIO[Int]] = Gen.successes(Gen.int).flatMap(Gen.concurrent)
  val constant: Gen[Any, UIO[Int]]    = Gen.concurrent(ZIO(3))
  ```

7. Parallel effects (`Gen.parallel`): A generator of effects that are the result of applying parallelism combinators to the specified effect that are guaranteed not to change its value.

  ```scala mdoc:compile-only
  val random: Gen[Any, UIO[String]] =
    Gen.successes(Gen.string).flatMap(Gen.parallel)
    
  val constant: Gen[Any, UIO[String]] =
    Gen.parallel(ZIO("Hello"))
  ```

## Generating Compound Types

1. tuples — We can combine generators using for-comprehension syntax and tuples:

  ```scala mdoc:compile-only
  val tuples: Gen[Any, (Int, Double)] =
    for {
      a <- Gen.int
      b <- Gen.double
    } yield (a, b)
  ```

2. `Gen.oneOf` — It takes variable number of generators and select one of them:

  ```scala mdoc:compile-only
  sealed trait Color
  case object Red extends Color
  case object Blue extends Color
  case object Green extends Color
  
  Gen.oneOf(Gen.const(Red), Gen.const(Blue), Gen.const(Green))
  // Sample Output: Green, Green, Red, Green, Red
  ```

4. `Gen.option` — A generator of _optional_ values:

  ```scala mdoc:compile-only
  val intOptions: Gen[Any, Option[Int]] = Gen.option(Gen.int)
  val someInts:   Gen[Any, Option[Int]] = Gen.some(Gen.int)
  val nons:       Gen[Any, Option[Nothing]]     = Gen.none
  ```

3. `Gen.either` — A generator of _either_ values:

  ```scala mdoc:compile-only
  val char: Gen[Any, Either[Char, Char]] =
    Gen.either(Gen.numericChar, Gen.alphaChar)
  ```

4. `Gen.collectAll` — Composes the specified generators to create a _cartesian product of elements_ with the specified function:

  ```scala mdoc:compile-only
  val gen: ZIO[Any, Nothing, List[List[Int]]] =
    Gen.collectAll(
      List(
        Gen.fromIterable(List(1, 2)),
        Gen.fromIterable(List(3)),
        Gen.fromIterable(List(4, 5))
      )
    ).runCollect
  // Output:
  // List(
  //  List(1, 3, 4),
  //  List(1, 3, 5),
  //  List(2, 3, 4),
  //  List(2, 3, 5)
  //)
  ```

5. `Gen.concatAll` — Combines the specified deterministic generators to return a new deterministic generator that generates all the values generated by the specified generators:

  ```scala mdoc:compile-only
  val gen: ZIO[Any, Nothing, List[Int]] =
    Gen.concatAll(
      List(
        Gen.fromIterable(List(1, 2)),
        Gen.fromIterable(List(3)),
        Gen.fromIterable(List(4, 5))
      )
    ).runCollect
  // Output: List(1, 2, 3, 4, 5)
  ```

## Sized Generators

1. `Gen.sized` — A sized generator takes a function from `Int` to `Gen[R, A]` and creates a generator by applying a size to that function:

  ```scala mdoc:compile-only
  Gen.sized(Gen.int(0, _))
    .runCollectN(10)
    .provideCustomLayer(Sized.live(5))
    .debug
  // Sample Output: List(5, 4, 1, 2, 0, 4, 2, 0, 1, 2)
  ```

2. `Gen.size` — A generator which accesses the _size_ from the environment and generates that:

  ```scala mdoc:compile-only
  Gen.size
    .runCollectN(5)
    .provideCustomLayer(Sized.live(100))
    .debug
  // Output: List(100, 100, 100, 100, 100)
  ```

There are also three sized generators, named _small_, _medium_ and _large_, that use an exponential distribution of size values:

1. `Gen.small` — The values generated will be strongly concentrated towards the lower end of the range but a few larger values will still be generated:

  ```scala mdoc:compile-only
  Gen.small(Gen.const(_))
    .runCollectN(10)
    .provideCustomLayer(Sized.live(1000))
    .debug
  // Output: List(6, 39, 73, 3, 57, 51, 40, 12, 110, 46)
  ```

4. `Gen.medium` — The majority of sizes will be towards the lower end of the range but some larger sizes will be generated as well:

  ```scala mdoc:compile-only
  Gen.medium(Gen.const(_))
    .runCollectN(10)
    .provideCustomLayer(Sized.live(1000))
    .debug
  // Output: List(93, 42, 58, 228, 42, 5, 12, 214, 106, 79)
  ```

5. `Gen.large` — Uses a uniform distribution of size values. A large number of larger sizes will be generated:

  ```scala mdoc:compile-only
  Gen.large(Gen.const(_))
    .runCollectN(10)
    .provideCustomLayer(Sized.live(1000))
    .debug
  // Output: List(797, 218, 596, 278, 301, 779, 165, 486, 695, 788)
  ```

---

## Getting Started With Property Checking

The fundamental idea behind property checking is to test the properties of the target function using random inputs.

So to test a system using property checking, two things are required:

1. Properties
2. Generators

A property of a system is a predicate that is always true regardless of the system's input. For example, the addition of two numbers is commutative. So it doesn't matter what numbers we pass to the addition function, for any pair of `a` and `b`, the result of `add(a, b)` is always the same as `add(b, a)`:

```scala
def add(a: Int, b: Int): Int = ???

def is_add_commutative(a: Int, b: Int): Boolean =
  add(a, b) == add(b, a)
```

The `is_add_commutative` predicate takes two inputs and checks if the `add` function is commutative or not. To check this property, we need some random integer pairs. This is where generators come in.

The `Gen[A]` data type is used to generate random values of type `A`. ZIO Test provides numerous `Gen` instances for common types:

```scala

val intGen: Gen[Any, Int] = Gen.int
val stringGen: Gen[Sized, String] = Gen.string
```

It is also composable, so we can combine them to generate random values of more complex types:

```scala
val stringIntGen: Gen[Sized, (String, Int)] = stringGen <*> intGen

case class Person(name: String, age: Int)
val personGen: Gen[Sized, Person] = stringIntGen.map(Person.tupled)
```

ZIO Test provides the `check` function for this purpose. It takes a list of generators and provides them to another taken function, which is a property checker:

```scala
def property[T1, T2](input1: T1, input2: T2, ...): Boolean = ???

val input1Gen: Gen[_, T1] = ???
val input2Gen: Gen[_, T2] = ???

check(input1Gen, input2Gen, ...) { (input1, input2, ...) =>
  assertTrue(property(input1, input2, ...))   
}
```

In our example, the `is_add_commutative` predicate takes two inputs. So we need to pass two generators of type `Int` to the `check` function:

```scala
def add(a: Int, b: Int): Int = ???

test("add is commutative") {
  check(Gen.int, Gen.int) { (a, b) =>
    assertTrue(add(a, b) == add(b, a))
  }
}
```

## Number of Samples

In the previous example, we used `check` to test if the `add` function is commutative. In other words, we try to generate samples of random pairs of integers and try to falsify the `is_add_commutative` predicate. If we find a pair of integers that falsifies the predicate, then we know that the property is violated.

By default, the `check` function, try to generate 200 samples. We can change this by using the `sample` test aspect:

```scala

object AdditionSpec extends ZIOSpecDefault {
  def spec =
    test("add is commutative") {
      check(Gen.int, Gen.int) { (a, b) =>
        assertTrue(add(a, b) == add(b, a))
      }
    } @@ TestAspect.samples(10)
}
```

To debug the test, we added a `println` statement inside the `check` function to see the generated samples.

---

## How Generators Work?

A `Gen[R, A]` represents a generator of values of type `A`, which requires an environment `R`. The `Gen` data type is the base functionality for generating test data for property-based testing. We use them to produce deterministic and non-deterministic (PRNG) random values.

It is encoded as a stream of optional samples:

```scala
case class Gen[-R, +A](sample: ZStream[R, Nothing, Option[Sample[R, A]]])
```

Before deep into the generators, let's see what property-based testing is and what problem it solves in the testing world.

## How Generators Work?

We can think of `Gen[R, A]` as a `ZStream[R, Nothing, A]`. For example, the `Gen.int` is a stream of random integers `ZStream.fromZIO(Random.nextInt)`.

To find out how a generator works, let's take a look at the following snippet.  It shows how the `Gen` data type is implemented.

:::caution
Although it doesn't provide the exact implementation, this condensed edition of the "Gen" data type is sufficient to grasp how generators operate.

For instance, we don't use a [pseudo-random generator](#random-generators-are-deterministic-by-default) throughout the following implementation. We haven't encoded the [shrinking algorithm](shrinking.md), either.
:::

```scala

case class Gen[R, A](sample: ZStream[R, Nothing, A]) {
  def map[B](f: A => B): Gen[R, B] = Gen(sample.map(f))

  def flatMap[R1 <: R, B](f: A => Gen[R1, B]): Gen[R1, B] = ???

  def runCollect: ZIO[R, Nothing, List[A]] = sample.runCollect.map(_.toList)
}

object Gen {
  // A constant generator of the specified value.
  def const[A](a: => A): Gen[Any, A] = 
    Gen(ZStream.succeed(a))
  
  // A random generator of integers.
  def int: Gen[Any, Int] = 
    Gen(ZStream.fromZIO(Random.nextInt))
  def int(min: Int, max: Int): Gen[Any, Int] = 
    ???
  
  // A random generator of specified values.
  def elements[A](as: A*): Gen[Any, A] = 
    if (as.isEmpty) Gen(ZStream.empty) else int(0, as.length - 1).map(as)
  
  // A constant generator of fixed values.
  def fromIterable[A](xs: Iterable[A]): Gen[Any, A] = 
    Gen(ZStream.fromIterable(xs))
}

Gen.const(42).runCollect.debug
// Output: List(42)

Gen.int.runCollect.debug
// Output: List(82) or List(3423) or List(-352) or ...

Gen.elements(1, 2, 3).runCollect.debug
// Output: List(1) or List(2) or List(3)

Gen.fromIterable(List(1, 2, 3))
// Output: List(1, 2, 3)
```

So we can see that the `Gen` data type is nothing more than a stream of random/constant values.

## Two Types of Generators

We have two types of generators:

1. **Deterministic Generators**— Generators that produce constant fixed values, such as `Gen.empty`, `Gen.const(42)`, and `Gen.fromIterable(List(1, 2, 3))`.
2. **Random Generators**— Generators that produce random values, such as `Gen.boolean`, `Gen.int`, and `Gen.elements(1, 2, 3)`.

## Random Generators Are Deterministic by Default

The important fact about random generators is that they produce deterministic values by default. This means that if we run the same random generator multiple times, it will always produce the same sequence of values to achieve reproducibility.

So let's add some debugging print lines inside a test and see what values are produced:

```scala

object ExampleSpec extends ZIOSpecDefault {
  def spec =
    test("example test") {
      check(Gen.int(0, 10)) { n =>
        println(n)
        assertTrue(n + n == 2 * n)
      }
    } @@ samples(5)
}
```

We can see, even though the `Gen.int` is a non-deterministic generator, every time we run the test, the generator will produce the same sequence of values:

```scala
runSpec
9
3
0
9
6
+ example test
```

This is due to the fact that the generator uses a pseudo-random number generator, which uses a deterministic algorithm.

The generator provides a fixed seed number to its underlying deterministic algorithm to generate random numbers. As the seed number is fixed, the generator will always produce the same sequence of values.

For more information, there is a separate page about this on [TestRandom](../services/random.md), which is the underlying service for generating test values.

This behavior helps us to have reproducible tests. However, if we need non-deterministic test values, we can use the `TestAspect.nondeterministic` to change the default behavior:

```scala

object ExampleSpec extends ZIOSpecDefault {
  def spec =
    test("example test") {
      check(Gen.int(0, 10)) { n =>
        println(n)
        assertTrue(n + n == 2 * n)
      }
    } @@ samples(5) @@ nondeterministic
}
```

## How Samples Are Generated?

When we run `check`, it creates an infinite stream by repeatedly sampling from the generator (using `forever`), then takes values from that stream—200 samples by default. We can override this using the `TestAspect.samples` test aspect, for example, use`@@ TestAspect.samples(5)` to take only 5 samples. Let's examine how samples are produced for each of the following generators:

- `check(Gen.const(42))(n => ???)` will repeatedly sample from `ZStream.succeed(42)`, producing: 42, 42, 42, ...
- `check(Gen.int)(n => ???)` will repeatedly sample from `ZStream.fromZIO(Random.nextInt)`, producing e.g.: 2, -3422, 33, 3991334, 98138, ...
- `check(Gen.elements(1, 2, 3))(n => ???)` will repeatedly sample from `ZStream.fromZIO(Random.nextIntBounded(3).map(Chunk(1, 2, 3)(_)))`, producing e.g.: 3, 1, 1, 3, 2, ...
- `check(Gen.fromIterable(List(1, 2, 3)))(n => ???)` will repeatedly sample from `ZStream.fromIterable(List(1, 2, 3))`, producing: 1, 2, 3, 1, 2, 3, ...

When we run the `check` function with multiple generators, the samples will be the Cartesian product of their streams. Let's try some examples:

```scala

test("two deterministic generators") {
  check(Gen.const(1), Gen.fromIterable(List("a", "b", "c"))) { (a, b) =>
    println((a, b))
    assertTrue(true)
  }
} @@ TestAspect.samples(5)
```

The output will be:

```scala
(1,a)
(1,b)
(1,c)
(1,a)
(1,b)
+ two deterministic generators
1 tests passed. 0 tests failed. 0 tests ignored.
```

So the example above is something like this:

```scala

{
  for {
    a <- ZStream.succeed(1)
    b <- ZStream.fromIterable(List("a", "b", "c"))
  } yield (a, b)
}.forever.take(5).runCollect.debug
```

Now let's try to use one non-deterministic generator and one deterministic generator:

```scala

test("one non-deterministic generator and one deterministic generator") {
  check(Gen.int(1, 3), Gen.fromIterable(List("a", "b", "c"))) { (a, b) =>
    println((a, b))
    assertTrue(true)
  }
} @@ TestAspect.samples(5)
```

Here is one example output:

```scala
(3,a)
(3,b)
(3,c)
(2,a)
(2,b)
+ one non-deterministic generator and one deterministic generator
1 tests passed. 0 tests failed. 0 tests ignored.
```

This is the same as the previous example; it is like we have the following stream:

```scala

{
  for {
    a <- ZStream.fromZIO(Random.nextIntBetween(1, 3))
    b <- ZStream.fromIterable(List("a", "b", "c"))
  } yield (a, b)
}.forever.take(5).runCollect.debug
```

## Running a Generator for Debugging Purposes

To run a generator, we can call the `runCollect` operation:

```scala

val ints: ZIO[Any, Nothing, List[Int]] = Gen.int.runCollect.debug
// Output: List(-2090696713)
```

This will return a `ZIO` effect containing all its values in a list, which in this example contains only one element.

To create more samples, we can use `Gen#runCollectN`, which repeatedly runs the generator as much as we need. In this example, it will generate a list containing 5 integer elements:

```scala
Gen.int.runCollectN(5).debug
```

In addition, there is an operator called `Gen#runHead`, which returns the first value generated by the generator.

---

## Introduction To Property Testing

## What is Property-Based Testing?

In property-based testing, instead of testing individual values and making assertions on the results, we rely on testing the properties of the system which is under the test.

To be more acquainted with property-based testing, let's look at how we can test a simple addition function. So assume we have a function `add` that adds two numbers:

```scala
def add(a: Int, b: Int): Int = ???
```

in a typical test we start with some well-known values as test inputs and check if the function returns the expected values for each of the pair inputs:

|   Input   |  Expected Output  |
|:---------:|:-----------------:|
|  (0, 0)   |         0         |
|  (1, 0)   |         1         |
|  (0, 1)   |         1         |
|  (0, -1)  |        -1         |
|  (-1, 0)  |        -1         |
|    ...    |        ...        |

Now we can test all the inputs and make sure the `add` function returns the expected values:

```scala

object AdditionSpec extends ZIOSpecDefault {

  def add(a: Int, b: Int): Int = ???

  val testData = Seq(
    ((0, 0), 0),
    ((1, 0), 1),
    ((0, 1), 1),
    ((0, -1), -1),
    ((-1, 0), -1),
    ((1, 1), 2),
    ((1, -1), 0),
    ((-1, 1), 0)
  )

  def spec =
    test("test add function") {
      assertTrue {
        testData.forall { case ((a, b), expected) =>
          add(a, b) == expected
        }
      }
    }
}
```

This is not a very good approach because it is very hard to find a set of inputs that will cover all possible behaviors of the addition function.

Instead, in property-based testing, we extract the set of properties that our function must satisfy. So let's think about the `add` function and find out what properties it must satisfy:

1. **Commutative Property**— It says that changing the order of addends does not change the result. So for all `a` and `b`, `add(a, b)` must be equal to `add(a, b)`:

```scala
assertTrue(add(a, b) == add(b, a))
```

2. **Associative Property**— This says that changing the grouping of addends does not change the result. So for all `a`, `b` and `c`, the `add(add(a, b), c)` must be equal to `add(a, add(b, c))`:

```scala
assertTrue(add(add(a, b), c) == add(a, add(b, c)))
```

3. **Identity Property**— For all `a`, `add(a, 0)` must be equal to `a`:

```scala
assertTrue(add(a, 0) == a)
```

If we test all of these properties we can be sure that the `add` function works as expected, so let's see how we can do that using the `Gen` data type:

```scala

object AdditionSpec extends ZIOSpecDefault {

  def add(a: Int, b: Int): Int = ???

  def spec = suite("Add Spec")(
    test("add is commutative") {
      check(Gen.int, Gen.int) { (a, b) =>
        assertTrue(add(a, b) == add(b, a))
      }
    },
    test("add is associative") {
      check(Gen.int, Gen.int, Gen.int) { (a, b, c) =>
        assertTrue(add(add(a, b), c) == add(a, add(b, c)))
      }
    },
    test("add is identitive") {
      check(Gen.int) { a =>
        assertTrue(add(a, 0) == a)
      }
    }
  )
}
```

---

## Operators

1. `Gen#zipWith` — Composes this generator with the specified generator to create a cartesian product of elements with the specified function:

  ```scala mdoc:compile-only
  Gen.elements("a", "b", "c").zipWith(Gen.elements("1", "2", "3"))(_ + _)
    .runCollectN(5)
  // Sample Output: List(b1, a2, c1, b1, b1)
  ```

2. `Gen#zip` — Composes this generator with the specified generator to create a cartesian product of elements.

  ```scala mdoc:compile-only
  Gen.elements("a", "b", "c").zip(Gen.elements("1", "2", "3"))
    .runCollectN(5)
  (Gen.elements("a", "b", "c") <*> Gen.elements("1", "2", "3"))
    .runCollectN(5)
    
  // Sample Output: List((a,3), (a,3), (c,3), (b,3), (c,2))
  ```

3. `Gen#collect` — Maps the values produced by this generator with the specified partial function, discarding any values the partial function is not defined at:

  ```scala mdoc:compile-only
  Gen.int(-10, +10)
    .collect { case n if n % 2 == 0 => n }
    .runCollectN(5)
    .debug
  // Smaple Output: List(-6, -8, -2, 4, -6)
  ```

4. `Gen#filter` — Filters the values produced by this generator, discarding any values that do not meet the specified predicate:

  ```scala mdoc:compile-only
  Gen.int(-10, +10).filter(_ % 2 == 0).runCollectN(5) 
  // Sample Output: List(-6, 10, 0, -8, 4)
  ```

Using `filter` can reduce test performance, especially if many values must be discarded. It is recommended to use combinators such as `map` and `flatMap` to create generators of the desired values instead:

  ```scala mdoc:compile-only
  Gen.int(-10, +10).map(_ * 2).runCollectN(5)
  // Sample Output: List(2, 6, -6, 20, -14)
  ```

---

## Shrinking

In Property-Based Testing, we specify certain properties of a program, then we ask the testing framework to generate random test data to discover counterexamples. The existence of counterexamples shows that our function, which is under the test, is not correct. Unfortunately, in almost all cases, the first counterexample is not the minimal one, and they are fairly large or complex. So it is not a pretty good sample to describe why our test is failing.

Shrinking is a mechanism that tries to find the smallest counterexample, which is the root cause of the test failure. So it helps a developer to find out why the test is failing.

Finding the smallest failing case is somehow cumbersome and requires many attempts. As a developer, we do not need to do shrinking ourselves. All generators in ZIO Test have built-in shrinkers, so when we test properties, in case of test failures, the ZIO Test attempts to reduce the counterexamples forward their own zero points.

Let's write a `reverse` function with an incorrect implementation:

```scala
def reverse[T](list: List[T]): List[T] =
  if (list.length > 6) list.reverse.dropRight(1) else list.reverse
```

We know that if we reverse a list twice, it should give us the original list, so let's check this property:

```scala

suite("ReverseSpec"){
  // ∀ xs. reverse(reverse(xs)) == xs
  test("reversing a list twice must give the original list")(
    check(Gen.listOf(Gen.int)) { list =>
      assertTrue(reverse(reverse(list)) == list)
    }
  )
}
```

The following messages, is a sample output of the test renderer, after running the test:

```
- ReverseSpec
  - reversing a list twice must give the original list
    Test failed after 7 iterations with input: List(0, 0, 0, 0, 0, 0, 0)
    Original input before shrinking was: List(724856966, 1976458409, -940069360, -191508820, -291932258, 1296893186, 2010410723, 1134770522, 1260002835)
    ✗ List(0, 0, 0, 0, 0, 0) was not equal to List(0, 0, 0, 0, 0, 0, 0)
    reverse(reverse(list)) == list
    reverse(reverse(list)) = List(0, 0, 0, 0, 0, 0)
```

The initial failing input discovered by ZIO Test is `List(724856966, 1976458409, -940069360, -191508820, -291932258, 1296893186, 2010410723, 1134770522, 1260002835)`. The ZIO Test then tries to find the simplest counterexample which is `List(0, 0, 0, 0, 0, 0, 0)`. So the property still fails with the final shrunk value. The original input is a list of 9 somewhat useless numbers, while after shrinking, we have a list of 7 zero numbers, so we can find the bug faster.

---

## Running Tests

We can run ZIO Tests in two ways:

1. If we [added](installation.md) `zio.test.sbt.ZTestFramework` to SBT's `testFrameworks`, our tests should be automatically picked up by SBT on invocation of `test`:

```bash
sbt test                      // run all tests
sbt testOnly HelloWorldSpec   // run a specific test
```

To run a specific test by their labels, we can use the `-t "<label>"` option. Assume we have multiple tests like the below:

```scala

object ExampleSpec extends ZIOSpecDefault {
  def spec = suite("clock")(
    test("foo") {
      assertTrue(true)
    },
    test("foo bar") {
      assertTrue(true)
    },
    test("foo bar baz") {
      assertTrue(true)
    }
  )
}
```

We can run those test that contains the "bar" label using the following SBT command:
```
testOnly ExampleSpec -- -t "bar"
```
You can also execute the single SBT command directly from the shell with this command that properly quotes and escapes arguments:
```
sbt "testOnly ExampleSpec -- -t \"bar\""
```

It will print the following results after running all tests containing the "bar" label:

```scala
sbt:zio-2.0> testOnly ExampleSpec -- -t "bar"
+ example suite
  + foo bar
  + foo bar baz
3 tests passed. 0 tests failed. 0 tests ignored.
```

2. However, if we're not using SBT or have some other special needs, the `ZIOSpecDefault` has a `main` method which can be invoked directly or with SBTs `Test/run` or `Test/runMain` commands:

```bash
sbt Test/run                       // prompt to choose which test to run
sbt Test/runMain HelloWorldSpec    // run a specific test
```

---

## TestClock

In most cases we want unit tests to be as fast as possible. Waiting for real time to pass by is a real killer for this.

ZIO exposes a `TestClock` that can control the time. We can deterministically and efficiently **test effects involving the passage of time** without actually having to wait for the full amount of time to pass.

Calls to `sleep` and methods derived from it will semantically block until the clock time is set/adjusted to on or after the time the effect is scheduled to run.

Instead of waiting for actual time to pass, `sleep` and methods implemented in terms of it schedule effects to take place at a given clock time. Users can adjust the clock time using the `adjust` and `setTime` methods, and all effects scheduled to take place on or before that time will automatically be run in order.

For example, here is how we can test `ZIO#timeout` using `TestClock`:

```scala

for {
  fiber  <- ZIO.sleep(5.minutes).timeout(1.minute).fork
  _      <- TestClock.adjust(1.minute)
  result <- fiber.join
} yield assertTrue(result.isEmpty)
```

Note how we forked the fiber that `sleep` was invoked on. Calls to `sleep` and methods derived from it will semantically block until the time is set to on or after the time they are scheduled to run.

If we didn't fork the fiber on which we called sleep we would never get to set the time on the line below. Thus, a useful pattern when using `TestClock` is to fork the effect being tested, then adjust the clock time, and finally verify that the expected effects have been performed.

Clock time is just like a clock on the wall, except that in our `TestClock`, the clock is broken. Instead of moving by itself, the clock time only changes when adjusted or set by the user, using the `adjust` and `setTime` methods. The clock time never changes by itself.

When the clock is adjusted, any effects scheduled to run on or before the new clock time will automatically be run, in order.

For example, here is how we can test an effect that recurs with a fixed delay:

```scala

for {
  q <- Queue.unbounded[Unit]
  _ <- q.offer(()).delay(60.minutes).forever.fork
  a <- q.poll.map(_.isEmpty)
  _ <- TestClock.adjust(60.minutes)
  b <- q.take.as(true)
  c <- q.poll.map(_.isEmpty)
  _ <- TestClock.adjust(60.minutes)
  d <- q.take.as(true)
  e <- q.poll.map(_.isEmpty)
} yield assertTrue(a && b && c && d && e)
```

Here we verify that no effect is performed before the recurrence period, that an effect is performed after the recurrence period, and that the effect is performed exactly once.

The key thing to note here is that after each recurrence the next recurrence is scheduled to occur at the appropriate time in the future, so when we adjust the clock by 60 minutes exactly one value is placed in the queue, and when we adjust the clock by another 60 minutes exactly one more value is placed in the queue.

## Examples

### Example 1

Thanks to the call to `TestClock.adjust(1.minute)` we moved the time instantly 1 minute.

```scala

test("One can move time very fast") {
  for {
    startTime <- currentTime(TimeUnit.SECONDS)
    _         <- TestClock.adjust(1.minute)
    endTime   <- currentTime(TimeUnit.SECONDS)
  } yield assertTrue((endTime - startTime) >= 60L)
}
```

### Example 2

`TestClock` affects also all code running asynchronously that is scheduled to run after a certain time:

```scala

test("One can control time as he see fit") {
  for {
    promise <- Promise.make[Unit, Int]
    _       <- (ZIO.sleep(10.seconds) *> promise.succeed(1)).fork
    _       <- TestClock.adjust(10.seconds)
    readRef <- promise.await
  } yield assertTrue(1 == readRef)
}
```

The above code creates a write-once cell that will be set to "1" after 10 seconds asynchronously from a different thread thanks to the call to `fork`. In the end, we wait on the promise until it is set.

With the call to `TestClock.adjust(10.seconds)` we simulate the passing of 10 seconds of time. Because of it, we don't need to wait for the real 10 seconds to pass and thus our unit test can run faster.

This is a pattern that will very often be used when `sleep` and `TestClock` are being used for testing of effects that are based on time. The fiber that needs to sleep will be forked and `TestClock` will used to adjust the time so that all expected effects are run in the forked fiber.

### Example 3

A more complex example leveraging dependencies and multiple services is shown below:

```scala

trait SchedulingService {
  def schedule(promise: Promise[Unit, Int]): ZIO[Any, Exception, Boolean]
}

trait LoggingService {
  def log(msg: String): ZIO[Any, Exception, Unit]
}

val schedulingLayer: ZLayer[LoggingService, Nothing, SchedulingService] =
  ZLayer.fromFunction { (loggingService: LoggingService) =>
    new SchedulingService {
      def schedule(promise: Promise[Unit, Int]): ZIO[Any, Exception, Boolean] =
        (ZIO.sleep(10.seconds) *> promise.succeed(1))
          .tap(b => loggingService.log(b.toString))
    }
}

test("One can control time for failing effects too") {
  val failingLogger = ZLayer.succeed(new LoggingService {
    override def log(msg: String): ZIO[Any, Exception, Unit] = ZIO.fail(new Exception("BOOM"))
  })

  val layer = failingLogger >>> schedulingLayer

  val testCase =
    for {
      promise <- Promise.make[Unit, Int]
      result <- ZIO.serviceWithZIO[SchedulingService](_.schedule(promise)).exit.fork
      _ <- TestClock.adjust(10.seconds)
      readRef <- promise.await
      result <- result.join
    } yield assertTrue((1 == readRef) && result.isFailure)
  testCase.provideLayer(layer)
}
```

In this case, we want to test an effect with dependencies that can potentially fail with an error. To do this we need to run the effect and use assertions that expect an `Exit` value.

### Example 4

The pattern with `Promise` and `await` can be generalized when we need to wait for multiple values using a `Queue`. We simply need to put multiple values into the queue and progress the clock multiple times and there is no need to create multiple promises.

Even if you have a non-trivial flow of data from multiple streams that can produce at different intervals and would like to test snapshots of data at a particular point in time `Queue` can help with that.

```scala

test("zipLatest") {
  val s1 = ZStream.iterate(0)(_ + 1).schedule(Schedule.fixed(100.milliseconds))
  val s2 = ZStream.iterate(0)(_ + 1).schedule(Schedule.fixed(70.milliseconds))
  val s3 = s1.zipLatest(s2)

  for {
    q      <- Queue.unbounded[(Int, Int)]
    _      <- s3.foreach(q.offer).fork
    fiber  <- ZIO.collectAll(ZIO.replicate(4)(q.take)).fork
    _      <- TestClock.adjust(1.second)
    result <- fiber.join
  } yield assertTrue(result == List(0 -> 0, 0 -> 1, 1 -> 1, 1 -> 2))
}
```

### Example 5

`TestClock` is used to speed up the tests by simulating the passage of time. This is useful for triggering scheduled effects, which is useful for testing time-dependent code. `TestClock` does nothing to advance events on its own and is initialized to 00:00 1/1/70.

However, we can use a live clock to simulate events, enabling a fixed ratio of 'real' to 'test' time. `TestClock.adjust()` can be used to advance time, but a real clock may be required to space out the advancements properly.

The test below creates a stream of 30 elements that are spaced 1 second apart. The test advances the clock by 1 second 30 times, allowing the stream to generate all 30 elements.

```scala

test("test clock") {
  val stream = ZStream.iterate(0)(_ + 1).schedule(Schedule.spaced(1.second))
  val s1 = stream.take(30)
  val sink = ZSink.collectAll[Int]
  for {
    fiber <- s1.run(sink).fork
    _ <- TestClock.adjust(1.second).repeat(Schedule.recurs(30))
    runner <- fiber.join
  } yield assert(runner.size)(equalTo(30))
}
```

The test doesn't work because the fast forward is too fast and by the time the stream generator takes the first element, it has moved all the way to the end of the 30 seconds. We need to slow down the rate at which the test clock advances.

The problem is that the `Schedule` needs a clock to do the spacing. We can't use the test clock since this is what we need to change. We opt to use the live clock instead with the `@@ withLiveClock` test aspect. We also use `Schedule.spaced` to dictate the spacing of the events in real time.

```scala

test("live clock") {
  val stream = ZStream.iterate(0)(_ + 1).schedule(Schedule.spaced(1.second))
  val s1 = stream.take(30)
  val sink = ZSink.collectAll[Int]
  for {
    fiber <- TestClock.adjust(1.second).repeat(Schedule.spaced(10.milliseconds)).fork
    _ <- fiber.join
    runner <- s1.run(sink)
  } yield assert(runner.size)(equalTo(30))
} @@ TestAspect.withLiveClock
```

Using this technique, we can simulate the advancement of 1 second in the test clock for every 10 milliseconds in real time using the live clock.

---

## TestConsole

`TestConsole` allows testing of applications that interact with the console by modeling working with standard input and output as writing and reading to and from internal buffers:

```scala

val consoleSuite = suite("ConsoleTest")(
  test("One can test output of console") {
    for {
      _              <- TestConsole.feedLines("Jimmy", "37")
      _              <- Console.printLine("What is your name?")
      name           <- Console.readLine
      _              <- Console.printLine("What is your age?")
      age            <- Console.readLine.map(_.toInt)
      questionVector <- TestConsole.output
      q1             = questionVector(0)
      q2             = questionVector(1)
    } yield {
      assertTrue(name == "Jimmy") &&
        assertTrue(age == 37) &&
        assertTrue(q1 == "What is your name?\n") &&
        assertTrue(q2 == "What is your age?\n")
    }
  }
)
```

The above code simulates an application that will ask for the name and age of the user. To test it we prefill buffers with answers with the call to `TestConsole.feedLines` method. Calls to `Console.readLine` will get the value from the buffers instead of interacting with the users keyboard.

Also, all output that our program produces by calling `Console.printLine` (and other printing methods) is being gathered and can be accessed with a call to `TestConsole.output`.

`TestConsole` provides a testable interface for programs interacting with the console by modeling input and output as reading from and writing to input and output buffers maintained by `TestConsole` and backed by a `Ref`.

All calls to `print` and `printLine` using the `TestConsole` will write the string to the output buffer and all calls to `readLine` will take a string from the input buffer. 

To facilitate debugging, by default output will also be rendered to standard output. We can enable or disable this for a scope using `debug`, `silent`, or the corresponding test aspects. 

`TestConsole` has several methods to access and manipulate the content of these buffers including:
- **`feedLines`** to feed strings to the input  buffer that will then be returned by calls to `readLine`.
- **`output`** to get the content of the output buffer from calls to `print` and `printLine`
- **`clearInput`** and **`clearOutput`** to clear the respective buffers.

Together, these functions make it easy to test programs interacting with the console.

---

## Introduction(Services)

ZIO Test has out of the box test implementations for all built-in ZIO services, such as `Console`, `Clock`, `Random` and `System` through the following modules:
- [`TestConsole`](./console.md)
- [`TestClock`](./clock.md)
- [`TestRandom`](./random.md)
- [`TestSystem`](./system.md)

Stability is what we expect from tests, at least those we consider unit tests. Consecutive runs should yield the same results and take more or less the same amount of time.

External services, such as payment APIs, object storages, HTTP APIs, are the biggest source of complexity during testing. It is normal to hide these kinds of services behind an interface and provide test instances to regain control and determinism.

However, there is another source of complexity that comes from the local infrastructure that is also hard to control without building prior abstractions. Things like stdin/stdout, clocks, random generators, schedulers can make writing tests hard or even impossible.

Fortunately, ZIO abstracted most of it in its runtime under `Environment` type. Thanks to this design ZIO Test could easily provide its own implementation named `TestEnvironment` which gives us test implementations of mentioned infrastructure.

If we are using ZIO Test and extending `ZIOSpecDefault` a `TestEnvironment` containing all of them will be automatically provided to each of our tests. Otherwise, the easiest way to use the test implementations in ZIO Test is by providing the `TestEnvironment` to our program:

```scala
myProgram.provide(testEnvironment)
```

Then all environmental effects, such as printing to the console or generating random numbers, will be implemented by the `TestEnvironment` and will be fully testable. When we do need to access the "live" environment, for example to print debugging information to the console, we just use the `live` combinator along with the effect as our normally would. 

If we are only interested in one of the test implementations for our application, we can also access them a la carte through the `make` method on each module. Each test module requires some data on initialization:

```scala
myProgram.provideCustom(TestConsole.make(TestConsole.Data()))
```

Finally, we can create a `Test` object that implements the test interface directly using the `makeTest` method. This can be useful when we want to access some testing functionality without using the environment type:

```scala
for {
  testRandom <- TestRandom.makeTest(TestRandom.DefaultData)
  n          <- testRandom.nextInt
} yield n
```

This can also be useful when we are creating a more complex environment to provide the implementation for test services that we mix in.

---

## Live

The `Live` trait provides access to the _live_ environment from within the test environment for effects such as printing test results to the console or timing out tests where it is necessary to access the real environment.

## Providing Live Environment 

### To the Entire Effect

The easiest way to access the _live_ environment is to use the `live` method with an effect that would otherwise access the test environment.

For example, within the test environment, when we use the `Clock.currentTime` function, it will not execute the live version of the `Clock` service, rather, it will run the test version of the `Clock`, which is instantiated with zero-state by default. Thus, the `Clock.currentTime` will return a `0L` value:

```scala

test("running clock methods in a test environment") {
  assertZIO(Clock.currentTime(TimeUnit.MILLISECONDS))(equalTo(0L)) 
}
```

To run the `Clock.currentTime` within the live environment, we should use the `Live.live` method:

```scala

test("live can access real environment") {
  for {
    live <- Live.live(Clock.currentTime(TimeUnit.MILLISECONDS))
  } yield assertTrue(live != 0L)
}
```

### To the Part of an Effect

The `withLive` method can be used to apply a transformation to an effect with the live environment while ensuring that the effect itself still runs with the test environment.

For example, assume we have a long-running task that is required to run within the test environment, and we want to timeout it before the assertion. To do this, we should run the timeout operation within the live environment:

```scala

val longRunningSUT =
  ZIO.attemptBlockingInterrupt {
    // ... 
    Thread.sleep(10000) // simulating a long-running blocking operation
    // ...
  }
  
test("withLive provides real environment to a single part of an effect") {
  assertZIO(Live.withLive(longRunningSUT)(_.timeout(3.seconds)))(anything)
}
```

---

## TestRandom

When working with randomness, testing might be hard because the inputs to the tested function change on every invocation. So our code behaves in an indeterministic way.

Precisely because of this reason `ZIO` exposes `TestRandom` module which allows for fully deterministic testing of code that deals with Randomness. `TestRandom` can operate in two modes based on the needed use-case. It can generate a sequence of psudeo-random values using an initial seed with series of internal state transition or by feeding predefined random values.

## Initial Seed with Series of Internal State Transition

In the first mode, the `TestRandom` is a purely functional pseudo-random number generator. It will generate pseudo-random values just like `scala.util.Random`. While the `scala.util.Random` doesn't have internal state, the `TestRandom` has an internal state. Instead, methods like `nextInt` describe state transitions from one random state to another that are automatically composed together through methods like `flatMap`. 

The random seed can be set using `setSeed` and `TestRandom` is guaranteed to return the same sequence of values for any given seed. This is useful for deterministically generating a sequence of pseudo-random values and powers the property based testing functionality in ZIO Test:

```scala

test("Use setSeed to generate stable values") {
  for {
    _ <- TestRandom.setSeed(27)
    r1 <- Random.nextLong
    r2 <- Random.nextLong
    r3 <- Random.nextLong
  } yield
    assertTrue(
      List(r1, r2, r3) == List[Long](
        -4947896108136290151L,
        -5264020926839611059L,
        -9135922664019402287L
      )
    )
}
// res0: Spec[Any, Nothing] = Spec(
//   caseValue = LabeledCase(
//     label = "Use setSeed to generate stable values",
//     spec = Spec(
//       caseValue = TestCase(
//         test = FlatMap(
//           trace = "repl.MdocSession.MdocApp.res0(random.md:17)",
//           first = Stateful(
//             trace = "repl.MdocSession.MdocApp.res0(random.md:17)",
//             onState = zio.ZIO$$$Lambda$19534/0x00007f02a6ebc5b0@38f30c0f
//           ),
//           successK = zio.test.package$ZTest$$$Lambda$19543/0x00007f02a6ec0cb0@7df18575
//         ),
//         annotations = Map(zio.test.TestAnnotation@b4aaf9ee -> List(SourceLocation(random.md,17)))
//       )
//     )
//   )
// )
```

## Feeding Predefined Random Values

In the second mode, `TestRandom` maintains an internal buffer of values that can be _fed_ with methods such as `feedInts` and then when random values of that type are generated they will first be taken from the buffer. This is useful for verifying that functions produce the expected output for a given sequence of _random_ inputs.

`TestRandom` will automatically take values from the buffer if a value of the appropriate type is available and otherwise generate a pseudo-random value, so there is nothing we need to do to switch between the two modes. Just generate random values as we normally would to get pseudo-random values, or feed in values of our own to get those values back.

```scala

test("One can provide its own list of ints") {
  for {
    _ <- TestRandom.feedInts(1, 9, 2, 8, 3, 7, 4, 6, 5)
    r1 <- Random.nextInt
    r2 <- Random.nextInt
    r3 <- Random.nextInt
    r4 <- Random.nextInt
    r5 <- Random.nextInt
    r6 <- Random.nextInt
    r7 <- Random.nextInt
    r8 <- Random.nextInt
    r9 <- Random.nextInt
  } yield assertTrue(
    List(1, 9, 2, 8, 3, 7, 4, 6, 5) == List(r1, r2, r3, r4, r5, r6, r7, r8, r9)
  )
}
```

We can also use methods like `clearInts` to clear the buffer of values of a given type, so we can fill the buffer with new values or go back to pseudo-random number generation.

---

## Sized

The `Sized` service enables the _Sized Generators_ to access the _size_ from the ZIO Test environment:

```scala
trait Sized extends Serializable {
  def size: UIO[Int]
  def withSize[R, E, A](size: Int)(zio: ZIO[R, E, A]): ZIO[R, E, A]
}
```

The `Sized` service has two APIs:
1. `Sized.size`
2. `Sized.withSize`

Let's go through each one:

## Operations

### size

To access the default _size_ value from the environment, we can use the `Sized.size` API. In ZIO Test, it is used to enable the _sized generators_ access the _size_ from the environment:

```scala
object Sized {
  def withSize[R, E, A](size: Int)(zio: ZIO[R, E, A]): ZIO[R, E, A] = ???
}
```

For example, the `Gen.sized` generator has the following signature:

```scala
object Gen {
  def sized[R, A](f: Int => Gen[R, A]): Gen[R, A] = ???
}
```

It applies the _size_ to the given function of type `Int => Gen[R, A]`. We should note that the `sized` generator obtains the `size` value from the environment internally.

In the following example, we are creating a sized generator, which generates integer values within the specified range. In this generator, the lower bound is bound to zero, and the upper bound is bound to the _size_ value, which is obtained from the ZIO environment:

```scala

val sizedInts: Gen[Any, Int] = 
  Gen.sized(Gen.int(0, _))
```

To generate some sample values, we can use `Gen#runCollectN` operator on that:

```scala
val samples: UIO[List[Int]] = 
  sizedInts.runCollectN(5).debug
```

### withSize

To change the default _size_ temporarily, we can use the `Size.withSize`. It takes a `size` and a ZIO effect, and runs that effect bounded with the given `size`:

```scala
object Sized {
  def withSize[R, E, A](size: Int)(zio: ZIO[R, E, A]): ZIO[R, E, A] = ???
}
```

```scala

val effect     : UIO[String] = ZIO.succeed("effect")
val sizedEffect: UIO[String] = Sized.withSize(10)(effect)
```

ZIO Test has a test aspect called `TestAspect.size` which is a helper method for this operation. This test aspect runs each test with the given _size_ value:

```scala

object SizedSpec extends ZIOSpecDefault {
  def spec =
    suite("sized") {
      test("bounded int generator shouldn't cross its boundaries") {
        check(Gen.sized(Gen.int(0, _))) { n =>
          assertTrue(n >= 0 && n <= 200)
        }
      } @@ TestAspect.size(200)
    }
}
```

---

## TestSystem

`TestSystem` supports deterministic testing of effects involving system properties. 

With the increased usage of containers and runtimes like Kubernetes, more and more applications are being configured by means of environment variables. It is important to test this logic just like other parts of an application.

For this purpose ZIO Test exposes `TestSystem` module. Additionally, to setting the _environment variables_. It also allows for setting _JVM system properties_ like in the code below:

```scala

for {
  _      <- TestSystem.putProperty("java.vm.name", "VM")
  result <- System.property("java.vm.name")
} yield assertTrue(result == Some("VM"))
```

Internally, `TestSystem` maintains mappings of environment variables and system properties that can be set and accessed. It is worth noticing that no actual environment variables or system properties will be accessed or set as a result of these actions. So there will be no impact on other parts of the system.

---

## TestConfig

The `TestConfig` service provides access to default configuration settings used by ZIO Test:

1. Repeats — The number of times to repeat tests to ensure they are stable.
2. Retries — The number of times to retry flaky tests.
3. Samples — The number of sufficient samples to check for a random variable.
4. Shrinks — The maximum number of [shrinkings](../property-testing/shrinking.md) to minimize large failures.

Currently, the live version of `TestEnvironment` contains the live version of this service with the following configs:

```scala
TestConfig.live(
  repeats0 = 100,
  retries0 = 100,
  samples0 = 200,
  shrinks0 = 1000
)
```

So by default, the ZIO Test runner will run tests with this config. Regular users do not need access to this service unless they need to change or access configurations from the ZIO Test environment.

---

## Sharing Layers Between Multiple Files

In the previous example, we used the `Spec#provideXYZShared` methods to share layers between multiple specs in one file. In most cases, when the number of tests and specs grows, this is not a good idea. We want a way to share layers between multiple specs in different files.

So in such situations, we can't use the previous pattern here, because specs are in entirely different files, and we don't want the boilerplate of creating a _master spec_ that references the other specs. ZIO has a solution to this problem. We can define the resource we want to share as part of the `bootstrap` layer of `ZIOApp`.

The `bootstrap` layer is responsible for creating and managing any services that our ZIO tests need. Using `boostrap` we can have one shared layer across multiple test specs.

:::note
When layers of the same type are defined at the class level within different test suites, they will be initialized separately for each test suite. This can lead to resource exhaustion especially with limited resources like database connections or connection pools. To avoid this, we need to ensure that such layers are defined centrally such as in the companion object to be shared properly across multiple test suites.
:::

Before we start, let's remember the `Counter` service in the previous [section](sharing-layers-within-the-same-file.md):

```scala

case class Counter(value: Ref[Int]) {
  def inc: UIO[Unit] = value.update(_ + 1)
  def get: UIO[Int] = value.get
}

object Counter {
  val layer =
    ZLayer.scoped(
      ZIO.acquireRelease(
        Ref.make(0).map(Counter(_)) <* ZIO.debug("Counter initialized!")
      )(c => c.get.debug("Number of tests executed"))
    )
  def inc = ZIO.service[Counter].flatMap(_.inc)
}
```

Now, let's assume we have two specs in different files, and we want to share the `Counter` service between them. First, we need to create a base class that contains the shared bootstrap layer:

```scala

abstract class SharedCounterSpec extends ZIOSpec[Counter] {
  override val bootstrap: ZLayer[Any, Nothing, Counter] = Counter.layer
}
```

Now it's time to create the specs. Each spec is extending the `SharedCounterSpec` class.

Spec1.scala:

```scala

object Spec1 extends SharedCounterSpec {
  override def spec =
    test("test1") {
      assertTrue(true)
    } @@ TestAspect.after(Counter.inc)
}
```

Spec2.scala:

```scala

object Spec2 extends SharedCounterSpec {
  override def spec: Spec[Scope with Counter, Any] =
    test("test2") {
      assertTrue(true)
    } @@ TestAspect.after(Counter.inc)
}
```

Now, when we run all specs (`sbt testOnly org.example.*`), we will see an output like this:

```
Counter initialized!
+ test1
+ test2
Number of tests executed: 2
```

The ZIO test runner will execute all specs with the shared bootstrap layer. This means that the `Counter` service will be created and managed only once, and will be shared between all specs.

---

## Sharing Layers within the Same File

The `Spec` data type has a mechanism to share layers within all tests in a suite. Instead of acquiring and releasing dependencies for each test, we can share the layer within all tests of a given suite. The suite acquires that layer for once and shares that between all tests. When the execution of all tests in that suite is finished, the layer will be released. To share layers between multiple specs we can use one of the provide methods ending with `Shared`:
- `provideShared`
- `provideCustomShared`
- `provideSomeShared`
- `provideLayerShared`
- `provideCustomLayerShared`
- `provideSomeLayerShared`

```scala
{
  test("test1")(???) +
    test("test2")(???)
}.provideShared(sharedLayer)

suite("suite1")(
  test("test1")(???),
  test("test2")(???)
).provideShared(sharedLayer)

suite("all suites")(
  suite("suite1")(
    test("test1")(???),
  ),
  suite("suite2")(
    test("test1")(???),
    test("test2")(???),
    test("test3")(???)
  )
).provideShared(sharedLayer)
```

To demonstrate this, let's try an example. In this example, instead of using the built-in features of the ZIO Test, we want to write our own basic solution to count the number of times tests are executed.

First, we need a counter service like the below:

```scala

case class Counter(value: Ref[Int]) {
  def inc: UIO[Unit] = value.update(_ + 1)
  def get: UIO[Int] = value.get
}

object Counter {
  val layer =
    ZLayer.scoped(
      ZIO.acquireRelease(
        Ref.make(0).map(Counter(_)) <* ZIO.debug("Counter initialized!")
      )(c => c.get.debug("Number of tests executed"))
    )
  def inc = ZIO.service[Counter].flatMap(_.inc)
}
```

We use this service to count the number of times the tests are executed, by calling the `Counter.inc` operator after each test:

```scala

object MySpecs extends ZIOSpecDefault {
  def spec = {
    suite("Spec1")(
      test("test1") {
        assertTrue(true)
      } @@ TestAspect.after(Counter.inc),
      test("test2") { assertTrue(true)
      } @@ TestAspect.after(Counter.inc)
    ) +
      suite("Spec2") {
        test("test1") {
          assertTrue(true)
        } @@ TestAspect.after(Counter.inc)
      }
  }.provideShared(Counter.layer)
}
```

If we execute all tests, we will see an output like this:

```
Counter initialized!
+ Spec1
  + test2
  + test1
+ Spec2
  + test1
Number of tests executed: 3
3 tests passed. 0 tests failed. 0 tests ignored.
```

In the above example, the `Counter.layer` is shared between all tests, and only acquired and released once.

---

## Spec

## Constructors

## Dependencies on Other Services

Just like the `ZIO` data type, the `Spec` requires an environment of type `R`. When we write tests, we might need to access a service through the environment. It can be a combination of the standard services such a `Clock`, `Console`, `Random` and `System` or test services like `TestClock`, `TestConsole`, `TestRandom`, and `TestSystem`, or any user-defined services.

## Using Standard Test Services

All standard test services are located at the `zio.test` package. They are test implementation of standard ZIO services. The use of these test services enables us to test functionality that depends on printing to or reading from a console, randomness, timings, and, also the system properties.

Let's see how we can test the `sayHello` function, which uses the `Console` service:

```scala

def sayHello: ZIO[Any, IOException, Unit] =
  Console.printLine("Hello, World!")

suite("HelloWorldSpec")(
  test("sayHello correctly displays output") {
    for {
      _      <- sayHello
      output <- TestConsole.output
    } yield assertTrue(output == Vector("Hello, World!\n"))
  }
)
```

There is a separate section in the documentation pages that covers [all built-in test services](services/index.md).

## Providing Layers

By using `Spec#provideXYZLayer`, a test or suite of tests can be provided with any dependencies in a similar way to how a ZIO data type can.

## Sharing Layers Between Multiple Specs

ZIO Test has the ability to share layers between multiple specs. This is useful when we want to have some common services available for all tests. We have two ways to do this:

1. Using `Spec#provideXYZShared` methods, which is useful to share layers between multiple specs that are residing in the same file.
2. Using the `bootstrap` layer, which is useful to share layers between multiple specs that are residing in different files.

## Operations

In ZIO Test, specs are just values like other data types in ZIO. So we can filter, map or manipulate these data types. In this section, we are going to learn some of the most important operations on the `Spec` data type:

---

## Test Hierarchies and Organization

A `Spec[R, E]` is the backbone of ZIO Test. All specs require an environment of type `R` and may potentially fail with an error of type `E`.

We can think of a spec as just a collection of tests. It is essentially a recursive data structure where every spec is just one individual test or a suite that itself can have multiple specs inside that each could be tests or sub suites. We can go down as far as we want in a recursive tree-like data structure.

## A Single Spec

The `test` constructor creates one single spec:

```scala

val mySpec = test("true is true") {
  assertTrue(true)
}
```

Real tests that run some logic and return testing result are created mostly with `test` function. It expects two arguments, first one will be the label of test which will be used for visual reporting back to the user, and an assertion which contains some testable logic specified about a target under the test.

## Collection of Multiple Specs

The `suite` creates a suite which contains other specs (tests or suites):

```scala

val mySuite =
  suite("A suite containing multiple tests")(
    test("the first test") {
      assertTrue(1 + 1 == 2)
    },
    test("the second test") {
      assertTrue(2 * 2 == 4)
    }
  )
```

Suites can contain other suites. We can have multiple suites and one big suite that will aggregate them all:

```scala

suite("int and string")(
  suite("int suite")(
    test("minus")(assertTrue(2 - 1 == 1)),
    test("plus")(assertTrue(1 + 1 == 2))
  ),
  suite("string suite")(
    test("concat")(assertTrue("a" + "b" == "ab")),
    test("length")(assertTrue("abc".length == 3))
  )
)
```

## Collection of Multiple **Smart** Specs

The `suite` method creates a spec from a collection of specs. So what we can do is to provide it with a collection of specs:

```scala

object ExampleSpec extends ZIOSpecDefault {

  def spec =
    suite("some suite")(
      test("test 1") {
        val stuff = 1
        assertTrue(stuff == 1)
      },
      test("test 2") {
        val stuff = Some(1)
        assertTrue(stuff == Some(1))
      }
    )

}
```

But what if we wanted to have a suite of tests that work on a common value, e.g. the same `stuff`? ZIO provides the `suiteAll` method that helps us to share the same `stuff` between all tests:

```scala

object ExampleSpec extends ZIOSpecDefault {

  def spec =
    suiteAll("some suite") {

      val stuff = "hello"

      test("test 1") {
        assertTrue(stuff.startsWith("h"))
      }

      val stuff2 = 5

      test("test 2") {
        assertTrue(stuff.length == stuff2)
      }
    }

}
```

## Sharing the result of an effectful value between tests

As we saw in the previous section, the `suiteAll` method allows tests to work with common values. But what if the value is a result of some effect and we want to use it in several tests, without running the effect every time? Suites in `zio-test` can be effectual, so we can use this feature to share effectual values:

```scala

object ExampleSpec extends ZIOSpecDefault {
  val spec =
    suite("some suite") {
      for {
        stuff <- ZIO.succeed("hello")
        stuff2 = 5
      } yield Chunk(

        test("test 1") {
          assertTrue(stuff.startsWith("h"))
        },

        test("test 2") {
          assertTrue(stuff.length == stuff2)
        }

      )
    }
}
```

---

## Why ZIO Test?

In this section, we will discuss important features of the ZIO Test which help us to test our effectual code easily.

## Test Environment

The library includes built-in _testable versions_ of all the standard ZIO services (`Clock`, `Console`, `System`, and `Random`). For example, the `TestClock` has some timing actions that enables us to control the passage of time. So instead of waiting for timeouts and passage of time, we can adjust the time in our test:

```scala

test("timeout") {
  for {
    fiber  <- ZIO.sleep(5.minutes).timeout(1.minute).fork
    _      <- TestClock.adjust(1.minute)
    result <- fiber.join
  } yield assertTrue(result.isEmpty)
}
```

In this example, to test the timeout function without waiting for one minute, we passed the time for one minute using the `adjust` operation. Sometimes, we may want to run these kinds of tests with the `nonFlaky` operator, which runs the test one hundred different times.

The `TestRandom` service has some extra functionality that enables us to test those functionalities with randomness. We can provide seed number to the `TestRandom`, and then we can have an exact expectation of the random function results.

Each of these services, comes with a bunch of functionality that makes it very easy to test effects.

Whenever we need to access the _live_ environment, we can use the `live` method in the `test` package or test annotations like `withLiveConsole`.

## Resource Management

We may need to set up and tear down some fixtures in our test code before and after running tests. ZIO Test manages this seamlessly for us. So, instead of providing `before`/`after`, `beforeAll`/`afterAll` hooks which are not composable, we can provide a `ZLayer` to each test or a test suite. The ZIO test takes care of acquiring, utilizing, and releasing that layer.

For example, if we have a Kafka layer, we can provide it to one test, or we can provide it to an entire suite of tests, just like the example below:

```scala
suite("a test suite with shared kafka layer")(test1, test2, test3)
  .provideCustomLayerShared(kafkaLayer)
```

This layer going to get acquired once, then we have access to that service within all these three tests within the suite and then it is guaranteed to be released at the end of our tests.

So in ZIO Test, we have nice resource management which enables us to have tests where:
- They are resource safe
- Resources can be acquired and released per test or across a suite of tests
- Fully composable

## Property Based Testing

Support for property based testing is included out-of-the-box through the `check` method and its variants and the `Gen` and `Sample` classes. For example, here is how we could write a property to test that integer addition is associative.

```scala

val associativity =
  check(Gen.int, Gen.int, Gen.int) { (x, y, z) =>
    assertTrue(((x + y) + z) == (x + (y + z)))
  }
```

If a property fails, the failure will be automatically shrunk to the smallest failing cases to make it easier for us to diagnose the problem. And shrinking is integrated with the generation of random variables, so we are guaranteed that any shrunk counter example will meet the conditions of our original generator.

ZIO Test also supports automatic derivation of generators using the ZIO Test Magnolia module:

```scala

case class Point(x: Double, y: Double)

val genPoint: Gen[Any, Point] = DeriveGen[Point]
 
sealed trait Color
case object Red   extends Color
case object Green extends Color
case object Blue  extends Color
 
val genColor: Gen[Any, Color] = DeriveGen[Color]
```

## Test Reporting

When tests do fail, it is easy to see what went wrong because the test reporter will show us the entire assertion that failed and the specific part of the assertion that failed. To facilitate this, a variety of assertion combinators are included in the `Assertion` class.

## Test Aspects

Test aspects are powerful tools for modifying behavior of individual tests or even entire suites that we have already written. Convenient syntax `@@` is provided for applying test aspects.

For example, we can apply a timeout to a test by using `test @@ timeout(60.seconds)` or only run a test on JavaScript by using `test @@ jsOnly`.

Test aspects are _highly composable_, so we can combine multiple test aspects together:

```scala

test("another zio test")(???) @@ timeout(60.seconds) @@ jvmOnly
```

## Zero Dependencies

As a library with zero third party dependencies, this project is available on the JVM, ScalaJS, Dotty, and Scala Native. So we can write our tests once and make sure that our code works correctly across all platforms that we support.

## Build tools other than SBT

This documentation uses SBT build tool exclusively;
other build tools that support ZIO Test are:
- [Mill](https://mill-build.org/mill/scalalib/testing.html)
- [Gradle](https://gradle.org) - with a [plugin](https://github.com/dubinsky/scalajs-gradle)

---

## Writing Our First Test

Any object that implements the `ZIOSpecDefault` trait is a runnable test. So to start writing tests we need to extend `ZIOSpecDefault`, which requires a `Spec`:

```scala

object HelloWorldSpec extends ZIOSpecDefault {
  def spec = 
    suite("HelloWorldSpec")(
      ??? // all tests go here
    )
}
```

:::note
In order to have runnable tests, the `ZIOSpecDefault` trait must be extended by an **object** that implements the `spec` method. If we extend this trait in a class, the test runner will not be able to find the tests.
:::

`ZIOSpecDefault` is very similar in its logic of operations to `ZIOAppDefault`. Instead of providing one `ZIO` application at the end of the world, we provide a suite that can be a tree of other suites and tests. 

```scala

object HelloWorld {
  def sayHello: ZIO[Any, IOException, Unit] =
    Console.printLine("Hello, World!")
}

object HelloWorldSpec extends ZIOSpecDefault {
  def spec = suite("HelloWorldSpec")(
    test("sayHello correctly displays output") {
      for {
        _      <- sayHello
        output <- TestConsole.output
      } yield assertTrue(output == Vector("Hello, World!\n"))
    }
  )
}
```

In the example above, our test involved the effect of printing to the console, but we didn't have to do anything differently in our test. Also note that the `helloWorld` method in the above program does not actually print a string to the console instead writes it to a buffer for testing.

---

## ZIO Guides

## Quickstart Guides

The following guides have been written to help you get started with ZIO with minimal effort and without the need to fully understand the underlying technology:

1. [ZIO Quickstart: Hello World](quickstarts/hello-world.md)
2. [ZIO Quickstart: Building a RESTful Web Service](quickstarts/restful-webservice.md)
3. [ZIO Quickstart: Building a GraphQL Web Service](quickstarts/graphql-webservice.md)
4. [ZIO Quickstart: Building a gRPC Server and Client](https://scalapb.github.io/zio-grpc/docs/quickstart/)

## Tutorial Guides

1. [How to Make a ZIO Application Configurable?](tutorials/make-a-zio-application-configurable.md)
2. [How to Encode and Decode JSON Data](tutorials/encode-and-decode-json-data.md)
3. [How to Enable Logging in a ZIO Application?](tutorials/enable-logging-in-a-zio-application.md)
4. [How to Create a Custom Logger for a ZIO Application?](tutorials/create-custom-logger-for-a-zio-application.md)
5. [How to Run Our First ZIO Project With VSCode?](tutorials/run-our-first-zio-project-with-vscode.md)
6. [How to Run Our First ZIO Project With IntelliJ IDEA?](tutorials/running-our-first-zio-project-with-intellij-idea.md) 
7. [How to Build a RESTful Web Service](tutorials/build-a-restful-webservice.md)
8. [How to Deploy a ZIO Application Using Docker?](tutorials/deploy-a-zio-application-using-docker.md)
9. [How to Monitor a ZIO Application Using ZIO's Built-in Metric System?](tutorials/monitor-a-zio-application-using-zios-built-in-metric-system.md)
10. [How to Produce/Consume Data To/From Kafka Topics?](/zio-kafka/tutorial)
11. [How to Debug a ZIO Application?](tutorials/debug-a-zio-application.md)

## Integration Guides

1. [How to Interop with Cats Effect?](interop/with-cats-effect.md)
2. [How to Interop with Future?](interop/with-future.md)
3. [How to Interop with Java?](interop/with-java.md)
4. [How to Interop with JavaScript?](interop/with-javascript.md)

## Migration Guides

1. [ZIO 2.x Migration Guide](migrate/migration-guide.md)
2. [How to Migrate from Cats Effect to ZIO?](migrate/from-cats-effect.md)
3. [How to Migrate from Monix to ZIO?](migrate/from-monix.md)

---

## How to Interop with Cats Effect?

## Introduction

[`interop-cats`](https://github.com/zio/interop-cats) has instances for the [Cats](https://typelevel.org/cats/), [Cats MTL](https://github.com/typelevel/cats-mtl) and [Cats Effect](https://typelevel.org/cats-effect/) libraries, which allow you to use ZIO with any libraries that rely on these, like [Doobie](https://github.com/tpolecat/doobie), [Http4s](https://github.com/http4s/http4s), [FS2](https://github.com/functional-streams-for-scala/fs2) or [Circe](https://github.com/circe/circe)

Depends on which version of Cats Effect we are using, we should pick the right version of `zio-interop-cats`. In this tutorial, whenever we're working with Cats Effect 2.x, we are using:

```scala
libraryDependencies += "dev.zio" %% "zio-interop-cats" % "2.5.1.0"
```

And whenever we are using Cats Effect 3.x instances, we are using:

```scala
libraryDependencies += "dev.zio" %% "zio-interop-cats" % "3.1.1.0"
```

Most of the interop functionality resides in the following package:

```scala

```

## Cats Effect Instances

ZIO integrates with Typelevel libraries by providing instances of Cats Effect type classes. These type classes are used by `fs2`, `doobie`, `http4s`, and a variety of other libraries in the functional Scala ecosystem.

Due to the limitations of the Cats Effect, ZIO cannot provide instances for arbitrary error types. Instead, we can obtain instances only for effects whose error type extends `Throwable`.

For convenience, ZIO includes the `Task` and `RIO` type aliases, which fix the error type to `Throwable`, and may be useful for interop with Cats Effect:

```scala

```

```scala
type Task[+A]    = ZIO[Any, Throwable, A]
type RIO[-R, +A] = ZIO[  R, Throwable, A]
```

### Providing Runtime Manually

To use Cats Effect instances for these types, we should have an implicit `Runtime[R]` in scope for the environment type of our effects. The following code snippet creates an implicit `Runtime` for all the modules built into ZIO:

```scala

object ZioCatsEffectInterop extends scala.App {
  def catsEffectApp[F[_]: cats.effect.Sync]: F[Unit] =
    cats.effect.Sync[F].delay(
        println("Hello from Cats Effect World!")
      )
      
  implicit val runtime: zio.Runtime[Any] = zio.Runtime.default

  val zioApp: zio.Task[Unit] = catsEffectApp[zio.Task]
  runtime.unsafeRun(zioApp.exitCode)
}
```

If we are working with Cats Effect 3.x, the `catsEffectApp[Task]` will be expanded as if we called the following code explicitly:

```scala

```

```scala
object ZioCatsEffectInterop extends scala.App {
  val runtime: zio.Runtime[Any] = zio.Runtime.default
  
  val zioApp: zio.Task[Unit] = catsEffectApp[zio.Task](
    zio.interop.catz.asyncRuntimeInstance(runtime) 
  )
  
  runtime.unsafeRun(zioApp.exitCode) 
}
```

And if we are working with Cats Effect 2.x, it will be expanded as if we called following code explicitly:

```scala
object ZioCatsEffectInterop extends scala.App {
  val runtime: zio.Runtime[Any] = zio.Runtime.default

  val zioApp = catsEffectApp[zio.Task](zio.interop.catz.taskConcurrentInstance)
  runtime.unsafeRun(zioApp.exitCode)
}
```

If we are using `RIO` for a custom environment `R`, then we will have to create our own `Runtime[R]`, and ensure that implicit wherever we need Cats Effect instances.

### Using `CatsApp` Runtime

As a convenience, our application can extend `CatsApp`, which automatically brings an implicit `Runtime[Any]` into our scope:

```scala

object ZioCatsEffectInteropWithCatsApp extends CatsApp {
  def catsEffectApp[F[_]: cats.effect.Sync]: F[Unit] =
    cats.effect.Sync[F].delay(println("Hello from Cats Effect World!"))

  override def run(args: List[String]): zio.URIO[Any, zio.ExitCode] = 
    catsEffectApp[zio.Task].exitCode
}
```

This example works properly in both Cats Effect 2.x and 3.x versions.

## Cats Effect 2.x

### Timer Instance

In order to get a `cats.effect.Timer[zio.Task]` instance, we need an extra import (`zio.interop.catz.implicits._`):

```scala

object ZioCatsEffectTimerInterop extends zio.interop.catz.CatsApp {
  override def run(args: List[String]): zio.URIO[Any, zio.ExitCode] =
    catsEffectTimerApp[zio.Task].exitCode

  def catsEffectTimerApp[F[_]: cats.effect.Clock: cats.effect.Timer: cats.effect.Sync]: F[Unit] = for {
    t2 <- cats.effect.Clock[F].monotonic(TimeUnit.SECONDS)
    _  <- cats.effect.Timer[F].sleep(2.seconds)
    t1 <- cats.effect.Clock[F].monotonic(TimeUnit.SECONDS)
    _  <- cats.effect.Sync[F].delay(println(t1 - t2))
  } yield ()
}
```

The reason a `cats.effect.Timer[zio.Task]` instance is not provided by the default _interop_ import is that it makes testing programs that require timing capabilities very difficult. The extra import (wherever needed) makes reasoning about timing-related effects much easier.

If we're using `RIO` for a custom environment then our environment must use the `Clock` service, e.g. `R <: Clock` to get a timer.

### Converting Resource to ZManaged

We have an extension method defined on `Resource` called `Resource#toManaged` which converts `Resource` to `ZManaged`.
For example, assume we have the following `File` API:

```scala
case class File[F[_]: cats.effect.Sync]() {

  def read: F[String] =
    cats.effect.Sync[F].delay(println("Reading file.")) *>
      cats.effect.Sync[F].pure("Hello, World!")
  def close: F[Unit]  =
    cats.effect.Sync[F].delay(println("Closing file."))
}

object File {

  def open[F[_]: cats.effect.Sync](name: String): F[File[F]] =
    cats.effect.Sync[F].delay(println(s"opening $name file")) *>
      cats.effect.Sync[F].delay(File())
}
```

And, also assume we have `fileResource` defined as follows:

```scala
def fileResource[F[_]: cats.effect.Sync](name: String): cats.effect.Resource[F, File[F]] =
  cats.effect.Resource.make(File.open[F](name))(_.close)
```

Let's convert that to `ZManaged`:

```scala
val resource: zio.ZManaged[Any, Throwable, File[zio.Task]] =
  fileResource[zio.Task]("log.txt").toManaged
```

Here is a complete working example:

```scala

object CatsEffectResourceInterop extends CatsApp {
  def fileResource[F[_]: cats.effect.Sync](name: String): cats.effect.Resource[F, File[F]] =
    cats.effect.Resource.make(File.open[F](name))(_.close)

  def myApp: zio.ZIO[Any, Throwable, Unit] = for {
    c <- fileResource[zio.Task]("log.txt").toManaged.use(_.read)
    _ <- zio.Console.printLine(s"file content: $c")
  } yield ()

  override def run(args: List[String]): zio.URIO[Any, zio.ExitCode] =
    myApp.exitCode
}
```

### Converting ZManaged to Resource

We have an extension method on `ZManaged` called `ZManaged#toResource` which converts a ZIO managed resource to Cats Effect resource:

```scala 
final class ZManagedSyntax[R, E, A](private val managed: ZManaged[R, E, A]) {
  def toResource[F[_]](implicit
      F: Async[F],
      ev: Effect[ZIO[R, E, *]]
  ): Resource[F, A] = ???
}
```

Let's try an example:

```scala

object ZManagedToResource extends cats.effect.IOApp {
  implicit val zioRuntime: zio.Runtime[Any] = zio.Runtime.default

  val resource: cats.effect.Resource[cats.effect.IO, java.io.InputStream] =
    zio.ZManaged
      .fromAutoCloseable(
        zio.ZIO.effect(
          java.nio.file.Files.newInputStream(
            java.nio.file.Paths.get("file.txt")
          )
        )
      )
      .toResource[cats.effect.IO]

  val effect: cats.effect.IO[Unit] =
    resource
      .use { is =>
        cats.effect.IO.delay(is.readAllBytes())
      }
      .flatMap(bytes =>
        cats.effect.IO.delay(
          println(s"file length: ${bytes.length}")
        )
      )

  override def run(args: List[String]): cats.effect.IO[cats.effect.ExitCode] =
    effect.as(cats.effect.ExitCode.Success)
}
```
### Converting Resource to Scoped

We have an extension method defined on `Resource` called `Resource#toScoped` which converts `Resource` to `ZIO` with `Scope`.

For example, assume we have the following `File` API:

```scala
case class File[F[_]: cats.effect.Sync]() {

  def read: F[String] =
    cats.effect.Sync[F].delay(println("Reading file.")) *>
      cats.effect.Sync[F].pure("Hello, World!")
  def close: F[Unit]  =
    cats.effect.Sync[F].delay(println("Closing file."))
}

object File {

  def open[F[_]: cats.effect.Sync](name: String): F[File[F]] =
    cats.effect.Sync[F].delay(println(s"opening $name file")) *>
      cats.effect.Sync[F].delay(File())
}
```

And, also assume we have `fileResource` defined as follows:

```scala
def fileResource[F[_]: cats.effect.Sync](name: String): cats.effect.Resource[F, File[F]] =
  cats.effect.Resource.make(File.open[F](name))(_.close)
```

Let's convert that to scoped `ZIO`:

```scala
val scoped: ZIO[Scope, Throwable, File[zio.Task]] =
  fileResource[zio.Task]("log.txt").toScoped
```

Here is a complete working example:

```scala

object CatsEffectResourceInterop extends CatsApp {
  def fileResource[F[_]: cats.effect.Sync](name: String): cats.effect.Resource[F, File[F]] =
    cats.effect.Resource.make(File.open[F](name))(_.close)

  def myApp: zio.ZIO[Scope, Throwable, Unit] = for {
    c <- fileResource[zio.Task]("log.txt").toScoped
    _ <- zio.Console.printLine(s"file content: $c")
  } yield ()

  override def run(args: List[String]): zio.URIO[Scope, zio.ExitCode] =
    myApp.exitCode
}
```

### Converting Scoped to Resource

We have an extension method defined on `Resource` called `Resource#scoped` which creates a `Resource` from `ZIO` with `Scope`.

Let's try an example:

```scala

object ZManagedToResource extends cats.effect.IOApp {
  implicit val zioRuntime: zio.Runtime[Any] = zio.Runtime.default

  val resource: cats.effect.Resource[cats.effect.IO, java.io.InputStream] = {
    val scopedZIO: ZIO[Any with Scope, Throwable, InputStream]= ZIO
      .fromAutoCloseable(
        zio.ZIO.attempt(
          java.nio.file.Files.newInputStream(
            java.nio.file.Paths.get("crawl.log")
          )
        )
      )
    
    Resource.scoped[IO, Any](scopedZIO)
  }

  val effect: cats.effect.IO[Unit] =
    resource
      .use { is =>
        cats.effect.IO.delay(is.readAllBytes())
      }
      .flatMap(bytes =>
        cats.effect.IO.delay(
          println(s"file length: ${bytes.length}")
        )
      )

  override def run(args: List[String]): cats.effect.IO[cats.effect.ExitCode] =
    effect.as(cats.effect.ExitCode.Success)
}
```

## Cats Effect 3.x

### Type class Instances for ZIO's Task

ZIO integrates with Cats Effect 3.x as well as 2.x. The `interop-cats` module provides `Concurrent`, `Temporal` and `Async` for `zio.Task`.

An example of ZIO interoperability with CE3:

```scala

object ZioCatsEffectInterop extends zio.interop.catz.CatsApp {

  def catsEffectTimerApp[F[_]: cats.effect.Async]: F[Unit] = for {
    t2 <- cats.effect.Clock[F].monotonic
    _  <- cats.effect.Temporal[F].sleep(2.seconds)
    t1 <- cats.effect.Clock[F].monotonic
    _  <- cats.effect.Sync[F].delay(println(t1 - t2))
  } yield ()
  
  override def run(args: List[String]): zio.URIO[Any, zio.ExitCode] = {
    catsEffectTimerApp[zio.Task].exitCode
  }
}
```

## Cats Core

There is another package in `interop-cats` module called `zio.interop.catz.core._` which helps us to interop with core data types. This package contains instances of `zio.Chunk` data type for Cats Core module like `cats.Order`, `cats.Hash`, `cats.Traverse`, and so forth.

In the following example, we are going to use `zio.Chunk` in a Cats Effect application:

```scala

object ZioInteropWithCatsCore extends cats.effect.IOApp {
  val chunk = zio.Chunk("1", "2", "3", "4", "5")

  def parseInt(s: String): Option[Int] =
    Either.catchOnly[NumberFormatException](s.toInt).toOption

  val parseAll = cats.Traverse[zio.Chunk].traverse(chunk)(parseInt)

  override def run(args: List[String]): cats.effect.IO[cats.effect.ExitCode] =
    cats.effect.IO.println(parseAll).as(cats.effect.ExitCode.Success)
}
```

This package also contains utilities to support `zio.NonEmptyChunk` interoperability with Cats Core module.

## FS2 Streams

The `interop-cats` module contains extension methods to convert _FS2 Stream_ to _ZStream_ and vice versa. These methods support both FS2 series, 2.x and 3.x:

### From FS2 Stream to ZStream

By importing `zio.stream.interop.fs2z._` into our application, the `fs2.Stream#toZStream` extension method converts a `fs2.Stream` to `ZStream`:

```scala

val zstream: ZStream[Any, Throwable, Int] = fs2.Stream.range(1, 10).toZStream()
```
### From ZStream to FS2 Stream

Also, the `ZStream#toFs2Stream` converts a ZIO Stream into FS2 Stream:

```scala

val fs2stream = ZStream.fromChunks(Chunk(1, 2, 3, 4)).toFs2Stream
```

## Using Queue with Cats Effect
The `interop-cats` library has an `import zio.interop.Queue` package to lift creation of `Queue` effect from `UIO[Queue[A]]` to `F[Queue[F, A]]` which enables us to run `Queue` under Cats Effect runtime. It supports all variants of `Queue` like `bounded`, `unbounded`, `sliding` and `dropping`.

```scala
def bounded[F[+_], A](capacity: Int)(implicit R: Runtime[ZEnv], F: LiftIO[F]): F[Queue[F, A]]
```

### Cats Effect 2.x

In the following example, we are going to lift the `Queue` creation effect to Cats `IO` effect. If we are integrating with 2.x Cats Effect library, this snippet works properly:

```scala

implicit val runtime = Runtime.default
def liftedToIO: IO[List[Int]] = for {
  q <- Queue.bounded[IO, Int](100)
  _  <- q.offer(1)
  _  <- q.offer(2)
  r <- q.takeAll
} yield (r)
```

### Cats Effect 3.x

To run `Queue` with Cats Effect 3.x we also need to provide an instance of `Dispatcher` to our contextual environment:

```scala

object ZioQueueInteropWithCats extends scala.App {

  implicit val ceRuntime: cats.effect.unsafe.IORuntime =
    cats.effect.unsafe.IORuntime.global

  implicit val zioRuntime: zio.Runtime[Any] =
    zio.Runtime.default

  implicit val ec: scala.concurrent.ExecutionContextExecutor =
    scala.concurrent.ExecutionContext.global

  implicit val dispatcher: cats.effect.std.Dispatcher[cats.effect.IO] =
    cats.effect.std
      .Dispatcher[cats.effect.IO]
      .allocated
      .unsafeRunSync()
      ._1

  def liftedToIO: cats.effect.IO[List[Int]] = for {
    q <- Queue.bounded[cats.effect.IO, Int](100)
    _ <- q.offer(1)
    _ <- q.offer(2)
    r <- q.takeAll
  } yield (r)

  val catsApp = liftedToIO
    .flatMap { e =>
      cats.effect.IO.println(s"List of elements retrieved from Queue: $e")
    }
    .as(cats.effect.ExitCode.Success)

  catsApp.unsafeRunSync()
}
```

## Using STM with Cats Effect

The `zio.interop.stm` provides a wrapper data type on `STM[Throwable, A]` which enables us to run `STM` with the Cats Effect library.

Currently, the `interop-cats` support `TRef`, `TPromise`,  `TQueue` and `TSemaphore` data types.

Let's try a working example using `STM` and `TRef`:

```scala

implicit val zioRuntime: zio.Runtime[Any] = zio.Runtime.default
implicit val catsRuntime: IORuntime            = IORuntime.global

def transferMoney(
    from: TRef[IO, Long],
    to: TRef[IO, Long],
    amount: Long
): STM[IO, Long] =
  for {
    senderBal <- from.get
    _ <-
      if (senderBal < amount)
        STM.fail[IO](new Exception("Not enough money"))
      else
        STM.unit[IO]
    _       <- from.update(existing => existing - amount)
    _       <- to.update(existing => existing + amount)
    recvBal <- to.get
  } yield recvBal

val program: IO[Long] = for {
  sndAcc <- STM.atomically[cats.effect.IO, TRef[IO, Long]](
    TRef.make[IO, Long](1000)
  )
  rcvAcc <- STM.atomically[cats.effect.IO, TRef[IO, Long]](
    TRef.make[IO, Long](200)
  )
  recvAmt <- STM.atomically(transferMoney(sndAcc, rcvAcc, 500L))
} yield recvAmt

program
  .flatMap(amount =>
    IO.println(s"Balance of second account after transaction: $amount")
  )
  .unsafeRunSync()
```

## Examples

Cats Effect and Type-Level libraries are older than the ZIO ecosystem. So there are very nice libraries like doobie and http4s, that a ZIO user would like to use in its application.

We have provided some full working example of using these important libraries:

### Using ZIO with Doobie

The following example shows how to use ZIO with Doobie (a library for JDBC access) and FS2 (a streaming library), which both rely on Cats Effect instances:

```scala
// This snippet works with both CE2 and CE3

implicit val zioRuntime: zio.Runtime[Any] = zio.Runtime.default

case class User(id: String, name: String, age: Int)

def xa: Transactor[Task] =
  Transactor.fromDriverManager[Task](
    "org.h2.Driver",
    "jdbc:h2:mem:users;DB_CLOSE_DELAY=-1"
  )

def createTable: doobie.ConnectionIO[Int] =
  sql"""|CREATE TABLE IF NOT EXISTS USERS(
        |id   INT SERIAL UNIQUE,
        |name VARCHAR NOT NULL UNIQUE,
        |age  SMALLINT
        |)""".stripMargin.update.run

def dropTable: doobie.ConnectionIO[Int] =
  sql"""DROP TABLE IF EXISTS USERS""".update.run

def insert(name: String, age: Int): doobie.ConnectionIO[Int] =
  sql"insert into users (name, age) values ($name, $age)".update.run

def loadUsers: Stream[doobie.ConnectionIO, User] =
  sql"""SELECT * FROM users""".query[User].stream

val doobieApp: Stream[doobie.ConnectionIO, User] = for {
  _ <- fs2.Stream.eval(dropTable)
  _ <- fs2.Stream.eval(createTable)
  _ <- fs2.Stream.eval(insert("Olivia", 21))
  _ <- fs2.Stream.eval(insert("Oliver", 30))
  u <- loadUsers
} yield u

val run: Stream[Task, User] = doobieApp.transact(xa)

val allUsers: List[User] =
  zioRuntime.unsafeRun(run.compile.toList)
```

Sounds good, but how can we specify a specialized transactor than the default one? Creating a customized transactor in CE2 differs from CE3. 

Let's try doing that in each of which:

#### Customized Transactor (CE2)

ZIO provides a specific blocking thread pool for blocking operations. The `doobie-hikari` module helps us create a transactor with two separated executors, one for blocking operations, and the other one for non-blocking operations. So we shouldn't run blocking JDBC operations or perform awaiting connections to the database on the main thread pool.

So let's fix this issue in the previous example. In the following snippet we are going to create a `ZMHikari` of Hikari transactor. In this example we are using `0.13.4` version of doobie which supports CE2:

```scala

def transactor: ZManaged[Any, Throwable, HikariTransactor[Task]] =
  for {
    rt <- ZIO.runtime[Any].toManaged
    be <- ZIO.blockingExecutor.toManaged                          // our blocking EC
    xa <- HikariTransactor
            .newHikariTransactor[Task](
              "org.h2.Driver",                                    // driver classname
              "jdbc:h2:mem:test;DB_CLOSE_DELAY=-1",               // connect URL
              "sa",                                               // username
              "",                                                 // password
              be.asExecutionContext,                              // await connection here
              Blocker.liftExecutionContext(be.asExecutionContext) // execute JDBC operations here
            )
            .toManagedZIO
  } yield xa
```

Now we can `transact` our `doobieApp` with this `transactor` and convert that to the `ZIO` effect:

```scala
val zioApp: ZIO[Any, Throwable, List[User]] =
  transactor.use(xa => doobieApp.transact(xa).compile.toList)
```

#### Customized Transactor (CE3)

In Cats Effect 3.x, the `cats.effect.Blocker` has been removed. So the transactor constructor doesn't require us a blocking executor; it happens under the hood using the `Sync[F].blocking` operation.

To create a `Transactor` in CE3, we need to create an instance of `Dispatcher` for `zio.Task`. The following example is based on Doobie's `1.0.0-M5` version which supports CE3:

```scala

implicit val zioRuntime: zio.Runtime[Any] =
  zio.Runtime.default

implicit val dispatcher: cats.effect.std.Dispatcher[zio.Task] =
  zioRuntime
    .unsafeRun(
      cats.effect.std
        .Dispatcher[zio.Task]
        .allocated
    )
    ._1

def transactor: ZManaged[Any, Throwable, HikariTransactor[Task]] =
  for {
    rt <- ZIO.runtime[Any].toManaged
    xa <-
      HikariTransactor
        .newHikariTransactor[Task](
          "org.h2.Driver",                             // driver classname
          "jdbc:h2:mem:test;DB_CLOSE_DELAY=-1",        // connect URL
          "sa",                                        // username
          "",                                          // password
          rt.runtimeConfig.executor.asExecutionContext // await connection here
        )
        .toManaged
  } yield xa
```

Now we can `transact` our `doobieApp` with this `transactor` and convert that to the `ZIO` effect:

```scala
val zioApp: ZIO[Any, Throwable, List[User]] =
  transactor.use(xa => doobieApp.transact(xa).compile.toList)
```

### Http4s

Here is the full working example of using http4s in ZIO App:

#### Cats Effect 2.x

The following example is based on http4s's `0.21.24` version which supports CE2:

```scala

object ZioHttp4sInterop extends CatsApp {
  def run(args: List[String]): URIO[Any, zio.ExitCode] =
    stream[Task].compile.drain.exitCode
  
  def stream[F[_]: ConcurrentEffect: Timer]: Stream[F, Nothing] = {

    val httpApp = helloWorldRoute[F].orNotFound
    for {
      _ <- BlazeClientBuilder[F](global).stream
      exitCode <- BlazeServerBuilder[F](global)
        .bindHttp(8080, "0.0.0.0")
        .withHttpApp(httpApp)
        .serve
    } yield exitCode
  }.drain

  def helloWorldRoute[F[_]: Sync]: HttpRoutes[F] = {
    val dsl = new Http4sDsl[F]{}

    HttpRoutes.strict[F] {
      case GET -> Root =>
        Ok("Hello, World!")
    }
  }
}
```

#### Cats Effect 3.x

The following example is based on http4s's `0.23.0-RC1` version which supports CE3:

```scala

object ZioHttp4sInterop extends zio.interop.catz.CatsApp {
  def stream[F[_]: Async]: Stream[F, Nothing] = {

    val httpApp = helloWorldRoute[F].orNotFound
    for {
      _ <- BlazeClientBuilder[F](global).stream
      exitCode <- BlazeServerBuilder[F](global)
        .bindHttp(8080, "0.0.0.0")
        .withHttpApp(httpApp)
        .serve
    } yield exitCode
  }.drain

  def helloWorldRoute[F[_]: Applicative]: HttpRoutes[F] = {
    val dsl = new Http4sDsl[F] {}

    HttpRoutes.strict[F] { case GET -> Root =>
      Ok("Hello, World!")
    }
  }

  def run(args: List[String]): URIO[Any, zio.ExitCode] =
    stream[Task].compile.drain.exitCode
}
```

---

## How to Interop with Future?

## Scala Future

Basic interoperability with Scala's `Future` is now provided by ZIO, and does not require a separate module.

### From Future

Scala's `Future` can be converted into a ZIO effect with `ZIO.fromFuture`:

```scala
def loggedFuture[A](future: ExecutionContext => Future[A]): UIO[Task[A]] = {
  ZIO.fromFuture { implicit ec =>
    future(ec).flatMap { result =>
      Future(println("Future succeeded with " + result)).map(_ => result)
    }
  }
}
```

Scala's `Future` can also be converted into a `Fiber` with `Fiber.fromFuture`:

```scala
def futureToFiber[A](future: => Future[A]): Fiber[Throwable, A] = 
  Fiber.fromFuture(future)
```

This is a pure operation, given any sensible notion of fiber equality.

### To Future

A ZIO `Task` effect can be converted into a `Future` with `ZIO#toFuture`:

```scala
def taskToFuture[A](task: Task[A]): UIO[Future[A]] = 
  task.toFuture
```

Because converting a `Task` into an (eager) `Future` is effectful, the return value of `ZIO#toFuture` is an effect. To actually begin the computation, and access the started `Future`, it is necessary to execute the effect with a runtime.

A ZIO `Fiber` can be converted into a `Future` with `Fiber#toFuture`:

```scala
def fiberToFuture[A](fiber: Fiber[Throwable, A]): UIO[Future[A]] = 
  fiber.toFuture
```

## Run to Future

You can call `.unsafe.runToFuture` on an instance of `Runtime` to execute a ZIO effect asynchronously and return a `Future` that will be completed when the execution of the effect is complete.

```scala

...
val zio: Task[String] = ...
val runtime = Runtime.default
val fut: Future[String] = Unsafe.unsafe { implicit unsafe =>
  runtime.unsafe.runToFuture(zio)
}
```

---

## How to Interop with Java?

ZIO has full interoperability with foreign Java code. Let me show you how it works and then *BOOM*, tomorrow you can show off your purely functional Java at work.

ZIO has built-in conversion between ZIO data types (like `ZIO` and `Fiber`) and Java concurrent data types like [`CompletionStage`](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html), [`Future`](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html) and [`CompletionHandler`](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/CompletionHandler.html).
 
## From Java CompletionStage and back

`CompletionStage` is the interface that comes closest to emulate a functional asynchronous effects API like ZIO's, so we start with it. It's a breeze:

```scala
def loggedStage[A](stage: => CompletionStage[A]): Task[A] =
    ZIO.fromCompletionStage(UIO {
        stage.thenApplyAsync { a =>
            println("Stage completed with " + a)
            a
        }
    })
```

By Jove, you can even turn it into fiber!

```scala
def stageToFiber[A](stage: => CompletionStage[A]): Fiber[Throwable, A] = 
  Fiber.fromCompletionStage(future)
````

This API creates a synthetic fiber which doesn't have any notion of identity.

Additionally, you may want to go the other way and convert a ZIO value into a `CompletionStage`. Easy as pie:

```scala
def taskToStage[A](task: Task[A]): UIO[CompletableFuture[A]] =
    task.toCompletableFuture
```

As you can see, it commits to a concrete class implementing the `CompletionStage` interface, i.e. `CompletableFuture`. It is worth to point out that any `IO[E, A]` can be turned into a completable future provided you can turn a value of type `E` into a `Throwable`:

```scala
def ioToStage[E, A](io: IO[E, A])(toThrowable: E => Throwable): UIO[CompletableFuture[A]] =
    io.toCompletableFutureWith(toThrowable)
```

## Java Future

You can embed any `java.util.concurrent.Future` in a ZIO computation via `ZIO.fromFutureJava`. A toy wrapper around Apache Async HTTP client could look like:

```scala
def execute(client: HttpAsyncClient, request: HttpUriRequest): RIO[Blocking, HttpResponse] =
    ZIO.fromFutureJava(UIO {
        client.execute(request, null)
    })
```

That's it. Just a bit of a warning here, mate. As you can see from the requirement on the produced value, ZIO uses the blocking `Future#get` call internally. It is running on the blocking thread pool, of course, but I thought you should know. If possible, use `ZIO.fromCompletionStage` instead, as detailed above.

Should you need it, it is also possible to convert a future into a fiber using `Fiber.fromFutureJava`. Same same, but different:

```scala
def execute(client: HttpAsyncClient, request: HttpUriRequest): Fiber[Throwable, HttpResponse] =
    Fiber.fromFutureJava {
        client.execute(request, null)
    }
```

## NIO Completion handler

Java libraries using channels from the NIO API for asynchronous, interruptible I/O can be hooked into by providing completion handlers. As in, reading the contents of a file:

```scala
def readFile(file: AsynchronousFileChannel): Task[Chunk[Byte]] = for {
    pos <- Ref.make(0)
    buf <- ZIO.succeed(ByteBuffer.allocate(1024))
    contents <- Ref.make[Chunk[Byte]](Chunk.empty)
    def go = pos.get.flatMap { p =>
        ZIO.asyncWithCompletionHandler[Chunk[Byte]] { handler =>
            file.read(buf, p, buf, handler)
        }.flatMap {
            case -1 => contents.get
            case n  =>
                ZIO.succeed {
                    val arr = Array.ofDim[Byte](n)
                    buf.get(arr, 0, n)
                    buf.clear()
                    Chunk.fromArray(arr)
                }.flatMap { slice =>
                    contents.update(_ ++ slice)
                } *> pos.update(_ + n) *> go
        }
    }
    dump <- go
} yield dump
```

As you can see, ZIO provides a CPS-style API here which is a bit different from the two sections above, but hey still super elegant.

---

## How to Interop with JavaScript?

Include ZIO in your Scala.js project by adding the following to your `build.sbt`:

```
scalaJSUseMainModuleInitializer := true
resolvers += Resolver.sonatypeRepo("snapshots")
libraryDependencies += "dev.zio" %%% "zio" % "2.1.26+32-45b68c11-SNAPSHOT"
```

## Example

Your main function can extend [`ZIOAppDefault`](../../reference/core/zioapp.md) as follows.

This example uses [scala-js-dom](https://github.com/scala-js/scala-js-dom) to access the DOM; to run the example you
will need to add that library as a dependency to your `build.sbt`.

It assumes you have a ``  somewhere in your HTML, into which the ZIO output is written.

```scala

object Main extends ZIOAppDefault {

  override def run = {
    // The node into which we write our ZIO program output
    val node: Element = dom.document.querySelector("#app")
    for {
      _      <- Console.printLine("Starting progress bar demo.")  // Outputs on browser console log.
      _      <- IO.succeed(node.appendChild(target)) // "node" might provided in this page by mdoc.
      target <- IO.succeed(document.createElement("pre"))
      _      <- update(target).repeat(Schedule.spaced(1.seconds))
    } yield ExitCode.success
  }

  def update(target: Element) = {
      for {
        time   <- currentTime(TimeUnit.SECONDS)
        output <- ZIO.succeed(progress((time % 11).toInt, 10))
        _      <- ZIO.succeed(target.innerHTML = output)
      } yield ()
  }

  def progress(tick: Int, size: Int) = {
      val bar_length = tick
      val empty_length = size - tick
      val bar = "#" * bar_length + " " * empty_length
      s"$bar $bar_length%"
  }

}
```

---

## How to Migrate From Akka to ZIO?

## Overview

Here, we summarized alternative ZIO solutions for Akka Actor features. So before starting the migration, let's see an overview of corresponding features in ZIO:

| Topics                      | Akka                                        | ZIO                                        |
|-----------------------------|---------------------------------------------|--------------------------------------------|
| Parallelism                 | [Akka Actor][1]                             | [ZIO][2], [Concurrent Data Types][3]       |
| Concurrent State Management | [Akka Actor][1]                             | [Ref][75], [FiberRef][76], [ZState][77]    |
| Buffering Workloads         | [Akka Mailboxes][78]                        | [Queue][79]                                |
| Streaming                   | [Akka Streams][4]                           | [ZIO Streams][5]                           |
| HTTP Applications           | [Akka Http][55]                             | [ZIO HTTP][56]                             |
| Event Sourcing              | [Lagom Framework][6], [Akka Persistence][8] | [ZIO Entity][7], [Edomata][9]              |
| Entity Sharding             | [Akka Cluster Sharding][74]                 | [Shardcake][73]                            |
| Scheduling                  | [Akka Scheduler][10]                        | [Schedule data type][11]                   |
| Cron-like Scheduling        | [Akka Quartz Scheduler][12]                 | [Schedule data type][11]                   |
| Resiliency                  | [Akka CircuitBreaker][13]                   | [Schedule data type][11], [Rezilience][14] |
| Logging                     | [Built-in Support][15]                      | [Built-in Support][16], [ZIO Logging][92]  |
| Testing                     | [Akka Testkit][17]                          | [ZIO Test][18]                             |
| Testing Streams             | [Akka Stream Testkit][19]                   | [ZIO Test][18]                             |
| Metrics                     | [Cluster Metric Extension][20]              | [Metrics][21], [ZIO Metrics][93]           |
| Supervision                 | [Yes][22]                                   | Yes                                        |
| Monitoring                  | [Yes][22]                                   | Yes                                        |

There are also several integration libraries for Akka that cover a wide range of technologies. If you use any of these technologies, you have a chance to use the equivalent of them in the ZIO ecosystem:

| Tools                | Alpakka                            | ZIO Connect                            |
|----------------------|------------------------------------|----------------------------------------|
| gRPC                 | [Akka gRPC][23]                    | [ZIO gRPC][24]                         |
| GraphQL              | [Sangria][25]                      | [Caliban][26]                          |
| Apache Kafka         | [Alpakka Kafka][27]                | [ZIO Kafka][28]                        |
| AWS                  |                                    | [ZIO AWS][30]                          |
| AWS S3               | [Alpakka S3][29]                   | [ZIO S3][32]                           |
| AWS SNS              | [Alpakka SNS][31]                  | [ZIO AWS SNS][30]                      |
| AWS SQS              | [Alpakka SQS][33]                  | [ZIO SQS][34], [ZIO AWS SQS][30]       |
| AMQP                 | [Alpakka AMQP][35]                 | [ZIO AMQP][36]                         |
| AWS Kinesis          | [Alpakka Kinesis][37]              | [ZIO Kinesis][38]                      |
| AWS DynamoDB         | [Alpakka DynamoDB][39]             | [ZIO DynamoDB][40]                     |
| Pulsar               | [Pulsar4s][41]                     | [ZIO Pulsar][42]                       |
| AWS Lambda           | [Alpakka AWS Lambda][43]           | [ZIO Lambda][44], [ZIO AWS Lambda][30] |
|                      | [Alpakka Cassandra][45]            | [ZIO Cassandra][46]                    |
| Elasticsearch        | [Alpakka Elasticsearch][47]        | [ZIO Elasticsearch][48]                |
| FTP                  | [Alpakka FTP][49]                  | [ZIO FTP][50]                          |
|                      |                                    | [ZIO MongoDB][51]                      |
| Redis                |                                    | [ZIO Redis][52]                        |
| Data Codecs          | [Alpakka Avro Parquet][53]         | [ZIO Schema][54]                       |
|                      |                                    | [ZIO NIO][57]                          |
| Slick                | [Alpakka Slick][58]                | [ZIO Slick Interop][59]                |
| Google Cloud Pub/Sub | [Alpakka Google Cloud Pub/Sub][62] | [ZIO GCP Pub/Sub][63]                  |
| Google Cloud Storage | [Alpakka Google Cloud Storage][64] | [ZIO GCP Storage][63]                  |
| Json                 | [Alpakka JSON Streaming][65]       | [ZIO JSON][66]                         |
| OrientDB             | [Alpakka OrientDB][67]             | [ZIO Quill OrientDB][68]               |
| Logging              | [Akka SL4J][69]                    | [ZIO Logging SLF4J][70]                |
| Caching              | [Akka HTTP Caching][71]            | [ZIO Cache][72]                        |

We also have several other libraries that may not be covered by the Akka ecosystem, but you can still use them. So we encourage you to check the ecosystem section of the ZIO website; take a look at the libraries, and see if they are suitable for your requirements for the migration process.

## Akka From the Perspective of a ZIO Developer

### They Are Not Composable

Akka actors are modeled as a partial function from `Any` to `Unit`:

```scala
type Actor = PartialFunction[Any, Unit]
```

In other words, an actor accepts something and does something with it. Both its input and output are not well typed. It is just like a blackbox. We know that having these types of functions is not composable. So it is hard to write small pieces of actors and compose them together to build large applications.

While in ZIO everything is composable. We can write composable computational abstractions and compose them together to build large applications. Due to the support for referential transparency, we can reason about small pieces of code and then make sure the whole application is correct.

### When All We Have Is A Hammer

The primary tool in object-oriented programming is object. Objects bundle the data and behavior. In OOP evertything is an object. So we have one tool to solve all problems.

Akka actors are the same. Each actor is an object which has its own state and behavior. Except that Akka actors are not synchronous. We can send messages to them asynchronously. This is where the Akka actors are different from the traditional objects. But the philosophy is the same: everything is an actor. So we have one tool to solve all problems.

But in reality, we don't need to use actors for everything. Many computations do not require any state. We just need to transform the input and produce the output. In ZIO We can use functions for them. Composing functions allows us to build large applications. Although if we need a state, we can use recursive functions or `Ref` to model the state without losing referential transparency and composability.

Other than actors, there are many lightweight tools that can solve concurrency and parallelism problems:

- [Fiber][80]
- [Ref][82]
- [Promise][84]
- [Semaphore][86]
- [Queue][79]
- [Hub][85]
- [Synchronization Primitives][81]

So it doesn't make sense to use actors for everything. It is better to choose the right tool for the right problem.

### Eager Evaluation

The primary evaluation strategy in ZIO is lazy evaluation. In other words, our code isn't evaluated until it's actually needed. So when we write ZIO applications, we are defining a description of the program. This enables us to model the program as a data structure. We can then transform the data structure to build a new program. So this makes us more productive by having reusable and composable abstractions.

So a ZIO application, is composed of series of ZIO values that are description of the whole workflow, finally when we provide the entire application to the `run` method, the ZIO runtime will execute the application.

In contrast, in Akka, the primary evaluation strategy is eager evaluation, except for some part (e.g. Akka Typed). So when we writing Akka application, we are writing a sequence of instructions. This makes it hard to reuse and compose the pieces of code. We can't transform the instructions to build a new program.

### Akka Actors And Futures

Akka actors are modeled on top of Scala `Future`. Both `Future` and `Actor` are asynchronous. Futures are mainly used in Akka by the `Ask` and `PipeTo` methods.

The `Future` is a data structure that represents a value that may not yet be available. It models the asynchronous computation, but the most problem with `Future` is that it is not referentially transparent. So even though we can compose `Future`s together, we can't reason about the whole program. As a result, we cannot ensure that the whole program is correct.

Furthermore, Futures are already running asynchronous constructs, so we cannot control their execution. For example, assume we have two `Future`s; we want to run them concurrently and return the first one that finishes. It is possible to do it with `Future`, but we cannot cancel the already running loser `Future`.

The `ZIO` data type is an alternative construct for `Future`. It is referentially transparent. We can compose `ZIO` values together and reason about the whole program. So we can make sure the entire program is correct.

ZIO support a nice interruption mechanism. We can easily cancel any running `ZIO` workflow using `Fiber#interrupt`. Also, there are high-level APIs on top of interruption. For example, we have `race` operators, where we can run multiple ZIO workflows and get the first successful result: `ZIO.succeed("Hello!").delay(1.second).race(ZIO.succeed("Hi!").delay(2.second))`. ZIO automatically cancels the loser computation.

## Akka Use-cases

Before starting the migration, we need to understand what types of use-cases developers use Akka for.

Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications. Here are the most common use-cases for Akka among developers:

1. [Parallelism](#1-parallelism)
2. [Concurrent State Management](#2-concurrent-state-management)
3. [Buffering Workloads](#3-buffering-workloads)
4. [Streaming](#4-streaming)
5. [HTTP Applications](#5-http-applications)
6. [Event Sourcing](#6-event-sourcing)
7. [Entity Sharding](#7-entity-sharding)
8. [Distributed Computing](#8-distributed-computing)

Let's see an example of each use-case in a simple application using Akka.

## 1. Parallelism

### Parallelism in Akka

We can achieve parallelism in Akka by creating multiple instances of an actor and sending messages to them. Akka takes care of how to route messages to these actors. In the following example, we have a simple `JobRunner` actor which accepts `Job` messages and runs them one by one:

```scala

case class Job(n: Int)

class JobRunner extends Actor {
  override def receive = { case Job(n) =>
    println(s"job$n — started")
    Thread.sleep(1000)
    println(s"job$n — finished")
  }
}
```

If we have plenty of jobs to run, we can create a pool of `JobRunner` actors and send them jobs to make them run in parallel:

```scala

object MainApp extends scala.App {
  val actorSystem = ActorSystem("parallel-app")
  val jobRunner = actorSystem.actorOf(
    Props[JobRunner].withRouter(RoundRobinPool(4)),
    "job-runner"
  )

  for (job <- (1 to 10).map(Job)) {
    jobRunner ! job
  }
}
```

### Parallelism in ZIO

In ZIO we can achieve the same functionality easily by using `ZIO.foreachPar` operators:

```scala

object MainApp extends ZIOAppDefault {

  def jobRunner(n: Int) =
    for {
      _ <- Console.printLine(s"job$n - started")
      _ <- ZIO.sleep(1.second)
      _ <- Console.printLine(s"job$n - finished")
    } yield ()

  val jobs = (1 to 10)
  
  def run = ZIO.foreachParDiscard(jobs)(jobRunner)
}
```

We use `ZIO.withParallelism` operator to change the default parallelism factor:

```scala
ZIO.withParallelism(4) {
  ZIO.foreachParDiscard(jobs)(jobRunner)
} 
```

## 2. Concurrent State Management

### State Management in Akka

The main purpose of Akka actors is to write concurrent stateful applications. Using Akka actors, we can have stateful actors without worrying about concurrent access to the shared state. In the following example, we have a simple `Counter` actor which accepts `inc` and `dec` messages and increments or decrements its internal state:

```scala

class Counter extends Actor {
  private var state = 0

  override def receive: Receive = {
    case "inc" =>
      state += 1
    case "dec" =>
      state -= 1
    case "get" =>
      sender() ! state
  }
}
```

Now we can create an instance of the `Counter` actor and send it `inc` and `dec` messages to increment and decrement its internal state:

```scala

object MainApp extends App {
  val system = ActorSystem("counter-app")
  val counterActor = system.actorOf(Props[Counter], "counter")

  implicit val ec: ExecutionContextExecutor = ExecutionContext.global
  implicit val timeout: Timeout = Timeout(1.second)

  counterActor ! "inc"
  counterActor ! "inc"
  counterActor ! "inc"
  counterActor ! "dec"

  (counterActor ? "get").onComplete {
    case Success(v) =>
      println(s"The current value of counter: $v")
    case Failure(e) =>
      println(s"Failed to receive the result from the counter: ${e.getMessage}")
  }

}
```

Outside the actor, we haven't access to its internal states, so we can't modify it directly. We can only send messages to the actor and let it handle the state management on its own. Using this approach, we can have safe concurrent state management. If multiple actors send messages to this actor concurrently, they can't make the state inconsistent.

### State Management in ZIO

State management is very easy in ZIO in presence of the `Ref` data type. `Ref` models a mutable state which is safe for concurrent access:

```scala

case class Counter(state: Ref[Int]) {
  def inc = state.update(_ + 1)
  def dec = state.update(_ - 1)
  def get = state.get
}

object Counter {
  def make = Ref.make(0).map(Counter(_))
}
```

That's it! Very simple! We now we can use the counter in a program:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      c <- Counter.make
      _ <- c.inc <&> c.inc <&> c.inc <&> c.dec
      _ <- c.get.debug("The current value of the counter")
    } yield ()
}
```

## 3. Buffering Workloads

Sometimes we have to deal with a high volume of incoming requests. In spite of parallelism and concurrency, we may not be able to handle all incoming requests within a short period of time. In such cases, we can use a buffer to store incoming requests temporarily and process them later.

### Buffering Workloads in Akka

Each actor in Akka has a mailbox that is used to store incoming messages. The mailbox is a FIFO queue. Actors only process one message at a time. If an actor receives more messages than it can process, the messages will be pending in the mailbox:

```scala

object MainApp extends scala.App {
  val actorSystem = ActorSystem("parallel-app")
  val worker = actorSystem.actorOf(Props[JobRunner], "worker")

  val jobs = (1 to 1000).map(Job)

  for (job <- jobs) {
    worker ! job
  }

  println("All messages were sent to the actor!")

  Await.result(actorSystem.whenTerminated, Duration.Inf)
}
```

If we run the above example, we can see that all messages are sent to the actor before, and the actor still processing messages from the mailbox one by one. By default, the mailbox is an unbounded FIFO queue. But we can also use a bounded mailbox or a custom priority mailbox.

### Buffering Workloads in ZIO

ZIO has a data type called `Queue` which is useful for buffering workloads:

```scala

trait Actor[-In] {
  def tell(i: In): UIO[Boolean]
}

object Actor {
  def make[In](receive: In => UIO[Unit]): ZIO[Scope, Nothing, Actor[In]] =
    for {
      queue <- Queue.unbounded[In]
      _     <- queue.take.flatMap(receive).forever.forkScoped
    } yield new Actor[In] {
      override def tell(i: In): UIO[Boolean] =
        queue.offer(i)
    }
}
```

Now we can send a high load of messages to the actor and the actor will process them one by one:

```scala

object MainApp extends ZIOAppDefault {
  def run = ZIO.scoped {
    for {
      actor <- Actor.make[Int](n => ZIO.debug(s"processing job-$n").delay(1.second))
      _     <- ZIO.foreachParDiscard(1 to 1000)(actor.tell)
      _     <- ZIO.debug("All messages were sent to the actor!")
      _     <- ZIO.never
    } yield ()
  }
}
```

## 4. Streaming

### Streaming in Akka

Akka stream is developed on top of Akka actors with backpressure support. There are three main components in Akka streams:

1. Source
2. Sink
3. Flow

Here is a simple example of how to have a streaming app in Akka:

```scala

object AkkaStreamApp extends App {
  implicit val system: ActorSystem = ActorSystem("stream")
  implicit val ec: ExecutionContextExecutor = system.dispatcher

  val source = Source(1 to 100)
  val factorial = Flow[Int].scan(BigInt(1))((acc, next) => acc * next)
  val serialize = Flow[BigInt].map(num => ByteString(s"$num\n"))
  val sink = FileIO.toPath(Paths.get("factorials.txt"))

  source
    .via(factorial)
    .via(serialize)
    .runWith(sink)
    .onComplete(_ => system.terminate())
}
```

### Streaming in ZIO

ZIO Streams is a purely functional, composable, effectful, and resourceful streaming library. It provides a way to model streaming data processing as a pure function. It is built on top of ZIO and supports backpressure using a pull-based model.

Like the Akka terminology, ZIO streams have three main components:

1. [ZStream](../../reference/stream/zstream/index.md)
2. [ZPipeline](../../reference/stream/zpipeline.md)
3. [ZSink](../../reference/stream/zsink/index.md)

Let's see how to implement the same example in ZIO:

```scala

object ZIOStreamApp extends ZIOAppDefault {
  val source    = ZStream.fromIterable(1 to 100)
  val factorial = ZPipeline.scan(BigInt(1))((acc, next: Int) => acc * next)
  val serialize = ZPipeline.map((num: BigInt) => Chunk.fromArray(s"$num".getBytes))
  val sink      = ZSink.fromFileName("factorials.txt")

  def run = 
    source
      .via(factorial)
      .via(serialize).flattenChunks
      .run(sink)
}
```

## 5. HTTP Applications

### HTTP Applications in Akka

Akka HTTP is a library for building HTTP applications on top of Akka actors and streams. It supports both server-side and client-side HTTP applications:

```scala

object AkkHttpServer extends App {
  implicit val system = ActorSystem(Behaviors.empty, "system")
  implicit val executionContext = system.executionContext

  Http().newServerAt("localhost", 8080).bind {
    path("hello") {
      get {
        complete(
          HttpEntity(
            ContentTypes.`text/html(UTF-8)`,
            "Say hello to akka-http"
          )
        )
      }
    }
  }
}
```

### HTTP Applications in ZIO

On the other hand, ZIO has a library called [ZIO HTTP][56] which is a pure functional library for building HTTP applications. It is on top of ZIO, ZIO Streams.

Let's see how the above Akka HTTP service can be written in ZIO:

```scala

object ZIOHttpServer extends ZIOAppDefault {
  val routes =
    Routes(
      Method.GET / "hello" -> handler(Response.html(Html.fromString("Say hello to zio-http")))
    )

  def run = Server.serve(routes).provide(Server.defaultWithPort(8080))
}
```

## 6. Event Sourcing

### Event Sourcing in Akka

In event sourcing, we store the events that happened in the past and use them to reconstruct the current state of the application. Akka has a built-in solution called Akka Persistence.

In the following example, we have a simple `PersistentCounter` actor which accepts `inc` and `dec` messages and increments or decrements in its internal state and also sores incoming events in persistent storage. When the actor is restarted, it will recover its state from the persistent storage:

```scala

class PersistentCounter extends PersistentActor {
  private var state: Int = 0

  override def receive = {
    case "inc" => persist("inc")(_ => state += 1)
    case "dec" => persist("dec")(_ => state -= 1)
    case "get" => sender() ! state
  }

  override def receiveRecover = {
    case "inc" => state += 1
    case "dec" => state -= 1
  }

  override def receiveCommand: Receive = _ => ()

  override def persistenceId: String = "my-persistence-id"
}

object MainApp extends App {
  val system = ActorSystem("counter-app")
  val counter = system.actorOf(Props[PersistentCounter], "counter")

  implicit val ec: ExecutionContextExecutor = ExecutionContext.global
  implicit val timeout: Timeout = Timeout(1.second)

  counter ! "inc"
  counter ! "inc"
  counter ! "inc"
  counter ! "dec"

  (counter ? "get").onComplete {
    case Success(v) =>
      println(s"Current value of the counter: $v")
    case Failure(e) =>
      println(s"Failed to receive the result from the counter: ${e.getMessage}")
  }

}
```

### Event Sourcing in ZIO

In the ZIO ecosystem, we have a community library called [ZIO Entity](https://github.com/thehonesttech/zio-entity) which aims to provide a distributed event-sourcing solution. Although it is not production-ready yet, there is nothing to worry about. We can write our toolbox or use other libraries from other functional effect ecosystems using ZIO Interop.

Note that based on your use case, you may don't need any event-sourcing framework or library. So before using any library, you should consider if you really need it or not. In many cases, this is very dependent on our domain and business requirements, and using a library may mislead you in the wrong direction.

Anyway, there is a purely functional library called [Edomata](https://edomata.ir/) which provides a simple solution for event sourcing. It is written in a tagless final style using [Cats Effect](https://typelevel.org/cats-effect/) and [Fs2](https://fs2.io/). We can use ZIO Interop to use the ZIO effect to run an Edomaton.

In the following example, we are going to use Edomata to implement a simple counter which is event sourced. To do so, we need to define Events, Commands, State, Rejections, and Transitions:

After finding out domain events, we can define them using enum or sealed traits. In this example, we have two `Increased` and `Decreased` events:

```scala
enum Event {
  case Increased, Decreased
}
```

Then we should define the commands our domain can handle:

```scala
enum Command {
  case Inc, Dec
}
```

To notify some information data to the outside world, we can use `Notification`s, let's define a simple notification data type:

```scala
case class Notification(message: String)
```

Now it's time to define the domain model and the state of the counter:

```scala

case class Counter(state: Int) {
  def inc = this.perform { Decision.accept(Event.Increased) }
  def dec = this.perform {
    if (state > 0) Decision.accept(Event.Decreased) else "decision rejected".reject
  }
}

object Counter extends DomainModel[Counter, Event, String] {
  override def initial: Counter = Counter(0)

  override def transition = {
    case Event.Increased => state => state.copy(state = state.state + 1).validNec
    case Event.Decreased => state => state.copy(state = state.state - 1).validNec
  }
}
```

We are ready to define the `CounterService`:

```scala
object CounterService extends Counter.Service[Command, Notification] {

  def apply(): PureApp[Unit] = router {
    case Command.Inc =>
      for {
        counter <- state.decide(_.inc)
        _ <- publish(Notification(s"state is going to become ${counter.state}"))
      } yield ()
    case Command.Dec => state.decide(_.dec).void
  }
}
```

So far, we have defined an edomaton called `CounterService`. To run it, we need a backend:

```scala

object BackendService {
  given BackendCodec[Event] = CirceCodec.jsonb // or .json
  given BackendCodec[Notification] = CirceCodec.jsonb
  given BackendCodec[Counter] = CirceCodec.jsonb
  given Network[Task] = Network.forAsync[Task]
  given Trace[Task] = Trace.Implicits.noop
  given Console[Task] = Console.make[Task]

  def backend =
    SkunkBackend(
      Session
        .single("localhost", 5432, "postgres", "postgres", Some("postgres"))
    )

  def buildBackend =
    backend
      .builder(CounterService, "counter")
      .withRetryConfig(retryInitialDelay = 2.seconds)
      .persistedSnapshot(200)
      .build
      .toScopedZIO

  type Service = CommandMessage[Command] => RIO[Scope, EitherNec[String, Unit]]

  def service: ZIO[Scope, Throwable, Service] =
    buildBackend
      .map(_.compile(CounterService().liftTo[Task]))
}
```

To demonstrate how the backend works, we can write a simple web service that accepts `Inc` and `Dec` commands:

```scala

object ZIOCounterHttpApp {

  def apply(service: Service) =
    Routes(
      // command: inc or dec
      // GET /{edomaton address}/{command}/{command id}
      Method.GET / string("address") / string("command") / string("commandId") ->
        handler { (address: String, command: String, commandId: String, _: Request) =>
          val cmd = command match {
            case "inc" => Command.Inc
            case "dec" => Command.Dec
          }

          service(CommandMessage(commandId, Instant.now, address, cmd))
            .map(r => Response.text(r.toString))
            .orDie
        })
}
```

Now we can wire everything and run the application:

```scala

object MainApp extends ZIOAppDefault {
  given Console[Task] = Console.make[Task]

  def run =
    ZIO.scoped {
      for {
        backendService <- BackendService.service
        _ <- ZIOCounterHttpApp(backendService).serve.provide(Server.defaultWithPort(8090))
      } yield ()
    }
}
```

To test the application, we can send the following requests:

```http
GET /FooCounter/inc/cf2209c9-6b41-44da-8c52-7e0dce109dc3
GET /FooCounter/inc/11aa920d-254e-4aa7-86f2-8002df80533b
GET /FooCounter/dec/b2e8a02c-77db-463d-8213-d462fc5a9439
GET /FooCounter/inc/9ac51e44-36d0-4daa-ac23-28e624bec174
```

If multiple commands with the same command id are sent, the backend only processes the first one and ignores the rest. If the command is rejected, the backend will not accept any subsequent commands.

To see all the events or the current state associated with the `FooCounter` adomaton, we can use the `Backend#repository` to query the database:

```scala

object ZIOStateAndHistory extends ZIOAppDefault {
  given Console[Task] = Console.make[Task]

  def run =
    ZIO.scoped {
      for {
        backendService <- BackendService.buildBackend.orDie
        history <- backendService.repository
          .history("FooCounter")
          .toZStream()
          .runCollect
        state <- backendService.repository.get("FooCounter")
        _ <- ZIO.debug(s"FooCounter History: $history")
        _ <- ZIO.debug(s"FooCounter State: $state")
      } yield ()
    }
}

// Output:
// FooCounter History: Chunk(Valid(Counter(0),0),Valid(Counter(1),1),Valid(Counter(2),2),Valid(Counter(1),3),Valid(Counter(2),4))
// FooCounter State: Valid(Counter(2),4)
```

That's it! By using functional programming instead of Akka actors, we implemented a simple event sourced counter.

## 7. Entity Sharding

Entity sharding is a technique for distributing a large number of entities across a cluster of nodes. It reduces resource contention by sharding the entities across the nodes. It also provides a way to scale out the system by adding more nodes to the system.

### Entity Sharding in Akka

Akka has a module called Akka Cluster Sharding that provides a way to distribute entities. Without further ado, in the following example, we are going to shard instances of the `Counter` entity type and then create a web service that can be used to increment or decrement each entity.

```scala

object Counter {
  sealed trait Message
  case class Increase(replyTo: ActorRef[Int]) extends Message
  case class Decrease(replyTo: ActorRef[Int]) extends Message

  def apply(entityId: String): Behavior[Message] = {
    def updated(value: Int): Behavior[Message] = {
      Behaviors.receive { (context, command) =>
        val log = context.log
        val address = context.system.address
        command match {
          case Increase(replyTo) =>
            log.info(s"executing inc msg for $entityId entity inside $address")
            val state = value + 1
            replyTo ! state
            updated(state)
          case Decrease(replyTo) =>
            log.info(s"executing dec msg for $entityId entity inside $address")
            val state = value - 1
            replyTo ! state
            updated(state)
        }
      }
    }
    updated(0)
  }
}
```

Now, it's time to create a simple web service that can be used to receive the `inc` and `dec` commands from clients:

```scala

object CounterHttpApp {
  implicit val timeout: Timeout = 1.seconds

  def routes(implicit
      system: ActorSystem[ShardingEnvelope[Counter.Message]]
  ): Route = {
    path(Segment / Segment) { case (entityId, command) =>
      get {
        val response = system.ask[Int](askReplyTo =>
          ShardingEnvelope(
            entityId,
            command match {
              case "inc" => Counter.Increase(askReplyTo)
              case "dec" => Counter.Decrease(askReplyTo)
            }
          )
        )
        onComplete(response) { value =>
          complete(value.toString)
        }
      }
    }
  }
}
```

To be able to shard instances of the `Counter` entity, let's define the guardian behavior:

```scala

object Guardian {
  def apply(): Behavior[ShardingEnvelope[Counter.Message]] =
    Behaviors.setup { context =>
      val TypeKey: EntityTypeKey[Counter.Message] =
        EntityTypeKey[Counter.Message]("counter")
      val clusterSharding = ClusterSharding(context.system)
      val shardRegion =
        clusterSharding.init(Entity(TypeKey)(c => Counter(c.entityId)))
      Behaviors.receiveMessage { msg =>
        shardRegion ! msg
        Behaviors.same
      }
    }
}
```

To be able to run multiple instances of the application, we need to define a seed node and also let the application read the port number from the environment. So, let's create the `application.conf` file in the `src/main/resources` directory:

```hocon
akka {
  actor {
    allow-java-serial ization =true 
    provider = "cluster"
  }
  remote.artery.canonical {
    hostname = "127.0.0.1" 
    port = 2551
    port=${?PORT}
  }
  cluster.seed-nodes= ["akka://system@127.0.0.1:2551"]
}

webservice {
  host = "127.0.0.1"
  port = 8082
  port = ${?HTTP_PORT}
}
```

The final step is to wire everything together to create the application:

```scala

object AkkaClusterShardingExample extends App {
  val config = ConfigFactory.load("application.conf")

  implicit val system: ActorSystem[ShardingEnvelope[Counter.Message]] =
    ActorSystem(Guardian(), "system", config)

  Http()
    .newServerAt(
      config.getString("webservice.host"),
      config.getInt("webservice.port")
    )
    .bind(CounterHttpApp.routes)
}
```

To run the application, we need to start the seed node first:

```bash
sbt -DHTTP_PORT=8081 -DPORT=2551 "runMain AkkaClusterShardingExample"
```

Then, we can start some more nodes:

```bash
sbt -DHTTP_PORT=8082 -DPORT=2552 "runMain AkkaClusterShardingExample"
sbt -DHTTP_PORT=8083 -DPORT=2553 "runMain AkkaClusterShardingExample"
```

Now, we can send some requests to any of theses nodes:

```bash
GET http://localhost:8081/foo/inc
GET http://localhost:8082/foo/inc
GET http://localhost:8083/foo/inc

GET http://localhost:8081/bar/inc
GET http://localhost:8082/bar/inc
GET http://localhost:8083/bar/inc

// ...
```

We can see that the each of `foo` and `bar` entities will be executed in a single node at a time, even if we are sending requests to different nodes.

### Entity Sharding in ZIO

In the ZIO community, there is a library called [Shardcake][73] that provides a purely functional API for entity sharding. It is highly configurable and customizable. Let's try to implement the same example as in the previous section using Shardcake.

First, we are going to define the `Counter` entity:

```scala

sealed trait CounterMessage
object CounterMessage {
  case class Increase(replier: Replier[Int]) extends CounterMessage
  case class Decrease(replier: Replier[Int]) extends CounterMessage
}

object Counter extends EntityType[CounterMessage]("counter") {

  def handleMessage(
    entityId: String,
    state: Ref[Int],
    message: CounterMessage
  ): ZIO[Sharding, Nothing, Unit] =
    podPort.flatMap { port =>
      message match {
        case CounterMessage.Increase(replier) =>
          state
            .updateAndGet(_ + 1)
            .debug(s"The $entityId counter increased inside localhost:$port pod")
            .flatMap(replier.reply)
        case CounterMessage.Decrease(replier) =>
          state
            .updateAndGet(_ - 1)
            .debug(s"The $entityId counter decreased inside localhost:$port pod")
            .flatMap(replier.reply)
      }
    }

  def behavior(
    entityId: String,
    messages: Dequeue[CounterMessage]
  ): ZIO[Sharding, Nothing, Nothing] =
    Ref.make(0).flatMap { state =>
      messages.take.flatMap(handleMessage(entityId, state, _)).forever
    }

  def podPort: UIO[String] =
    System
      .env("PORT")
      .some
      .orDieWith(_ => new Exception("Application started without any specified port!"))
}
```

To be able to receive messages from the clients, let's define a web service:

```scala

object WebService {
  def apply(counter: Messenger[CounterMessage]): Routes[Sharding, Nothing] =

    Routes(
      Method.GET / string("entityId") / "inc" -> handler { (entityId: String, _: Request) =>
        counter
          .send(entityId)(CounterMessage.Increase)
          .map(r => Response.text(r.toString))
      },
      Method.GET / string("entityId") / "dec" -> handler { (entityId: String, _: Request) =>
        counter
          .send(entityId)(CounterMessage.Decrease)
          .map(r => Response.text(r.toString))
      }
    ).sandbox
}
```

In this example, we are going to use Redis as the storage backend for the sharding. So, let's define a live layer for the sharding:

```scala

object RedisLive {
  val layer: ZLayer[Any, Throwable, Redis] =
    ZLayer.scopedEnvironment {
      implicit val runtime: zio.Runtime[Any] = zio.Runtime.default
      
      implicit val logger: Log[Task] = new Log[Task] {
        override def debug(msg: => String): Task[Unit] = ZIO.logDebug(msg)
        override def error(msg: => String): Task[Unit] = ZIO.logError(msg)
        override def info(msg: => String): Task[Unit]  = ZIO.logInfo(msg)
      }

      (for {
        client   <- RedisClient[Task].from("redis://localhost")
        commands <- Redis[Task].fromClient(client, RedisCodec.Utf8)
        pubSub   <- PubSub.mkPubSubConnection[Task, String, String](client, RedisCodec.Utf8)
      } yield ZEnvironment(commands, pubSub)).toScopedZIO
    }
}
```

We also need a configuration layer for the sharding:

```scala

object ShardConfig {
  val layer: ZLayer[Any, SecurityException, Config] =
    ZLayer(
      System
        .env("PORT")
        .map(
          _.flatMap(_.toIntOption)
            .fold(Config.default)(port => Config.default.copy(shardingPort = port))
        )
    )
}
```

Now we are ready to create our application:

```scala

object HttpApp extends ZIOAppDefault {

  def run: Task[Unit] =
    ZIO.scoped {
      for {
        port    <- System.env("HTTP_PORT").map(_.flatMap(_.toIntOption).getOrElse(8080))
        _       <- Sharding.registerEntity(Counter, Counter.behavior)
        _       <- Sharding.registerScoped
        counter <- Sharding.messenger(Counter)
        _       <- Server.serve(WebService(counter)).provideSome[Sharding](Server.defaultWithPort(port))
      } yield ()
    }.provide(
      ShardConfig.layer,
      ZLayer.succeed(GrpcConfig.default),
      ZLayer.succeed(RedisConfig.default),
      RedisLive.layer,
      StorageRedis.live,
      KryoSerialization.live,
      ShardManagerClient.liveWithSttp,
      GrpcPods.live,
      Sharding.live,
      GrpcShardingService.live
    )
}
```

To manage sharding, we should run a separate application which is called `ShardManager`:

```scala

object ShardManagerApp extends ZIOAppDefault {
  def run: Task[Nothing] =
     com.devsisters.shardcake.Server.run.provide(
      ZLayer.succeed(ManagerConfig.default),
      ZLayer.succeed(GrpcConfig.default),
      ZLayer.succeed(RedisConfig.default),
      RedisLive.layer,
      StorageRedis.live, // store data in Redis
      PodsHealth.local,  // just ping a pod to see if it's alive
      GrpcPods.live,     // use gRPC protocol
      ShardManager.live  // Shard Manager logic
    )
}
```

That's it! Now it's time to run the application. First, let's run an instance of Redis using docker:

```bash
docker run -d -p 6379:6379 --name sampleredis redis
```

Then, we can run the `ShardManager` application:

```bash
sbt "runMain ShardManagerApp"
```

Now, we can run multiple instances of `HttpApp`:

```
sbt -DHTTP_PORT=8081 -DPORT=8091 "runMain HttpApp"
sbt -DHTTP_PORT=8082 -DPORT=8092 "runMain HttpApp"
sbt -DHTTP_PORT=8083 -DPORT=8093 "runMain HttpApp"
```

Finally, we can send requests to the `HttpApp` instances:

```bash
curl http://localhost:8081/foo/inc
curl http://localhost:8082/foo/inc
curl http://localhost:8083/foo/inc

curl http://localhost:8081/bar/inc
curl http://localhost:8082/bar/inc
curl http://localhost:8083/bar/inc
```

At the same time, each entity is running only in one instance of `HttpApp`. So if we send a request for an entity to one of the instances of `HttpApp` where that entity doesn't belong, that request will be routed to the correct instance.

## 8. Distributed Computing

The most important feature of Akka is its distributed computing capabilities. It provides a set of well-established tools for building distributed systems, like Akka Cluster, Akka Distributed Data, Akka Remoting, Akka gRPC, etc.

ZIO has started a couple of projects to support distributed computing and there are also some community projects. However, some of them are still in the early stages of development. Hence, if you are heavily relying on distributed Akka technologies, you may need to make a decision with more caution.

In this section, we are going to iterate over the available options and what is the current progress:

First of all, we have a production-ready project for gRPC called [ZIO gRPC][24]. It is a ZIO wrapper around [ScalaPB][87]. It also supports streaming RPC calls using ZIO Streams.

The next fantastic project is [ZIO Schema][88]. Using ZIO Schema, you can define your data types as schemas and then generate codecs for them. It also supports distributed computing by providing a way to serialize and deserialize computations. So we can both move data and computations over the network and execute them remotely.

Again, as we [mentioned](#entity-sharding-in-zio) in this article, if you need to scale out your application using Entity Sharding, you can use [ShardCake][73]. It provides location transparency for your entities, and you can run them in a distributed manner.

ZIO has another project in development called [ZIO Flow][89]. It is a distributed workflow executor. We can think of `ZFlow` as a distributed version of `ZIO`. Using `ZFlow` we can describe a distributed workflow without worrying about the underlying concerns like transactional guarantees, fault tolerance, manual retries, etc. It is still in the early stages of development and it is not ready for production use.

[ZIO Keeper][90] is another project in development. It aims to provide solutions for the following distributed computing problems:

- Transport: A transport layer for sending and receiving messages between nodes. Currently, it supports unicast and broadcast messages.
- Membership: A membership layer for discovering nodes that are part of the same distributed system and also providing algorithms for joining and leaving the cluster. Currently, it uses SWIM and HyParView algorithms.
- Consensus: A consensus layer provides a solution for this problem: "What if the leader becomes unavailable?". Which is not developed yet.

There is also a work-in-progress implementation of the Raft protocol called [ZIO Raft][91] which is worth mentioning.

[1]: https://doc.akka.io/docs/akka/current/index.html
[2]: https://zio.dev/reference/core/zio/
[3]: https://zio.dev/reference/concurrency/
[4]: https://doc.akka.io/docs/akka/current/stream/index.html
[5]: ../../reference/stream
[6]: https://www.lagomframework.com/
[7]: https://github.com/thehonesttech/zio-entity
[8]: https://doc.akka.io/docs/akka/current/persistence.html
[9]: https://edomata.ir/
[10]: https://doc.akka.io/docs/akka/2.5.32/scheduler.html
[11]: ../../reference/schedule
[12]: https://github.com/enragedginger/akka-quartz-scheduler
[13]: https://doc.akka.io/docs/akka/current/common/circuitbreaker.html
[14]: https://www.vroste.nl/rezilience/
[15]: https://doc.akka.io/docs/akka/current/logging.html
[16]: ../../guides/tutorials/enable-logging-in-a-zio-application.md
[17]: https://doc.akka.io/docs/akka/current/typed/testing.html
[18]: ../../reference/test
[19]: https://doc.akka.io/docs/akka/current/stream/stream-testkit.html
[20]: https://doc.akka.io/docs/akka/current/cluster-metrics.html
[21]: ../../reference/observability/metrics
[22]: https://doc.akka.io/docs/akka/current/general/supervision.html
[23]: https://doc.akka.io/docs/akka-grpc/current/index.html
[24]: ../../ecosystem/community/zio-grpc.md
[25]: https://github.com/sangria-graphql/sangria-akka-http-example
[26]: ../../ecosystem/community/caliban.md
[27]: https://doc.akka.io/docs/alpakka-kafka/current/home.html
[28]: ../../zio-kafka/index.md
[29]: https://doc.akka.io/docs/alpakka/current/s3.html
[30]: ../../zio-aws/index.md
[31]: https://doc.akka.io/docs/alpakka/current/sns.html
[32]: https://github.com/zio/zio-s3
[33]: https://doc.akka.io/docs/alpakka/current/sqs.html
[34]: ../../zio-sqs/index.md
[35]: https://doc.akka.io/docs/alpakka/current/amqp.html
[36]: ../../ecosystem/community/zio-amqp.md
[37]: https://doc.akka.io/docs/alpakka/current/kinesis.html
[38]: ../../ecosystem/community/zio-kinesis.md
[39]: https://doc.akka.io/docs/alpakka/current/dynamodb.html
[40]: https://github.com/zio/zio-dynamodb
[41]: https://doc.akka.io/docs/alpakka/current/external/pulsar.html
[42]: ../../ecosystem/community/zio-pulsar.md
[43]: https://doc.akka.io/docs/alpakka/current/awslambda.html
[44]: https://github.com/zio/zio-lambda
[45]: https://doc.akka.io/docs/alpakka/current/cassandra.html
[46]: https://github.com/palanga/zio-cassandra
[47]: https://doc.akka.io/docs/alpakka/current/elasticsearch.html 
[48]: https://github.com/aparo/zio-elasticsearch
[49]: https://doc.akka.io/docs/alpakka/current/ftp.html
[50]: ../../zio-ftp/index.md
[51]: https://github.com/mbannour/zio-mongodb
[52]: ../../zio-redis/index.md
[53]: https://doc.akka.io/docs/alpakka/current/avroparquet.html
[54]: ../../zio-schema/index.md
[55]: https://doc.akka.io/docs/akka-http/current/index.html
[56]: https://github.com/zio/zio-http
[57]: ../../zio-nio/index.md
[58]: https://doc.akka.io/docs/alpakka/current/slick.html
[59]: ../../ecosystem/community/zio-slick-interop.md
[60]: https://doc.akka.io/docs/alpakka/current/external/tcp.html
[61]: https://github.com/searler/zio-tcp
[62]: https://doc.akka.io/docs/alpakka/current/google-cloud-pub-sub.html
[63]: https://github.com/zio/zio-gcp 
[64]: https://doc.akka.io/docs/alpakka/current/google-cloud-storage.html
[65]: https://doc.akka.io/docs/alpakka/current/data-transformations/json.html
[66]: ../../zio-json/index.md
[67]: https://doc.akka.io/docs/alpakka/current/orientdb.html
[68]: ../../zio-quill/index.md
[69]: https://doc.akka.io/docs/akka/current/typed/logging.html
[70]: ../../zio-logging/index.md
[71]: https://doc.akka.io/docs/akka-http/current/common/caching.html
[72]: ../../zio-cache/index.md
[73]: https://devsisters.github.io/shardcake/ 
[74]: https://doc.akka.io/docs/akka/current/typed/cluster-sharding.html
[75]: ../../reference/state-management/global-shared-state.md
[76]: ../../reference/state-management/fiberref.md
[77]: ../../reference/state-management/zstate.md
[78]: https://doc.akka.io/docs/akka/current/typed/mailboxes.html
[79]: ../../reference/concurrency/queue.md
[80]: ../../reference/fiber/fiber.md
[81]: ../../reference/sync/index.md
[82]: ../../reference/concurrency/ref.md
[83]: ../../reference/concurrency/refsynchronized.md
[84]: ../../reference/concurrency/promise.md
[85]: ../../reference/concurrency/hub.md
[86]: ../../reference/concurrency/semaphore.md
[87]: https://scalapb.github.io/ 
[88]: ../../zio-schema/index.md
[89]: https://github.com/zio/zio-flow 
[90]: https://zio.github.io/zio-keeper/
[91]: https://github.com/ariskk/zio-raft
[92]: ../../zio-logging/index.md
[93]: ../../zio-metrics/index.md

---

## How to Migrate from Cats Effect to ZIO?

Cats `IO[A]` can be easily replaced with ZIO's `Task[A]` (an alias for `ZIO[Any, Throwable, A]`).
Translation should be relatively straightforward. Below, you'll find tables showing the ZIO equivalents of
various `cats.*`'s methods.

### Methods on cats.FlatMap.Ops

| cats           | ZIO        |
|----------------|------------|
| `flatMap`      | `flatMap`  |
| `flatten`      | `flatten`  |
| `productREval` | `zipRight` |
| `productLEval` | `zipLeft`  |
| `mproduct`     | `zipPar`   |
| `flatTap`      | `tap`      |

---

## How to Migrate from Monix to ZIO?

Monix's `Task[A]` can be easily replaced with ZIO's `Task[A]` (an alias for `ZIO[Any, Throwable, A]`).
Translation should be relatively straightforward. Below, you'll find tables showing the ZIO equivalents of 
 `monix.eval.Task`'s methods. 

Once you've completed the initial translation, you'll find that ZIO is outfitted with many additional
methods which have no Monix equivalents, so have fun exploring the API and see if you can rewrite some
of your logic at a higher level of abstraction, with more powerful combinators and fewer lines code.

If you are using operators from Cats Effect extension methods see also 
[here](from-cats-effect.md).

### Methods on Trait

| Monix                | ZIO                          |
|----------------------|------------------------------|
| `attempt`            | `either`                     |
| `bracketCase`        | `acquireReleaseExitWith`     |
| `bracketE`           | `acquireReleaseExitWith`     |
| `bracket`            | `acquireReleaseWith`         |
| `delayExecution`     | `delay`                      |
| `dematerialize`      | `absolve`                    |
| `doOnCancel`         | `onInterrupt`                |
| `doOnFinish`         | `onExit`                     |
| `failed`             | `flip`                       |
| `flatMap`            | `flatMap`                    |
| `flatten`            | `flatten`                    |
| `guaranteeCase`      | `ensuringExit`               |
| `guarantee`          | `ensuring`                   |
| `loopForever`        | `forever`                    |
| `materialize`        | `either`                     |
| `memoize`            | `memoize`                    |
| `onErrorFallbackTo`  | `orElse`                     |
| `onErrorHandleWith`  | `catchAll`                   |
| `onErrorRecoverWith` | `catchSome`                  |
| `onErrorRestart`     | `retryN`                     |
| `redeemWith`         | `foldZIO`                    |
| `redeem`             | `fold`                       |
| `restartUntil`       | `repeatUntil`                |
| `start`              | `fork`                       |
| `timed`              | `timed`                      |
| `timeout`            | `timeout`                    |
| `uncancelable`       | `uninterruptible`            |

### Methods on Companion Object

| Monix          | ZIO              |
|----------------|------------------|
| `apply`        | `apply`          |
| `asyncF`       | `asyncZIO`       |
| `async`        | `async`          |
| `cancelable`   | `asyncInterrupt` |
| `deferFuture`  | `fromFuture`     |
| `defer`        | `suspend`        |
| `delay`        | `attempt`        |
| `eval`         | `attempt`        |
| `fromEither`   | `fromEither`     |
| `fromFuture`   | `fromFuture`     |
| `fromTry`      | `fromTry`        |
| `map2`         | `mapN`           |
| `mapBoth`      | `mapParN`        |
| `never`        | `never`          |
| `now`          | `succeed`        |
| `parMap2`      | `mapParN`        |
| `parSequenceN` | `collectAllParN` |
| `parSequence`  | `collectAllPar`  |
| `parTraverseN` | `foreachParN`    |
| `parTraverse`  | `foreachPar`     |
| `parZip2`      | `tupledPar`      |
| `pure`         | `succeed`        |
| `racePair`     | `raceWith`       |
| `race`         | `raceFirst`      |
| `raiseError`   | `fail`           |
| `sequence`     | `collectAll`     |
| `shift`        | `yield`          |
| `sleep`        | `sleep`          |
| `suspend`      | `suspend`        |
| `traverse`     | `foreach`        |
| `unit`         | `unit`           |

### Data Structures

| Monix / Cats Effect | ZIO         |
|---------------------|-------------|
| `Deferred`          | `Promise`   |
| `Fiber`             | `Fiber`     |
| `MVar`              | `Queue`     |
| `Ref`               | `Ref`       |
| `Semaphore`         | `Semaphore` |
| `TaskApp`           | `App`       |
| `TaskLocal`         | `FiberRef`  |
| `Task`              | `Task`      |

---

## ZIO 2.x Migration Guide

In this guide we want to introduce the migration process to ZIO 2.x. So if you have a project written in ZIO 1.x and want to migrate that to ZIO 2.x, this article is for you. 

## Automatic Migration

Before you migrate your own codebase, confirm that all of your ZIO-related dependencies have been migrated to ZIO 2.x with our [ZIO Ecosystem Tool](https://zio-ecosystem.herokuapp.com/).

ZIO uses the [Scalafix](https://scalacenter.github.io/scalafix/) for automatic migration. Scalafix is a code migration tool that takes a rewrite rule and reads the source code, converting deprecated features to newer ones, and then writing the result back to the source code. 

ZIO has a migration rule named `Zio2Upgrade` which migrates a ZIO 1.x code base to ZIO 2.x. This migration rule covers most of the changes. Therefore, to migrate a ZIO project to 2.x, we prefer to apply the `Zio2Upgrade` rule to the existing code. After that, we can go to the source code and fix the remaining compilation issues:

1. First, we should ensure that all of our direct and transitive dependencies [have released their compatible versions with ZIO 2.x](https://zio-ecosystem.herokuapp.com/). Note that we shouldn't update our dependencies to the 2.x compatible versions, before running scalafix.

2. Next, we need to install the [Scalafix SBT Plugin](https://github.com/scalacenter/sbt-scalafix), by adding the following line into `project/plugins.sbt` file:
    ```scala
    // project/plugins.sbt
    addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "<version>")
    ```

3. We are ready to apply the migration rule:
    ```bash
    sbt "scalafixEnable; scalafixAll github:zio/zio/Zio2Upgrade?sha=series/2.x" 
    ```

4. After running scalafix, it's time to upgrade ZIO dependencies. If we are using one of the following dependencies, we need to bump them into the `2.x` version:

    ```scala
    libraryDependencies += "dev.zio" %% "zio"         % "2.0.0"
    libraryDependencies += "dev.zio" %% "zio-streams" % "2.0.0"
    libraryDependencies += "dev.zio" %% "zio-test"    % "2.0.0"
    ```

   Other than ZIO, we should upgrade all other (official or community) ZIO libraries we are using in our `build.sbt` file.

5. Now, we have performed most of the migration. Finally, we should fix the remaining compilation errors with the help of the remaining sections in this article.

## Guidelines for Library Authors

As a contributor to ZIO ecosystem libraries, we also should cover these guidelines:

1. We should add _implicit trace parameter_ to all our codebase, this prevents the guts of our library from messing up the user's execution trace. 

Let's see an example of that in the ZIO source code:

```diff
trait ZIO[-R, +E, +A] {
-  def map[B](f: A => B): ZIO[R, E, B] =
     flatMap(a => ZIO.succeed(f(a)))
+  def map[B](f: A => B)(implicit trace: Trace): ZIO[R, E, B] = 
     flatMap(a => ZIO.succeed(f(a)))
}
```

Assume we have written the `FooLibrary` as below:

```scala

object FooLibrary {
  def foo = bar.flatMap(x => ZIO.succeed(x * 2))  // line 4
  private def bar = baz.flatMap(x => ZIO.succeed(x * x))  // line 5
  private def baz = ZIO.fail("Oh uh!").as(5)              // line 6
}
```

Without _implicit trace parameter_, the user of our library will get so many unrelated stack trace messages:

```scala

object MainApp extends ZIOAppDefault {
  def run = FooLibrary.foo
}
// timestamp=2022-04-05T11:37:59.336623325Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
//	at <empty>.FooLibrary.baz(MainApp.scala:6)
//	at <empty>.FooLibrary.bar(MainApp.scala:5)
//	at <empty>.FooLibrary.foo(MainApp.scala:4)"
```

To avoid messing up our user's execution trace, we should add implicit trace parameters to our methods:

```scala

object FooLibrary {
  def foo(implicit trace: Trace) = bar.flatMap(x => ZIO.succeed(x * 2))
  private def bar(implicit trace: Trace) = baz.flatMap(x => ZIO.succeed(x * x))
  private def baz(implicit trace: Trace) = ZIO.fail("Oh uh!").as(5)
}

object MainApp extends ZIOAppDefault {
  def run = FooLibrary.foo // line 10
}
//timestamp=2022-04-05T11:47:59.773409363Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-2" java.lang.String: Oh uh!
//	at <empty>.MainApp.run(MainApp.scala:10)"
```

2. All parameters to operators returning an effect [should be by-name](#lazy-evaluation-of-parameters). Also, we should be sure to capture any parameters that are referenced more than once as values suspended in a ZIO constructor such as `suspendSucceed` to prevent _double evaluation_. 
    
    The overall pattern in implementing such methods will be:

    ```diff
    - def foreachParN[A](n: Int)(a: Iterable[A]) = {
        ... // The function body
    - }
    + def foreachParN[A](n0: => Int)(a0: => Iterable[A]) = 
    +   ZIO.suspendSucceed {
    +     val n = n0 
    +     val a = a0
          ... // The function body
    +   }
    ```
   
    As a result, the code will be robust to _double evaluation_ as well as to _side-effects embedded within parameters_.

3. We should update names to match [ZIO 2.0 naming conventions](#zio-20-naming-conventions).

4. ZIO 2.0 introduced [new structured concurrently operators](#compositional-concurrency) which helps us to change the regional parallelism settings of our application. So if applicable, we should use these operators instead of the old parallel operators.

5. If we are exposing [unsafe operators](#unsafe-marker) in one of our interfaces we should use the `Unsafe` data type to indicate this. By convention we define these operators in an `UnsafeAPI` trait in our interface that can be accessed using as `unsafe` operator.

```diff
trait MyInterface {
-  def unsafeDoSomething(): Unit
+  def unsafe: UnsafeAPI
+
+  trait UnsafeAPI {
+    def doSomething()(implicit unsafe: Unsafe): Unit
+  }
}
```

## Deletion of Type Alias Companion Objects

In ZIO 1.x, using the type aliases as objects created another way to do things and potentially led to confusion about whether these were the same or somehow different with little benefit.

In ZIO 2.x, we removed companion objects for type aliases. We still can use type aliases such as `UIO[Int]`, but we couldn't do `UIO.succeed(1)` anymore:

```diff
- val effect: UIO[Int] = UIO.succeed(1)
+ val effect: UIO[Int] = Exit.succeed(1)

// another examp:
- val stream: UStream[Int] = UStream.succeed(1)
+ val stream: UStream[Int] = ZStream.succeed(1)
```

The [migration script](#automatic-migration) will automatically convert all the usages of type aliases to the corresponding objects.

## Deletion of Has Data Type

The Has data type, which was used for combining services, was removed. Therefore, we no longer need to wrap services in the `Has` data type.

For example, in ZIO 1.x, the following layer denotes this layer requires `Logging`, `Random`, `Database` and produce the `UserRepo`:

```scala
val userRepo: ZLayer[Has[Logging] with Has[Random] with Has[Database], Throwable, Has[UserRepo]] = ???
```

In ZIO 2.x, the `Has` has been removed and simplified for better ergonomics:

```scala
val userRepo: ZLayer[Logging with Random with Database, Throwable, UserRepo] = ???
```

Also in ZIO 2.x instead of the `Has` data type, a type-level map called `ZEnvironment` has been built into ZIO. Let's see how this changes the way we can provide a service to the environment.

Using the following code snippet, we demonstrate how we used to access and provide instances of `Config` service to the application environment using ZIO 1.x:

```scala
// ZIO 1.x

case class Config(url: String, port: Int)

object ConfigExample extends zio.App {

  val app: ZIO[Has[console.Console.Service] with Has[Config], Nothing, Unit] = for {
    config <- ZIO.service[Config]
    _      <- console.putStrLn(s"application config: $config").orDie
  } yield ()
  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] = {

    app.provideSome[Has[console.Console.Service]](_ ++ Has(Config("localhost", 8080))).exitCode
  }
}
```

To migrate this snippet to ZIO 2.x, we need to remove all the `Has` service wrappers, and finally, we will use the `ZIO#provideSomeEnvironment` method to append the `Config` instance to the `ZEnvironment`:

```scala
// ZIO 2.x

case class Config(url: String, port: Int)

object ConfigExample extends ZIOAppDefault {
  val app: ZIO[Config, Nothing, Unit] = for {
    config <- ZIO.service[Config]
    _      <- Console.printLine(s"application config: $config").orDie
  } yield ()
    
  def run = 
    app.provideEnvironment(ZEnvironment(Config("localhost", 8080)))
}
```

Note that in ZIO 2.x, default services (e.g `Console`) are eliminated from the environment.

## ZIO

### Removed Methods

**Arrow Combinators** — (`+++`, `|||`, `onSecond`, `onFirst`, `second`, `first`, `onRight`, `onLeft`, `andThen`, `>>>`, `compose`, `<<<`, `identity`, `swap`, `join`)

In ZIO 2.0, all arrow combinators are removed, and we need to use alternatives like doing monadic for-comprehension style `flatMap` with combinators like `provide`, `zip`, and so on.

### ZIO 2.0 Naming Conventions

In ZIO 2.0, the name of constructors and operators becomes more ergonomic and simple. They reflect more about their purpose rather than just using idiomatic jargon of category theory or functional terms in functional programming with Haskell.

Here are some of the most important changes:

- **Multiple ways of doing the same thing are removed** — For example:
    - Both `ZIO.succeed` and `ZIO.effectTotal` do the same thing. So in ZIO 2.0 we just have one version of these constructors which is `ZIO.succeed`.
    - The bind operator `>>=` is removed. So we just have one way to flatMap which is the `flatMap` method. Therefore, the `>>=` method doesn't surprise the non-Haskellers.
    - The `ZIO#get` method was essentially a more constrained version of `ZIO#some`. So the `get` method is deprecated.

- **`ZIO.attempt` instead of `ZIO.effect`** — In ZIO 2.0 all ZIO constructors like `ZIO.effect*` that create a ZIO from a side effect are deprecated and renamed to the `ZIO.attempt*` version. For example, when we are reading from a file, it's more meaningful to say we are attempting to read from a file instead of saying we have an effect of reading from a file.

- **`ZIO` instead of the `M` suffix** — In effectful operations, the `M` suffix is renamed to the `ZIO` suffix. In ZIO 1.x, the `M` suffix in an effectful operation means that the operation works with monad in a monadic context. This naming convention is the legacy of Haskell jargon. In ZIO 2.x, all these suffixes are renamed to `ZIO`. For example, the `ifM` operator is renamed to `ifZIO`.

- **`Discard` instead of the underscore `_` suffix** — The underscore suffix is another legacy naming convention from Haskell's world. In ZIO 1.x, the underscore suffix means we are going to discard the result. The underscore version works exactly like the one without the underscore, but it discards the result and returns `Unit` in the ZIO context. For example, the `collectAll_` operator renamed to `collectAllDiscard`.

- **`as`, `to`, `into` prefixes** — The `ZIO#to` is renamed to the `ZIO#intoPromise`. So now we have three categories of conversion:
    1. **as** — The `ZIO#as` method and its variants like `ZIO#asSome`, `ZIO#asSomeError` and `ZIO#asService` are used when transforming the `A` inside of a `ZIO`, generally as shortcuts for `map(aToFoo(_))`.
    2. **to** — The `ZIO#to` method and its variants like `ZIO#toFuture` are used when the `ZIO` is transformed into something else other than the `ZIO` data-type.
    3. **into** — All `into*` methods, accept secondary data-type, modify it with the result of the current effect (e.g. `ZIO#intoPromise`, `ZStream#intoHub`, and `ZStream#intoQueue`)

| ZIO 1.x                        | ZIO 2.x                           |
|--------------------------------|-----------------------------------|
| `ZIO#>>=`                      | `ZIO#flatMap`                     |
| `ZIO#bimap`                    | `ZIO#mapBoth`                     |
| `ZIO#mapEffect`                | `ZIO#mapAttempt`                  |
| `ZIO#filterOrElse_`            | `ZIO#filterOrElse`                |
| `ZIO#foldCauseM`               | `ZIO#foldCauseZIO`                |
| `ZIO#foldM`                    | `ZIO#foldZIO`                     |
| `ZIO#foldTraceM`               | `ZIO#foldTraceZIO`                |
|                                |                                   |
| `ZIO#get`                      | `ZIO#some`                        |
| `ZIO#optional`                 | `ZIO#unsome`                    |
| `ZIO#someOrElseM`              | `ZIO#someOrElseZIO`               |
|                                |                                   |
| `ZIO.forkAll_`                 | `ZIO.forkAllDiscard`              |
| `ZIO#forkInternal`             | `ZIO#fork`                        |
| `ZIO#forkOn`                   | `ZIO#onExecutionContext(ec).fork` |
| `ZIO.fromFiberM`               | `ZIO.fromFiberZIO`                |
| `ZIO.require`                  | `ZIO.someOrFail`                  |
| `ZIO#on`                       | `ZIO#onExecutionContext`          |
| `ZIO#rejectM`                  | `ZIO#rejectZIO`                   |
| `ZIO#run`                      | `ZIO#exit`                        |
| `ZIO#timeoutHalt`              | `ZIO#timeoutFailCause`            |
|                                |                                   |
| `ZIO#to`                       | `ZIO#intoPromise`                 |
|                                |                                   |
| `ZIO.access`                   | `ZIO.environmentWith`             |
| `ZIO.accessM`                  | `ZIO.environmentWithZIO`          |
| `ZIO.fromFunction`             | `ZIO.environmentWith`             |
| `ZIO.fromFunctionM`            | `ZIO.environmentWithZIO`          |
| `ZIO.services`                 | `ZIO.service`                     |
|                                |                                   |
| `ZIO.bracket`                  | `ZIO.acquireReleaseWith`          |
| `ZIO.bracketExit`              | `ZIO.acquireReleaseExitWith`      |
| `ZIO.bracketAuto`              | `ZIO.acquireReleaseWithAuto`      |
| `ZIO#bracket`                  | `ZIO#acquireReleaseWith`          |
| `ZIO#bracket_`                 | `ZIO#acquireRelease`              |
| `ZIO#bracketExit`              | `ZIO#acquireReleaseExitWith`      |
| `ZIO#bracketExit`              | `ZIO#acquireReleaseExitWith`      |
| `ZIO#bracketOnError`           | `ZIO#acquireReleaseOnErrorWith`   |
|                                |                                   |
| `ZIO.collectAll_`              | `ZIO.collectAllDiscard`           |
| `ZIO.collectAllPar_`           | `ZIO.collectAllParDiscard`        |
| `ZIO.collectAllParN_`          | `ZIO.collectAllParNDiscard`       |
| `ZIO#collectM`                 | `ZIO#collectZIO`                  |
|                                |                                   |
| `ZIO.effect`                   | `ZIO.attempt`                     |
| `ZIO.effectAsync`              | `ZIO.async`                       |
| `ZIO.effectAsyncInterrupt`     | `ZIO.asyncInterrupt`              |
| `ZIO.effectAsyncM`             | `ZIO.asyncZIO`                    |
| `ZIO.effectAsyncMaybe`         | `ZIO.asyncMaybe`                  |
| `ZIO.effectBlocking`           | `ZIO.attemptBlocking`             |
| `ZIO.effectBlockingCancelable` | `ZIO.attemptBlockingCancelable`   |
| `ZIO.effectBlockingIO`         | `ZIO.attemptBlockingIO`           |
| `ZIO.effectBlockingInterrupt`  | `ZIO.attemptBlockingInterrupt`    |
| `ZIO.effectSuspend`            | `ZIO.suspend`                     |
| `ZIO.effectSuspendTotal`       | `ZIO.suspendSucceed`              |
| `ZIO.effectTotal`              | `ZIO.succeed`                     |
|                                |                                   |
| `ZIO.foreach_`                 | `ZIO.foreachDiscard`              |
| `ZIO.foreachPar_`              | `ZIO.foreachParDiscard`           |
| `ZIO.foreachParN_`             | `ZIO.foreachParNDiscard`          |
| `ZIO#replicateM`               | `ZIO#replicateZIO`                |
| `ZIO#replicateM_`              | `ZIO#replicateZIODiscard`         |
|                                |                                   |
| `ZIO.halt`                     | `ZIO.failCause`                   |
| `ZIO.haltWith`                 | `ZIO.failCauseWith`               |
|                                |                                   |
| `ZIO.ifM`                      | `ZIO.ifZIO`                       |
| `ZIO.loop_`                    | `ZIO.loopDiscard`                 |
| `ZIO.whenCaseM`                | `ZIO.whenCaseZIO`                 |
| `ZIO.whenM`                    | `ZIO.whenZIO`                     |
| `ZIO.unlessM`                  | `ZIO.unlessZIO`                   |
| `ZIO#unlessM`                  | `ZIO#unlessZIO`                   |
| `ZIO#whenM`                    | `ZIO#whenZIO`                     |
| `ZIO#repeatUntilM`             | `ZIO#repeatUntilZIO`              |
| `ZIO#repeatWhileM`             | `ZIO#repeatWhileZIO`              |
| `ZIO#retryUntilM`              | `ZIO#retryUntilZIO`               |
| `ZIO#retryWhileM`              | `ZIO#retryWhileZIO`               |
| `ZIO.replicateM`               | `ZIO.replicateZIO`                |
| `ZIO.replicateM_`              | `ZIO.replicateZIODiscard`         |
|                                |                                   |
| `ZIO.validate_`                | `ZIO.validateDiscard`             |
| `ZIO.validatePar_`             | `ZIO.validateParDiscard`          |
|                                |                                   |
| `ZIO.tapCause`                 | `ZIO.tapErrorCause`               |

### Lazy Evaluation of Parameters

In ZIO 2.x, we changed the signature of those functions that return effects to use _by-name parameters_. And we also encourage library authors to do the same for any functions that return effects.

Our motivation for this change was a common mistake among new users of ZIO, which they _accidentally embed raw effects_ inside the function they pass to ZIO constructors and operators. This mistake may produce some unwanted behaviors.

Let's see an example of this anti-pattern in ZIO 1.x:

```scala
ZIO.bracket({
  val random = scala.util.Random.nextInt()
  ZIO.succeed(random)
})(_ => ZIO.unit)(x => console.putStrLn(x.toString)).repeatN(2)
```

The newbie user expects that this program prints 3 different random numbers, while the output would be something as follows:

```
1085597917
1085597917
1085597917
```

This is because the user incorrectly introduced a raw effect into the `acquire` parameter of `bracket` operation. As the `acquire` is _by-value parameter_, the value passed to the function evaluated _eagerly_, only once:

```scala
def bracket[R, E, A](acquire: ZIO[R, E, A]): ZIO.BracketAcquire[R, E, A]
```

If we make the `acquire` to _by-name parameter_, we can prevent these mistakes:

```diff
- def bracket[R, E, A](acquire: ZIO[R, E, A]): ZIO.BracketAcquire[R, E, A]
+ def bracket[R, E, A](acquire: => ZIO[R, E, A]): ZIO.BracketAcquire[R, E, A]
```

So, in ZIO 2.x if we accidentally introduce an effect to the ZIO parameters, the lazy parameter prevents the program from producing undesired behaviors:

```scala
// Note that in ZIO 2.x, the `bracket` is deprecated and renamed to the `acquireReleaseWith`. In this example to prevent the consistency of our example, we used the `bracket`.

ZIO.bracket({
  val random = scala.util.Random.nextInt()
  ZIO.succeed(random)
})(_ => ZIO.unit)(x => console.putStrLn(x.toString)).repeatN(2)
```

The output would be something like this:

```scala
355191016
2046799548
333146616
```

### Composable Zips

In ZIO 2.x, when we are zipping together different effects:
- `Tuple`s are not nested.
- `Unit`s do not contribute to the output.

Assume we have these effects:

```scala
val x1: UIO[Int]     = ZIO.succeed(???)
val x2: UIO[Unit]    = ZIO.succeed(???)
val x3: UIO[String]  = ZIO.succeed(???)
val x4: UIO[Boolean] = ZIO.succeed(???)
```

In ZIO 1.x, the output of zipping together these effects are nested:

```scala
val zipped = x1 <*> x2 <*> x3 <*> x4
```

While in ZIO 2.x, we have more ergonomics result type and also the `Unit` data-type doesn't contribute to the output:

```scala
val zipped = x1 <*> x2 <*> x3 <*> x4
```

This change is not only for the `ZIO` data type but also for all other data types like `ZStream`, `ZSTM`, etc.

As we have compositional zips, we no longer need higher arity zips in ZIO 1.x like `mapN`, `mapParN`, `Gen#zipN`, and `Gen#crossN`. They are deprecated in ZIO 2.x.

Here is the list of `zip` variants that are deprecated:

| ZIO 1.x         | ZIO 2.x      |
|-----------------|--------------|
| `ZIO#&&&`       | `ZIO#zip`    |
| `ZIO.tupled`    | `ZIO.zip`    |
| `ZIO.tupledPar` | `ZIO.zipPar` |
| `ZIO.mapN`      | `ZIO.zip`    |
| `ZIO.mapParN`   | `ZIO.zipPar` |

### Compositional Concurrency

We introduced two operations that modify the parallel factor of a concurrent ZIO effect, `ZIO#withParallelism` and `ZIO#withParallelismUnbounded`. This makes the maximum number of fibers for parallel operators as a regional setting. Therefore, all parallelism operators ending in `N`, such as `foreachParN` and `collectAllParN`, have been deprecated:

| ZIO 1.x                   | ZIO 2.x                  |
|---------------------------|--------------------------|
| `foreachParN`             | `foreachPar`             |
| `foreachParN_`            | `foreachParDiscard`      |
| `collectAllParN`          | `collectAllPar`          |
| `collectAllParN_`         | `collectAllParDiscard`   |
| `collectAllWithParN`      | `collectAllWithPar`      |
| `collectAllSuccessesParN` | `collectAllSuccessesPar` |

Having separate methods for changing the parallelism factor of a parallel effect deprecates lots of extra operators and makes concurrency more compositional.

So instead of writing a parallel task like this:

```scala
ZIO.foreachParN(8)(urls)(download)
```

We should use the `withParallelism` method:

```scala
ZIO.foreachPar(urls)(download).withParallelism(8)
```

The `withParallelismUnbounded` method is useful when we want to run a parallel effect with an unbounded maximum number of fibers:

```scala
ZIO.foreachPar(urls)(download).withParallelismUnbounded
```

### Either Values

In ZIO 1.x, the `ZIO#left` and `ZIO#right` operators are lossy, and they don't preserve the information on the other side of `Either` after the transformation.

For example, assume we have an effect of type `ZIO[Any, Throwable, Left[Int, String]]`:

```scala
val effect         = Task.effect(Left[Int, String](5))
// effect: ZIO[Any, Throwable, Left[Int, String]]
val leftProjection = effect.left
// leftProjection: ZIO[Any, Option[Throwable], Int]
```

The error channel of `leftProjection` doesn't contain type information of the other side of the `Left[Int, String]`, which is `String`. So after projecting to the left, we can not go back to the original effect.

In ZIO 2.x, the `ZIO#left` and `ZIO#right`, contains all type information so then we can `unleft` or `unright` to inverse that projection:

```scala
val effect         = ZIO.attempt(Left[Int, String](5))
val leftProjection = effect.left
val unlefted       = leftProjection.map(_ * 2).unleft
```

So the error channel of the output of `left` and `right` operators is changed from `Option` to `Either`.

### Descriptive Errors

ZIO's type system uses implicit evidence to ensure type safety, and some level of correctness at compile time. In ZIO 2.x, the _subtype evidence_, `<:<` replaced by these two descriptive implicit evidences:

1. **`IsSubtypeOfOutput`** — The `O1 IsSubtypeOfOutput O2` ensures that the output type `O1` is subtype of `O2`

2. **`IsSubtypeOfError`** — The `E1 IsSubtypeOfError E2` ensures that the error type `E1` is a subtype of `E2`

Now we have more descriptive errors at compile time in the vast majority of operators.

Let's just see an example of each one. In ZIO 1.x, the compiler print obscurant error messages:

```scala
ZIO.fail("Boom!").orDie
// error: Cannot prove that String <:< Throwable.
// ZIO.fail("Boom!").orDie
// ^^^^^^^^^^^^^^^^^^^^^^^

ZIO.succeed(Set(3,4)).head
// error: Cannot prove that scala.collection.immutable.Set[Int] <:< List[B].
// ZIO.succeed(Set(3, 4)).head
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Now in ZIO 2.x we have such informative error messages:

```scala
ZIO.fail("Boom!").orDie
// error: This operator requires that the error type be a subtype of Throwable but the actual type was String.
// ZIO.fail("Boom!").orDie
// ^^^^^^^^^^^^^^^^^^^^^^^

ZIO.succeed(Set(3, 4, 3)).head
// error: This operator requires that the output type be a subtype of List[B] but the actual type was scala.collection.immutable.Set[Int].
// ZIO.succeed(Set(3, 4, 3)).head
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

## Elimination of Default Services From The ZIO Environment

In ZIO 1.x we used default ZIO services such as `Clock`, `Console`, `Random`, and `System` along with the service pattern. So each time we used one of these services by obtaining them from the environment, the requirement of our effect becomes bigger and bigger. Finally, at the end of the world, we had two options, one was to use the default implementation of these services, and the other one was to use our own implementations.

For example, in ZIO 1.x, we have the following boilerplate code to print random numbers every second. The environment type of the `myApp` effect is `Console with Clock with Random`:

```scala

object MainApp extends App {
  val myApp: ZIO[Clock with Console with Random, IOException, Unit] =
    for {
      rnd <- random.nextIntBounded(100)
      _   <- console.putStrLn(s"Random number: $rnd")
      _   <- clock.sleep(1.second)
    } yield ()
  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.forever.exitCode  
    // or we can provide our own implementation 
    // myApp.forever.provideLayer(Console.live ++ Clock.live ++ Random.live).exitCode
}
```

But these services did not fit well with the _service pattern_ because they were too low level and users frequently used them directly in their code to use the default implementation out of the box. So in most cases, users were not meant to provide their own implementation of these services. In another hand, as they are low level, they are used very often, so they pollute the environment type of the effect. They are too small to be used with the _service pattern_.

To improve on this, in ZIO 2.x, we deleted default services from the environment, instead, we built these services into the ZIO Runtime. So these services can still be modified and testable. In ZIO 2.x we encourage using the environment for higher-level services.

Therefore, the previous example In ZIO 2.x can be rewritten very simply as below:

```scala

object MainApp extends App {
  val myApp: ZIO[Any, IOException, Unit] =
    for {
      rnd <- Random.nextIntBounded(100)
      _   <- Console.printLine(s"Random number: $rnd")
      _   <- Clock.sleep(1.second)
    } yield ()

  def run = myApp.forever
}
```

In nutshell, to migrate from ZIO 1.x to ZIO 2.x, we need follow these steps:

1. We aren't required to obtain default services from the environment using functions like `ZIO.service[Console]`, instead we should obtain the `Console` service using `ZIO.console`. So there is no need to access these services from the environment anymore, they are built into the ZIO Runtime. If we want to access them, we can use these functions instead:
- `ZIO.console`/`ZIO.consoleWith`
- `ZIO.clock`/`ZIO.clockWith`
- `ZIO.random`/`ZIO.randomWith`
- `ZIO.system`/`ZIO.systemWith`

```diff
for {
-  random <- ZIO.service[Random]
+  random <- ZIO.random
} yield ()
```

2. By removing these services from the environment, all usage of `ZEnv`, `Console`, `Clock`, `Random`, or `System` in the environment type of `ZIO`, `ZStream` and `ZLayer` should be generally deleted:

```diff
- val myApp: ZIO[Clock with Console with Random with UserRepo with Logging, IOException, Unit] = ???
+ val myApp: ZIO[UserRepo with Logging, IOException, Unit] = ???
```

3. If we want to use the live version in tests we can use these test aspects instead of providing them as layers:
   `withLiveClock`
   `withLiveConsole`
   `withLiveRandom`
   `withLiveSystem`
   `withLiveEnvironment`

For example:

```diff
- testM("TestLiveClock") { ... }.provideLayer(Clock.live)
+ test("TestLiveClock") { ... } @@ withLiveClock
```

4. In ZIO 1.x, whenever we wanted to provide our own versions of ZIO default services, we could do that using one of the `ZIO#provide*` operators. In ZIO 2.x if we need to modify the implementation of one of these services on a more fine-grained basis we can use of the following combinators:
  - `ZIO.withConsole`/`ZIO.withConsoleScoped`
  - `ZIO.withClock`/`ZIO.withClockScoped`
  - `ZIO.withRandom`/`ZIO.withRandomScoped`
  - `ZIO.withSystem`/`ZIO.withSystemScoped`

```scala

object MyClockLive extends Clock {
  ... 
}

ZIO.withClock(MyClockLive)(effect)
```

5. According to the removal of default services from the ZIO environment we no longer need layers defined in the ZIO library which produce default ZIO services. So all these layers were removed, such as the following layers:
- `Console.live`, `Clock.any`
- `Clock.live`, `Clock.javaClock`, `Clock.any`
- `Random.live`, `Random.scalaRandom`, `Random.any`
- `System.live`, `System.any`

6. In ZIO some services have an alternative implementation rather than the default one. In ZIO 1.x, the default implementation of these services was provided by the environment. So when we wanted to use the default implementation, we didn't have to provide them explicitly at the end of the world. But in case we wanted to use an alternative implementation, we had to provide them explicitly. For example, to use the java implementation of `Clock`, we had to provide the `Clock.javaClock` layer:

```scala

object MainApp extends App {
  def run(args: List[String]) =
    clock.localDateTime
      .debug("local date time")
      .provideCustomLayer(
        ZLayer.succeed(
          java.time.Clock.systemDefaultZone()
        ) >>> Clock.javaClock
      )
      .orDie
      .exitCode
}
```

By removal of default services from the environment, their corresponding layers were removed. So we should call them directly as follows:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    Clock.ClockJava(java.time.Clock.systemDefaultZone())
      .currentDateTime
      .debug("current date time")
}
```

The same approach applies to the `Random` service:

| ZIO 1.x (ZLayer)     | ZIO 2.x              |
|----------------------|----------------------|
| `Clock.javaClock`    | `Clock.ClockJava`    |
| `Random.scalaRandom` | `Random.RandomScala` |

7. In ZIO 1.x, the `ZEnv` was the type alias for all default services used in the ZIO environment. In ZIO 2.x, as the default services were removed from the environment, the `ZEnv` type alias was removed. And we have the `DefaultServices.live` to access the live implementation of default services.

## ZIO App

### ZIOApp

In ZIO 1.x, we were used to writing ZIO applications using the `zio.App` trait:

```scala

object MyApp extends zio.App {
  def run(args: List[String]) = 
    startMyApp(args).exitCode
}
```

Now in ZIO 2.x, the `zio.App` trait is deprecated and, we have the `zio.ZIOAppDefault` trait which is simpler than the former approach (Note that the `ZApp` is also deprecated, and we should use the `ZIOAppDefault` instead):

```scala

object MyApp extends ZIOAppDefault {
  def run =
    for {
      arguments <- getArgs
      _         <- startMyApp(arguments) 
    } yield ()
}
```

In ZIO 1.x, `run` is the main function of our application, which will be passed the command-line arguments to our application:

```scala
def run(args: List[String]): URIO[R, ExitCode]
```

While in most cases we don't write command-line applications, and we don't use it, in ZIO 2.x, we created the `ZIOAppArgs` service and a helper method called `ZIOApp#args` which obtains access to the command-line arguments of our application:

```scala
trait ZIOApp { self =>
  final def args: ZIO[ZIOAppArgs, Nothing, Chunk[String]] = ZIO.service[ZIOAppArgs].map(_.args)
}
```

## Fiber

We deprecated the `Fiber.ID` and moved it to the `zio` package and called it the `FiberId`:

| ZIO 1.0        | ZIO 2.x       |
|----------------|---------------|
| `zio.Fiber.ID` | `zio.FiberID` |

## Runtime, Platform and Executor

### Unsafe Marker

To run a ZIO workflow, we usually use `ZIOAppDefault` or `ZIOApp` traits. These traits provide the `run` method which will run the workflow using their default `Runtime` system. But when we want to work with a low-level API or want to integrate with a legacy code, we need to unsafely run the workflow.

In ZIO 1.x, we used the `zio.Runtime.unsafeRun` method to run a ZIO workflow:

```scala
trait Runtime[+R] {
  def unsafeRun[E, A](zio: => ZIO[R, E, A]): A
}
```

For example, if we wanted to integrate a ZIO workflow with a legacy unsafe code, we used to write something like this:

```scala

object MainApp {
  val zioWorkflow: ZIO[Any, Nothing, Int] = ???
  
  def legacyApplication(input: Int): Unit = ???
  
  def zioApplication: Int = 
    Runtime.default.unsafeRun(zioWorkflow)
  

  def main(args: Array[String]): Unit = {
    legacyApplication(zioApplication)
  }

}
```

In ZIO 2.x, we added the `Unsafe` data type to help developers to differentiate lower-level codes that are not purely functional from the higher-level codes which are always pure, total, and type safe. So the `Unsafe` is just a marker capability to indicate that something is unsafe:

```scala
object Unsafe {
  def unsafe[A](f: Unsafe => A): A = ???
}

trait Runtime[+R] { self =>
  def unsafe: UnsafeAPI
  
  trait UnsafeAPI {
    def run[E, A](zio: ZIO[R, E, A])(implicit trace: Trace, unsafe: Unsafe): Exit[E, A]
  }
}
```

So to migrate the previous code to ZIO 2.x, we need to use the `Unsafe` data type like below:

```diff

object MainApp {
  val zioWorkflow: ZIO[Any, Nothing, Int] = ???

  def legacyApplication(input: Int): Unit = ???

  def zioApplication: Int =
-    Runtime.default.unsafeRun(zioWorkflow)
+    Unsafe.unsafe { implicit unsafe =>
+      Runtime.default.unsafe.run(zioWorkflow).getOrThrowFiberFailure()
+    }

  def main(args: Array[String]): Unit = {
    legacyApplication(zioApplication)
  }

}
```

This way it is easy to distinguish between _safe_ and _unsafe_ variants of the same operator. 

To run an unsafe operator, we need implicit value of `Unsafe` in scope. This works particularly well in Scala 3 due to its support for implicit function types championed by Martin Odersky. In Scala 3 we can use the `Unsafe.unsafely` operator to create a block of code in which we can freely call unsafe operators:

```scala
Unsafe.unsafely {
  Runtime.default.unsafe.run(Console.printLine("Hello, World!"))
}
```

If we want to support Scala 2 we need to use a slightly more verbose syntax with `unsafe` and a lambda that takes an implicit value of `Unsafe`:

```scala

Unsafe.unsafe { implicit unsafe =>
  Runtime.default.unsafe.run(Console.printLine("Hello, World!"))
}
```

In summary, here are the rules for migrating from ZIO 1.x to ZIO 2.x corresponding to the unsafe operators:

|         | ZIO 1.0                | ZIO 2.x                                                                               |
|---------|------------------------|---------------------------------------------------------------------------------------|
| Scala 2 | `runtime.unsafeRun(x)` | `Unsafe.unsafe { implicit unsafe => runtime.unsafe.run(x).getOrThrowFiberFailure() }` |
| Scala 3 | `runtime.unsafeRun(x)` | `Unsafe.unsafely { runtime.unsafe.run(x).getOrThrowFiberFailure() }`                    |

### Unsafe Variants

In ZIO 1.x, the `Runtime` had several methods for running ZIO workflows unsafely:

```scala
trait Runtime[+R] {
  def unsafeRun[E, A](zio: => ZIO[R, E, A]): A
  def unsafeRunTask[A](task: => RIO[R, A]): A
  def unsafeRunSync[E, A](zio: => ZIO[R, E, A]): Exit[E, A]
  def unsafeRunAsync[E, A](zio: => ZIO[R, E, A])(k: Exit[E, A] => Any): Unit
  def unsafeRunAsyncCancelable[E, A](zio: => ZIO[R, E, A])(k: Exit[E, A] => Any): Fiber.Id => Exit[E, A]
  def unsafeRunAsync_[E, A](zio: ZIO[R, E, A]): Unit
  def unsafeRunToFuture[E <: Throwable, A](zio: ZIO[R, E, A]): CancelableFuture[A]
}
```

We can group these unsafe methods into two categories: synchronous and asynchronous. The synchronous operators are the ones that are used when we want to wait for the result of the workflow to be available. The asynchronous operators are used when we want to execute the workflow asynchronously by providing a callback function that will be called when the workflow is completed.

In the previous section, we described the new `run` method inside the `unsafe` object of the `Runtime` trait. We can use this method to unsafely run workflows synchronously. There is another method, called `fork`, that can be used to unsafely run workflows asynchronously:

```scala
trait Runtime {
  def unsafe: UnsafeAPI
  
  trait UnsafeAPI {
    def run[E, A](zio: ZIO[R, E, A])(implicit unsafe: Unsafe): Exit[E, A]

    def fork[E, A](zio: ZIO[R, E, A])(implicit unsafe: Unsafe): Fiber.Runtime[E, A]
  }
}
```

The `fork` method returns a `Fiber.Runtime` that can be used to control the execution of the workflow. We have added a new `unsafe` object to the `Fiber.Runtime` class that has several unsafe methods including the `addObserver`:

```scala
object Fiber {
  sealed abstract class Runtime[+E, +A] extends Fiber[E, A] {
    def unsafe: UnsafeAPI

    trait UnsafeAPI {
      def addObserver(observer: Exit[E, A] => Unit)(implicit unsafe: Unsafe): Unit
    }
  }
}
```

Using the `addObserver` method, we can add a callback function of type `Exit[E, A] => Unit` to the underlying fiber. This callback function will be called when the fiber completes. Using these new functionalities, we can implement asynchronous unsafe operators like before. For example, assume we have the following code in ZIO 1.x:

```scala
// ZIO 1.x

Runtime.default.unsafeRunAsync(
  console.putStrLn("After 3 seconds I will return 5").delay(3.seconds).as(5)
)(
  _.fold(
    e => println(s"Failure: $e"),
    v => println(s"Success: $v")
  )
)
```

We can rewrite it in ZIO 2.x as follows:

```scala
// ZIO 2.x

Unsafe.unsafe { implicit unsafe =>
  Runtime.default.unsafe
    .fork(
      Console
        .printLine("After 3 seconds I will return 5")
        .delay(3.second)
        .as(5)
    )
    .unsafe
    .addObserver(
      _.fold(
        e => println(s"Failure: $e"),
        v => println(s"Success: $v")
      )
    )
}
```

Similarly, we can do the same for other unsafe operators. Here are some of them:

| ZIO 1.0                        | ZIO 2.x                                                                   |
|--------------------------------|---------------------------------------------------------------------------|
| `runtime.unsafeRunSync(x)`     | `Unsafe.unsafe { implicit unsafe => runtime.unsafe.run(x) }`              |
| `runtime.unsafeRunTask(x)`     | `Unsafe.unsafe { implicit unsafe => runtime.unsafe.run(x).getOrThrow() }` |
| `runtime.unsafeRunAsync_(x)`   | `Unsafe.unsafe { implicit unsafe => runtime.unsafe.fork(x) }`             |
| `runtime.unsafeRunToFuture(x)` | `Unsafe.unsafe { implicit unsafe => runtime.unsafe.runToFuture(x) }`      |

### Runtime Customization using Layers

In ZIO 2.x we deleted the `zio.internal.Platform` data type, and instead, we use layers to customize the runtime. This allows us to use ZIO workflows in customizing our runtime (e.g. loading some configuration information to set up logging).

In ZIO 1.x, we had the `Platform` data type useful for providing custom execution configurations to the runtime:
- `Platform#withExecutor`— To provide a custom `Executor`
- `Platform#withTracing` to config tracing functionality
- `Platform#withSupervisor` to provide a `Supervisor`
- `Platform#withScheduler` to provide a `Scheduler`
- etc.

Here is an example of creating a custom `Runtime` in ZIO 1.x:

```scala

object MainApp extends zio.App {
  val customExecutor: Executor = ???

  val myApp: UIO[Unit] =
    ZIO.debug("Application started")

  def run(args: List[String]): URIO[ZEnv, ExitCode] =
    ZIO
      .runtime[ZEnv]
      .map { runtime =>
        Unsafe.unsafe { implicit unsafe =>
          runtime
            .mapPlatform(_.withExecutor(customExecutor))
            .unsafe
            .run(myApp)
            .getOrThrowFiberFailure()
        }
      }
      .exitCode
}
```

In ZIO 2.x, the whole `Platform` was deleted and instead, we have several out-of-the-box layers for runtime customization, defined in the companion object of the `Runtime` trait. Here are some of them:
- `Runtime.addLogger` to add a logger
- `Runtime.setExecutor` to provide a custom `Executor`
- `Runtime.enableOpLog` to log runtime information
- `Runtime.enableRuntimeMetrics` to track runtime metrics
- etc.

Let's see how a previous example can be rewritten in ZIO 2.x:

```scala

object MainApp extends ZIOAppDefault {
  val customExecutor: zio.Executor = ???

  val myApp = 
    ZIO.debug("Application started")

  def run =
    myApp.provide(
      Runtime.setExecutor(customExecutor)
    )
}
```

Note that ZIO ecosystem libraries like ZMX may have their own layers that install all necessary functionality.

### Runtime Customization Using ZIO Data Type

To access information about the configuration of our ZIO program as we are running, there are some more specific operators that we can use, such as:
- `ZIO.executor`/`ZIO.executorWith`
- `ZIO.logger`/`ZIO.loggerWith`
- `ZIO.isFatal`/`ZIO.isFatalWith`

### Runtime Configurations are Scoped

When we access a `Runtime` using `ZIO.runtime` it will inherit all the configuration of the current workflow so if we use it to run effects they will be run with the same logger and so on:

```scala

object MainApp extends ZIOAppDefault {
  val workflow1 = ZIO.debug("workflow1 is running") *> ZIO.log("This line will never get logged")
  val workflow2 = ZIO.debug("workflow2 is running") *> ZIO.log("This line will get logged")
  val workflow3 = ZIO.debug("workflow3 is running") *> ZIO.log("This line will never get logged")

  def run =
    ZIO.provideLayer(Runtime.removeDefaultLoggers) {
      ZIO.runtime[Any].flatMap(_.run(workflow1)) *>
        ZIO.provideLayer(Runtime.addLogger(Runtime.defaultLoggers.head)) {
          ZIO.runtime[Any].flatMap(_.run(workflow2))
        } *> workflow3
    }
}
```

### Custom Runtime for Mixed Applications

In ZIO 2.x, to create a custom runtime in mixed applications we combine all the layers that do our customization and then perform the `Runtime.unsafe.fromLayer` operation:

```scala

object MainApp {
  val sl4jlogger: ZLogger[String, Any] = ???

  def legacyApplication(input: Int): Unit = ???

  val zioWorkflow: ZIO[Any, Nothing, Int] = ???

  def zioApplication(): Int =
      Unsafe.unsafe { implicit unsafe =>
        Runtime
          .unsafe
          .fromLayer(
            Runtime.removeDefaultLoggers ++ Runtime.addLogger(sl4jlogger)
          )
          .unsafe
          .run(zioWorkflow)
          .getOrThrowFiberFailure()
      }

  def main(args: Array[String]): Unit = {
    val result = zioApplication()
    legacyApplication(result)
  }  

}
```

### Executor

We moved the `Executor` from `zio.internal` to the `zio` package:

| ZIO 1.0                 | ZIO 2.x        |
|-------------------------|----------------|
| `zio.internal.Executor` | `zio.Executor` |

### Auto-Blocking

In ZIO 1.x, we have two groups of constructors for importing _synchronous side effects_, one for importing synchronous side effects, such as `zio.effect` and the other one for importing synchronous side effects that are known to be blocking, such as `zio.blocking.effectBlocking`.

The first one uses an asynchronous thread pool to execute side effects, while the second one uses a blocking thread pool.

For performance reasons, the number of asynchronous threads is limited to a fixed number. So if the programmer mistakenly imports a blocking operation using the `ZIO.effect` instead of `zio.blocking.effectBlocking` it might block all limited threads in the asynchronous pool, and then starvation might occur.

So if we run the following code if the `ioBoundWorkflow` starts executing before the `cpuBoundWorkflow` for some amount of time, all threads in the asynchronous thread pool will be blocked and the `cpuBoundWorkflow` will never get executed:

```scala
// ZIO 1.x

object MainApp extends App {

  def fib(n: Int): BigInt = {
    @tailrec
    def go(n: BigInt, a: BigInt, b: BigInt): BigInt = {
      if (n == 0) a
      else go(n - 1, b, a + b)
    }
    go(n, 0, 1)
  }

  def ioBoundWorkflow =
    ZIO.debug("Starting I/O bound workflow") *>
      ZIO.foreachPar_(1 to 100)(_ => ZIO.effect(Thread.sleep(Long.MaxValue))) *>
      ZIO.debug("Finished I/O bound workflow")

  def cpuBoundWorkflow =
    ZIO.debug("Starting CPU bound workflow") *>
      ZIO.foreachPar_(1 to 100)(i => ZIO.effect(fib(i))) *>
      ZIO.debug("Finished CPU bound workflow")

  // the delay for the CPU bound workflow is not needed but we want to take 
  // a chance that all threads in the asynchronous thread pool will be blocked
  def run(args: List[String]) =
    (ioBoundWorkflow <&> cpuBoundWorkflow.delay(1.second)).exitCode
}
```

This is why we should import blocking synchronous side effects using the `zio.blocking.effectBlocking` instead of the `ZIO.effect`.

In ZIO 2.x, the same as in ZIO 1.x, we encourage separating blocking operations (I/O work operations) from the ordinary side effects (CPU work operations).

But the one thing that makes ZIO 2.x more powerful than ZIO 1.x is that if the programmer accidentally imports a blocking synchronous side effect using the `ZIO.attempt` instead of `ZIO.attemptBlocking` the runtime scheduler will automatically detect blocking workflows and shift them to the blocking executor:

```scala
// ZIO 2.x

object MainApp extends ZIOAppDefault {

  def fib(n: Int): BigInt = {
    @tailrec
    def go(n: BigInt, a: BigInt, b: BigInt): BigInt = {
      if (n == 0) a
      else go(n - 1, b, a + b)
    }
    go(n, 0, 1)
  }

  def ioBoundWorkflow =
    ZIO.debug("Starting I/O bound workflow") *>
      ZIO.foreachParDiscard(1 to 100)(_ =>
        ZIO.attempt(Thread.sleep(Long.MaxValue))
      ) *>
      ZIO.debug("Finished I/O bound workflow")

  def cpuBoundWorkflow =
    ZIO.debug("Starting CPU bound workflow") *>
      ZIO.foreachParDiscard(1 to 100)(i => ZIO.attempt(fib(i))) *>
      ZIO.debug("Finished CPU bound workflow")

  def run = ioBoundWorkflow <&> cpuBoundWorkflow.delay(1.second)
}
```

In the above example, although we imported the blocking operation wrongly, the runtime scheduler will detect that and prevent the blocking operation from being executed in the asynchronous thread pool. So the `cpuBoundWorkflow` will be executed without any starvation problem.

## ZLayer

### Constructing Layers

In ZIO 1.x, when we want to write a service that depends on other services, we need to use `ZLayer.fromService*` variants with a lot of boilerplate:

```scala
val live: URLayer[FooService with BarService, BazService] =
  ZLayer.fromServices[FooService.Service, BarService.Service, BazService.Service] {
    (fooService: FooService.Service, barService: BarService.Service) =>
      new BazService.Service {
        override def baz: UIO[Unit] =
          for {
            _ <- fooService.foo
            _ <- barService.bar
          } yield ()
      }
  }
```

ZIO 2.x deprecates all `ZLayer.fromService*` functions. Instead, we use a for comprehension:

```scala
trait FooService {
  def foo: UIO[Unit]
}

trait BarService {
  def bar: UIO[Unit]
}

trait BazService {
  def baz: UIO[Unit]
}

case class BazServiceImpl(fooService: FooService, barService: BarService) extends BazService {
  override def baz: UIO[Unit] =
    for {
      _ <- fooService.foo
      _ <- barService.bar
    } yield ()
}

object LoggingLive {
  val layer: ZLayer[FooService & BarService, Nothing, BazService] =
    ZLayer {
      for {
        fooService <- ZIO.service[FooService]
        barService <- ZIO.service[BarService]
      } yield BazServiceImpl(fooService, barService)
    }
}
```

### Accessing a Service from the Environment

Assume we have a service named `Logging`:

```scala
trait Logging {
  def log(line: String): UIO[Unit]
}
```

In ZIO 1.x, when we wanted to access a service from the environment, we used the `ZIO.access` + `Has#get` combination (`ZIO.access(_.get)`):

```scala
val logging: URIO[Logging, Logging] = ZIO.access(_.get)
```

Also, to create accessor methods, we used the following code:

```scala
def log(line: String): URIO[Logging, Unit] = ZIO.accessM(_.get.log(line))
```

ZIO 2.x reduces one level of indirection by using `ZIO.service` operator:

```scala
val logging : URIO[Logging, Logging] = ZIO.service
```

And to write the accessor method in ZIO 2.x, we can use `ZIO.serviceWithZIO` operator:

```scala
def log(line: String): URIO[Logging, Unit] = ZIO.serviceWithZIO(_.log(line))
```

Note that [accessor methods](../../reference/service-pattern/accessor-methods.md) are no longer recommended.

### Accessing Multiple Services in the Environment

In ZIO 1.x, we could access multiple services using higher arity service accessors like `ZIO.services`.

```scala
for {
  (fooService, barService) <- ZIO.services[FooService, BarService]
  foo                      <- fooService.foo()
  bar                      <- barService.bar()
  _                        <- console.putStrLn(s"foo: $foo, bar: $bar")
} yield ()
```

These accessor methods are now _deprecated_ as we can achieve the same functionality using `ZIO.service` with for-comprehension syntax, which is more idiomatic and scalable way of accessing multiple services in the environment:

```scala
for {
  fooService <- ZIO.service[FooService]
  barService <- ZIO.service[BarService]
  foo        <- fooService.foo()
  bar        <- barService.bar()
  _          <- Console.printLine(s"foo: $foo, bar: $bar")
} yield ()
```

### Building the Dependency Graph

To create the dependency graph in ZIO 1.x, we should compose the required layer manually. As the ordering of layer compositions matters, and also we should care about composing layers in both vertical and horizontal manner, it would be a cumbersome job to create a dependency graph with a lot of boilerplates.

Assume we have the following dependency graph with two top-level dependencies:

```
           DocRepo                ++          UserRepo
      ____/   |   \____                       /     \
     /        |        \                     /       \
 Logging  Database  BlobStorage          Logging   Database
    |                    |                  |
 Console              Logging            Console
                         |       
                      Console    
```

In ZIO 1.x, we had to compose these different layers together to create the whole application dependency graph:

```scala
val appLayer: URLayer[Any, DocRepo with UserRepo] =
  (((Console.live >>> Logging.live) ++ Database.live ++ (Console.live >>> Logging.live >>> BlobStorage.live)) >>> DocRepo.live) ++
    (((Console.live >>> Logging.live) ++ Database.live) >>> UserRepo.live)
    
val res: ZIO[Any, Nothing, Unit] = myApp.provideLayer(appLayer)
```

As the development of our application progress, the number of layers will grow, and maintaining the dependency graph would be tedious and hard to debug.

For example, if we miss the `Logging.live` dependency, the compile-time error would be very messy:

```scala
myApp.provideLayer(
  ((Database.live ++ BlobStorage.live) >>> DocRepo.live) ++
    (Database.live >>> UserRepo.live)
)
```

```
type mismatch;
 found   : zio.URLayer[zio.Logging with zio.Database with zio.BlobStorage,zio.DocRepo]
    (which expands to)  zio.ZLayer[zio.Logging with zio.Database with zio.BlobStorage,Nothing,zio.DocRepo]
 required: zio.ZLayer[zio.Database with zio.BlobStorage,?,?]
    ((Database.live ++ BlobStorage.live) >>> DocRepo.live) ++
```

In ZIO 2.x, we can automatically construct dependencies with friendly compile-time hints, using `ZIO#provide` operator:

```scala
val res: ZIO[Any, Nothing, Unit] =
  myApp.provide(
    Logging.live,
    Database.live,
    BlobStorage.live,
    DocRepo.live,
    UserRepo.live
  )
```

The order of dependencies doesn't matter:

```scala
val res: ZIO[Any, Nothing, Unit] =
  myApp.provide(
    DocRepo.live,
    BlobStorage.live,
    Logging.live,
    Database.live,
    UserRepo.live
  )
```

If we miss some dependencies, it doesn't compile, and the compiler gives us the clue:

```scala
val app: ZIO[Any, Nothing, Unit] =
  myApp.provide(
    DocRepo.live,
    BlobStorage.live,
//    Logging.live,
    Database.live,
    UserRepo.live
  )
```

```
  ZLayer Wiring Error  

❯ missing Logging
❯     for DocRepo.live

❯ missing Logging
❯     for UserRepo.live
```

We can also directly construct a layer using `ZLayer.make`:

```scala
val layer = ZLayer.make[DocRepo with UserRepo](
  Logging.live,
  DocRepo.live,
  Database.live,
  BlobStorage.live,
  UserRepo.live
)
```

And also the `ZLayer.makeSome` helps us to construct a layer which requires some service and produces some other services (`URLayer[Int, Out]`) using `ZLayer.makeSome[In, Out]`:

```scala
val layer = ZLayer.makeSome[Logging, DocRepo with UserRepo](
  DocRepo.live,
  Database.live,
  BlobStorage.live,
  UserRepo.live
)
```

In ZIO 1.x, the `ZIO#provideSomeLayer` provides environment partially:

```scala
val app: ZIO[Console, Nothing, Unit] =
  myApp.provideSomeLayer[Console](
    ((Logging.live ++ Database.live ++ (Console.live >>> Logging.live >>> BlobStorage.live)) >>> DocRepo.live) ++
      (((Console.live >>> Logging.live) ++ Database.live) >>> UserRepo.live)
  )
```

In ZIO 2.x, we have a similar functionality but for injection, which is the `ZIO#provideSome[Rest](l1, l2, ...)` operator:

```scala
val app: ZIO[Any, Nothing, Unit] =
  myApp.provideSome[Logging](
    DocRepo.live,
    Database.live,
    BlobStorage.live,
    UserRepo.live
  )
```

:::note
In ZIO 2.x, the `ZIO#provide` method — together with its variant `ZIO#provideSome` — is a default and easier way of injecting dependencies to the environmental effect. We do not have to create the dependency graph manually, it will be built automatically. In contrast, the `ZIO#provideLayer` — and its variant `ZIO#provideSomeLayer` — is useful for low-level and custom cases.
:::

### ZLayer Debugging

To debug ZLayer construction, we have two built-in layers, i.e., `ZLayer.Debug.tree` and `ZLayer.Debug.mermaid`. For example, by including `ZLayer.Debug.mermaid` into our layer construction, the compiler generates the following debug information:

```scala
val layer = ZLayer.make[DocRepo with UserRepo](
  Logging.live,
  DocRepo.live,
  Database.live,
  BlobStorage.live,
  UserRepo.live,
  ZLayer.Debug.mermaid
)
```

```scala
[info]   ZLayer Wiring Graph  
[info] 
[info] ◉ DocRepo.live
[info] ├─◑ Logging.live
[info] ├─◑ Database.live
[info] ╰─◑ BlobStorage.live
[info]   ╰─◑ Logging.live
[info] 
[info] ◉ UserRepo.live
[info] ├─◑ Logging.live
[info] ╰─◑ Database.live
[info] 
[info] Mermaid Live Editor Link
[info] https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGhcbiAgICBDb25zb2xlLmxpdmVcbiAgICBCbG9iU3RvcmFnZS5saXZlIC0tPiBMb2dnaW5nLmxpdmVcbiAgICBMb2dnaW5nLmxpdmUgLS0+IENvbnNvbGUubGl2ZVxuICAgIFVzZXJSZXBvLmxpdmUgLS0+IExvZ2dpbmcubGl2ZVxuICAgIFVzZXJSZXBvLmxpdmUgLS0+IERhdGFiYXNlLmxpdmVcbiAgICBEb2NSZXBvLmxpdmUgLS0+IERhdGFiYXNlLmxpdmVcbiAgICBEb2NSZXBvLmxpdmUgLS0+IEJsb2JTdG9yYWdlLmxpdmVcbiAgICBEYXRhYmFzZS5saXZlXG4gICAgIiwibWVybWFpZCI6ICJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCAidXBkYXRlRWRpdG9yIjogdHJ1ZSwgImF1dG9TeW5jIjogdHJ1ZSwgInVwZGF0ZURpYWdyYW0iOiB0cnVlfQ==
```

### Descriptive ZIOApp Environment Compiler Errors

In ZIO 1.x, if we mistakenly forget to provide some requirements, we have some complicated compile errors. Assume we have the following example:

```scala
// ZIO 1.x

case class Config(host: String, port: Int)

trait Logger {
  def log(line: Any): Task[Unit]
}

object MainApp extends App {
  val myApp =
    for {
      config <- ZIO.service[Config]
      logger <- ZIO.service[Logger]
      _ <- logger.log(s"Application started with the following config: $config")
    } yield ()

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.exitCode
}
```

This program needs `Config` and `Logger` layers, but we missed them. The compiler prints this mystic error, which is hard to reason about:

```scala
type mismatch;
found   : zio.URIO[zio.Has[Config] with zio.Has[Logger] with zio.console.Console,zio.ExitCode]
(which expands to)  zio.ZIO[zio.Has[Config] with zio.Has[Logger] with zio.Has[zio.console.Console.Service],Nothing,zio.ExitCode]
required: zio.URIO[zio.ZEnv,zio.ExitCode]
(which expands to)  zio.ZIO[zio.Has[zio.clock.Clock.Service] with zio.Has[zio.console.Console.Service] with zio.Has[zio.system.System.Service] with zio.Has[zio.random.Random.Service] with zio.Has[zio.blocking.Blocking.Service],Nothing,zio.ExitCode]
myApp.exitCode
```

In ZIO 2.x, we have descriptive errors. Let's try the above example in ZIO 2.x:

```scala

case class Config(host: String, port: Int)

trait Logger {
  def log(line: Any): Task[Unit]
}

object MainApp extends ZIOAppDefault {
  def run =
    for {
      config <- ZIO.service[Config]
      logger <- ZIO.service[Logger]
      _ <- logger.log(s"Application started with the following config: $config")
    } yield ()
}
```

This will print the following error message:

```
[error] ──── ZIO APP ERROR ───────────────────────────────────────────────────
[error] 
[error]  Your effect requires a service that is not in the environment.
[error]  Please provide a layer for the following type:
[error] 
[error] 
[error]    1. Logger
[error] 
[error] 
[error]  Call your effect's provide method with the layers you need.
[error]  You can read more about layers and providing services here:
[error]  
[error]    https://zio.dev/version-1.x/dataypes/contextual/index
[error] 
[error] ──────────────────────────────────────────────────────────────────────
[error] 
[error]       logger <- ZIO.service[Logger]
[error]              ^
[error] one error found
```

It also warns if we provide more layers than needed:

```scala

case class Config(host: String, port: Int)

trait Logger {
  def log(line: Any): Task[Unit]
}

case class LoggerLive(console: Console) extends Logger {
  override def log(line: Any): Task[Unit] =
    console.printLine(line)
}

object LoggerLive {
  val layer =
    ZLayer {
      for {
        console <- ZIO.service[Console]
      } yield LoggerLive(console)
    }
}

object MainApp extends ZIOAppDefault {
  val myApp = for {
    config <- ZIO.service[Config]
    _ <- Console.printLine(s"Application started with the following config: $config")
  } yield ()

  def run =
    myApp.provide(
      Random.live,
      ZLayer.succeed(Config("localhost", 8080)),
      LoggerLive.layer
    )
}
```

It will print the following warning message:

```
[error] ──── ZLAYER WARNING ──────────────────────────────────────────────────
[error] 
[error]  You have provided more than is required.
[error]  You may remove the following 2 layers:
[error]    
[error]    1. Random.live
[error]    2. LoggerLive.layer
[error] 
[error]   
[error] ──────────────────────────────────────────────────────────────────────
[error] 
[error]     myApp.provide(
[error]                  ^
[error] one error found
```

### Eliminators for Environmental Effects

In ZIO 2.x, layers become eliminators for environmental effects:

```scala
trait Foo
trait Bar

val fooLayer   : ZLayer[Any, Nothing, Foo]          = ZLayer.succeed(???)
val fooWithBar : ZIO[Foo with Bar, Throwable, Unit] = ZIO.succeed(???)
val bar        : ZIO[Bar, Any, Unit]                = fooLayer(fooWithBar)
```

In this example, by applying the `fooLayer` to the `fooWithBarWithBaz` effect it will eliminate the contextual `Foo` service from the environment of the effect.

Furthermore, we can compose multiple layers together and then eliminate services from the environmental effects:

```scala

trait Foo
trait Bar
trait Baz

object MainApp extends ZIOAppDefault {

  val needsFooAndBarAndBaz: ZIO[Foo & Bar & Baz, Nothing, Unit] =
    for {
      foo <- ZIO.service[Foo]
      bar <- ZIO.service[Bar]
      baz <- ZIO.service[Baz]
      _ <- ZIO.debug(s"Foo: $foo, Bar: $bar, Baz: $baz")
    } yield ()

  val fooAndBarLayer: ULayer[Foo with Bar] =
    ZLayer.succeed(new Foo {}) ++ ZLayer.succeed(new Bar {})

  val needsBaz: ZIO[Baz, Nothing, Unit] =
    fooAndBarLayer(needsFooAndBarAndBaz)

  def run = needsBaz.provide(ZLayer.succeed(new Baz {}))

}
```

It helps us to provide contextual environments like the DSL below:

```scala

trait Connection

val dbTransaction: ZLayer[Any, Throwable, Connection] = ZLayer.succeed(???)
val effect: ZIO[Connection, Throwable, Unit] = ZIO.succeed(???)

val result: ZIO[Any, Throwable, Unit] =
  dbTransaction {
    effect
  }
```

The `dbTransaction` is a `ZLayer` of type `ZLayer[Any, Throwable, Connection]` which can eliminate the `Connection` service from the `effect`.

### Service Pattern

The _Service Pattern_, formerly called "Module Pattern", is one of the most important changes in ZIO 2.x. Let's take a look at services in ZIO 1.x before discussing changes. 

Assume we have already defined the `Foo` and `Bar` services using _Service Pattern 1.0_ as follows:

```scala
// ZIO 1.x
object foo {
  type Foo = Has[Foo.Service]

  object Foo {
    trait Service {
      def foo: UIO[String]
    }
  }
}

object bar {
  type Bar = Has[Bar.Service]

  object Bar {
    trait Service {
      def bar: UIO[String]
    }
  }
}
```

Here is a `Baz` service that uses the `Foo` and `Bar` services:

```scala
// ZIO 1.x
object baz {
  // Defining the service type by wrapping the service interface with Has[_] data type
  type Baz = Has[Baz.Service]

  // Companion object that holds service interface and its live implementation
  object Baz {
    trait Service {
      def baz(input: String): UIO[Unit]
    }
    
    // Live implementation of the Foo service
    val live: ZLayer[Foo with Bar, Nothing, Baz] =
      ZLayer.fromServices[Foo.Service, Bar.Service, Baz.Service] {
        (fooSrv: Foo.Service, barSrv: Bar.Service) =>
          new Baz.Service {
            override def baz(input: String): UIO[Unit] =
              for {
                _ <- fooSrv.foo(input)
                _ <- barSrv.bar(input)
              } yield ()
          }
      }
  }

  // Accessor Methods
  def baz(input: String): URIO[Baz, Unit] =
    ZIO.accessM(_.get.baz(input))
}
```

The `Baz` service is a service which depends on the `Foo` and `Bar` services.

ZIO 2.x introduces the _Service Pattern 2.0_ which is much more concise and has more ergonomics. Let's see how the `Baz` service can be implemented using this new pattern.

Assume we have already defined the `Foo` and `Bar` services, as below:

```scala
// ZIO 2.x

trait Foo {
  def foo(input: String): UIO[Unit]
}

trait Bar {
  def bar(input: String): UIO[Unit]
}
```

Now, here is the implementation of the `Baz` service based on the _Service Pattern 2.0_:

```scala
// ZIO 2.x

// Defining the Service Interface
trait Baz {
  def baz(input: String): UIO[Unit]
}

// Implementation of the Service Interface
case class BazLive(fooSrv: Foo, barSrv: Bar) extends Baz {
  override def baz(input: String): UIO[Unit] =
    for {
      _ <- fooSrv.foo(input)
      _ <- barSrv.bar(input) 
    } yield ()
}

// Converting the Service Implementation into the ZLayer
object BazLive {
  val layer: URLayer[Foo & Bar, Baz] =
    ZLayer {
      for {
        fooSrv <- ZIO.service[Foo]
        barSrv <- ZIO.service[Bar]
      } yield BazLive(fooSrv, barSrv)
    }
}
```

As we see, we have the following changes:

1. **Deprecation of Type Alias for `Has` Wrappers** — In _Service Pattern 1.0_ although the type aliases were to prevent using `Has[ServiceName]` boilerplate everywhere, they were confusing, and led to doubly nested `Has[Has[ServiceName]]`. So the _Service Pattern 2.0_ doesn't anymore encourage using type aliases. Also, they were removed from all built-in ZIO services. So the `type Foo = Has[Foo.Service]` is removed and the `Foo.Service` will just be `Foo`.

2. **Introducing Constructor-based Dependency Injection** — In _Service Pattern 1.0_ when we wanted to create a layer that depends on other services, we had to use `ZLayer.fromService*` constructors. The problem with those constructors is that there are too many of them, each one is useful for a specific use-case, but people had trouble figuring out which one to use. 

    In _Service Pattern 2.0_ we don't worry about all these different `ZLayer` constructors. The recommendation is to  **provide dependencies as interfaces through the case class constructor**, and then we have direct access to all of the dependencies to implement the service. Finally, to create the `ZLayer` we use a for comprehension.

3. **Separated Interface** — In _Service Pattern 2.0_, ZIO supports the _Separated Interface_ pattern which encourages keeping the implementation of an interface decoupled from the client and its definition.

    As our application grows, where we define our layers matters more. _Separated Interface_ is a very useful pattern while we are developing a complex application. It helps us to reduce the coupling between application components. 

    Following two changes in _Service Pattern_ we can define the service definition in one package but its implementations in other packages:
    
   1. **Flattened Structure** — In the new pattern, everything is at the top level in a file. So the developer is not limited to package service definition and service implementation in one package.
   
      :::note
      Service Pattern 2.0 supports the idea of _Separated Interface_, but it doesn't enforce us grouping them into different packages and modules. The decision is up to us, based on the complexity and requirements of our application.
      :::
   
   2. **Decoupling Interfaces from Implementation** — Assume we have a complex application, and our interface is `Baz` with different implementations that potentially depend on entirely different modules. Putting layers in the service definition means anyone depending on the service definition needs to depend on all the dependencies of all the implementations, which is not a good practice.
   
    In _Service Pattern 2.0_, layers are defined in the implementation's companion object, not in the interface's companion object. So instead of calling `Baz.live` to access the live implementation we call `BazLive.layer`.

4. **Accessor Methods** — The new pattern reduced one level of indirection on writing accessor methods. So instead of accessing the environment (`ZIO.access/ZIO.accessM`) and then retrieving the service from the environment (`Has#get`) and then calling the service method, _Service Pattern 2.0_ introduced `ZIO.serviceWith` that is a more concise way of writing accessor methods. For example, instead of `ZIO.accessM(_.get.baz(input))` we write `ZIO.serviceWithZIO(_.baz(input))`.

_Service Pattern 1.0_ was somewhat complicated and had some boilerplate. [_Service Pattern 2.0_](../../reference/service-pattern/service-pattern.md) is much more familiar for people coming from an object-oriented world. It is simpler and easier to learn for newcomers.

### Other Changes

Here is list of other deprecated methods:

| ZIO 1.x                    | ZIO 2.x                      |
|----------------------------|------------------------------|
| `ZLayer.fromEffect`        | `ZLayer.fromZIO`             |
| `ZLayer.fromEffectMany`    | `ZLayer.fromZIOMany`         |
| `ZLayer.fromFunctionM`     | `ZLayer.fromFunctionZIO`     |
| `ZLayer.fromFunctionManyM` | `ZLayer.fromFunctionManyZIO` |
| `ZLayer.identity`          | `ZLayer.service`             |
| `ZLayer.requires`          | `ZLayer.service`             |

## Scopes

ZIO 2.x introduced a new data type called `Scope`. Scopes are a huge simplification to resource management in ZIO 2.0 that brings new levels of simplicity, power, and performance. They are a replacement for the old `ZManaged` data type in ZIO 1.x.

### ZManaged

In ZIO 1.x, we used a data type called `ZManaged` to provide resource safety. Although inspired by the Managed data type from Haskell, ZIO Managed innovated in a number of ways over both Haskell and Cats Effect Resource:

1. `ZManaged` supported interruptible acquisition, which is useful for concurrent data structures like semaphores, where it is safe to interrupt acquisition because the cleanup can determine (by inspecting the in-memory state) whether the acquisition succeeded or not.

2. `ZManaged` supported parallel operations, all with the most desirable semantics possible. For example, if we acquired resources in parallel, then they would be released in parallel, and if anything went wrong, of course, the acquisition would be aborted and resources released.

3. `ZManaged` has an API almost identical to `ZIO`, by design, so our knowledge of `ZIO` transfers to `Managed`. The main difference is that in `ZManaged`, `flatMap` lets us _use and keep open a resource_, while `use` lets us _use and release the resource (going back to `ZIO`)_.

4. In addition, `ZManaged` has new constructors, so we can create them from a pair of acquire/release actions, or from just a finalizer (which would be invoked during finalization).

Despite the innovation and numerous benefits, however, `ZManaged` presents some serious drawbacks:

- First, `ZManaged` is yet another thing to teach to developers. Many new `ZIO` developers try to avoid using `ZManaged`, because they are not sure exactly what it's for or how it differs from `ZIO`. Those who use it, sometimes wonder when to use `ZIO` versus `ZManaged`.

- Second, all the methods on `ZIO` must be manually and painstakingly re-implemented on `ZManaged`, but with much more complex implementations due to the complications of handling resource-safety in the presence of concurrency. In practice, `ZIO` still has more methods than `ZManaged`.

- Third, `ZManaged` is a layer over `ZIO`, and is slower than `ZIO` itself, because of the additional complications and wrapping. Unlike other approaches, `ZIO` uses an executable encoding, so it's able to avoid double-interpretation, but it still has measurable overhead over `ZIO`.

Despite the drawbacks of `ZManaged`, the benefits of concurrent resource safety are significant, so we documented, supported, and tried to optimize `ZManaged` over the life of ZIO 1.x, not having a suitable alternative.

### Scopes

The concept of scopes has been implicit in ZIO since before ZIO 1.0, including in `ZManaged`, `FiberRef`, interruptibility, and thread pool shifting. In each of these cases we "do something" at the beginning of the scope (e.g. acquire a resource, set a `FiberRef`, change the interruptibility of the thread pool) and "do something else" (release the resource, restore the `FiberRef`, restore the interruptibility or thread pool) at the end of the scope.

However, scopes have not been first-class values, which has required the use of other data types such as `ZManaged` to represent this concept. With ZIO 2.0, all of this is radically changing for the better! Thanks to other ZIO 2.0 innovations, including the [removal of `Has`](#deletion-of-has-data-type) (which bakes a compositional environment directly into the ZIO data type), we have found a way to delete `ZManaged` entirely, while preserving all of its benefits!

ZIO 2.x addresses this by introducing the concept of a Scope as a first class value:

```scala
trait Scope {
  def addFinalizerExit(finalizer: Exit[Any, Any] => UIO[Any]): UIO[Unit]
  def close(exit: Exit[Any, Any]): UIO[Unit]
}
```

That is, a `Scope` is something that we can add finalizers to and eventually close, running all of the finalizers in the scope. Operations that acquire resources add their finalizers directly to the current scope, stored in ZIO Environment. So we found that the ZIO Environment is now powerful enough to provide resource-safety by itself!

For example, an operation that opens a file might have this type signature:

```scala
def openFile(name: String): ZIO[Scope, IOException, FileInputStream] =
  ZIO.acquireRelease(acquire)(release)
```

The `openFile` workflow requires a `Scope` to be run and its implementation will add a finalizer to the scope that will close the file. So, in combination with the environment, we can use `Scope` to represent resources.

This allows us to work with the resource and compose it with other resources, much like we do with `ZManaged`. Then, when we are ready to close the scope we use `ZIO.scoped` to provide the scope and eliminate it from the environment, much the same way we do with `use` on `ZManaged`:

```scala
ZIO.scoped {
  openFile(name).flatMap(file => useFile(file))  // ZIO[Scoped, IOException, Unit]
}                                                // ZIO[Any, IOException, Unit]
```

`ZIO.scoped` eliminates `Scope` from the environment, leaving the rest of the environment unchanged. It converts the type of enclosed effect from `ZIO[Scoped, IOException, Unit]` to `ZIO[Any, IOException, Unit]`. So, we can think of it as an algebraic effect handler that handles the `Scope` effect by eliminating it from the set of algebraic effects being used. This is another use case of [eliminators for environmental effect](#eliminators-for-environmental-effects).

This simple, beautiful, and powerful design gives us bulletproof parallel and concurrent operators that may acquire resources with well-defined and optimal semantics in successful and failure scenarios. All the `ZManaged` semantics arise for free atop `ZIO`.

Scopes are simple because they don't require us to learn any new data types. Scopes are powerful because ZIO has more operators than `ZManaged`, and is always _up to date_ with the latest and greatest. Scopes are fast because there are no layers atop ZIO.

In addition to providing simpler, more powerful, and faster resource management, the replacement of Managed with scopes is going to tremendously simplify the ZIO API: there will not be any more `toManaged`, `mapManaged`, etc, variants. Layers will always be constructed with `ZIO`.

### Migration from `ZManaged` to `Scope`

#### Deferred Migration
For reasons of backward compatibility, `ZManaged` won't actually be deleted, but rather, moved to a separate library called `zio-managed` that our ZIO 2.0 application can depend on. However, ZIO Core, including ZIO Streams and ZIO Test, will no longer use `ZManaged`.

So, if we have a lot of code that used `ZManaged` and we are not ready to deal with it right now we can still use the `ZManaged` data type and compile our code. We can add the `zio-managed` dependency into the `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-managed" % "<2.x version>"
```
And then by adding `import zio.managed._` we can access all `ZManaged` capabilities including extension methods on ZIO data types. This helps us to compile the ZIO 1.x code base which uses the `ZManaged` data type. Then we can smoothly refactor it to use the `Scope` data type instead.

#### Immediate Migration

Migration to `Scope` is easy and straightforward. As the `ZManaged` data type is removed from `ZIO` and all usages in ZIO Core, ZIO Stream, and ZIO Test are reimplemented in terms of `Scope`. We should follow these steps to migrate the `ZManaged` codebase to `Scope`:

1. Replace all references to `ZManaged[R, E, A]` with `ZIO[R with Scope, E, A]`.

Example: 

```diff
object HttpClient {
-   def make(): ZManaged[Config, IOException, HttpClient] = ???
+   def make(): ZIO[Config with Scope, IOException, HttpClient] = ???
}
```

2. Replace all references to `resource.use(f)` with `ZIO.scoped(resource.flatMap(f))`:

```diff
- resource.use(f)
+ ZIO.scoped {
+  resource.flatMap(f) 
+ }
```

Example:

```diff
- ZManaged
-  .fromAutoCloseable(zio.blocking.effectBlockingIO(scala.io.Source.fromFile("file.txt")))
-  .use(x => ZIO.succeed(x.getLines().length))
+ ZIO.scoped {
+   ZIO
+    .fromAutoCloseable(ZIO.attemptBlockingIO(scala.io.Source.fromFile("file.txt")))
+    .flatMap(x => ZIO.succeed(x.getLines().length))
+ }
```

3. Replace all `ZManaged` constructors with `ZIO.acquireRelease` or one of its variants:

```diff
- ZManaged.make(acquire)(release)
+ ZIO.acquireRelease(acquire)(release)
```

Example: 

```diff
- ZManaged.fromAutoCloseable(
-   zio.blocking.effectBlockingIO(new FileInputStream("file.txt")) 
- )
+ ZIO.fromAutoCloseable(
+   ZIO.attemptBlockingIO(new FileInputStream("file.txt")) 
+ )
```

4. Replace all usages of `ZLayer(resource)` or `resource.toLayer` with `ZLayer.scoped(resource)`, all references to `ZStream.managed(resource)` with `ZStream.scoped(resource)`, and so on for similar constructors:

```diff
- ZLayer {
-   resource // with type of ZManaged[R, E, A]
- }
+ ZLayer.scoped {
+   resource // with type of ZIO[R with Scope, E, A]
+ }

- resource.toLayer
+ ZLayer.scoped(resource)

- ZStream.managed(resource)
+ ZStream.scoped(resource)
```

5. Delete all uses of `toManaged_`:

```diff
- effect.toManaged_
+ effect 
```

6. Replace all uses of `toManaged(finalizer)` with `withFinalizer(finalizer)`:

```diff
- effect.toManaged(finalizer)
+ effect.withFinalizer(finalizer)
```

Example:

```diff
val effect: ZIO[Any, IOException, FileInputStream] = ???
- effect.toManaged(is => ZIO.succeed(is.close))
+ effect.withFinalizer(is => ZIO.succeed(is.close()))
```

Finally, let's try a full example of converting a ZManaged codebase to the Scoped one. Assume we have written the following `transfer` function in ZIO 1.x using `ZManaged`:

```scala

def close(resource: Closeable): URIO[Blocking, Unit] =
  effectBlockingIO(resource.close()).orDie

def is(file: String): ZManaged[Blocking, IOException, FileInputStream] =
  ZManaged.make(effectBlockingIO(new FileInputStream(file)))(close)

def os(file: String): ZManaged[Blocking, IOException, FileOutputStream] =
  ZManaged.make(effectBlockingIO(new FileOutputStream(file)))(close)

def copy(
    from: FileInputStream,
    to: FileOutputStream
): ZIO[Blocking, IOException, Unit] =
  effectBlockingIO {
    val buf = new Array[Byte](1024)
    var length = 0
    length = from.read(buf)

    while (length > 0) {
      to.write(buf, 0, length)
      length = from.read(buf)
    }
  }

def transfer(from: String, to: String): ZIO[Blocking, IOException, Unit] = {
  val resource = for {
    from <- is(from)
    to   <- os(to)
    _    <- copy(from, to).toManaged_
  } yield ()
  resource.useNow
}
```

As of ZIO 2.x, we should rewrite it as follows:

```scala

def close(resource: Closeable): UIO[Unit] =
  ZIO.attempt(resource.close()).orDie

def is(file: String): ZIO[Scope, IOException, FileInputStream] =
  ZIO.acquireRelease(ZIO.attemptBlockingIO(new FileInputStream(file)))(close)

def os(file: String): ZIO[Scope, IOException, FileOutputStream] =
  ZIO.acquireRelease(ZIO.attemptBlockingIO(new FileOutputStream(file)))(close)

def copy(
    from: FileInputStream,
    to: FileOutputStream
): IO[IOException, Unit] =
  ZIO.attemptBlockingIO(???)

def transfer(from: String, to: String): IO[Throwable, Unit] =
  ZIO.scoped {
    for {
      from <- is(from)
      to   <- os(to)
      _    <- copy(from, to)
    } yield ()
  }
```

As we can see, the migration is quite straightforward, and it doesn't require much extra work.

## Simplification of Concurrent Data Types

Even though highly polymorphic versions of ZIO concurrent data structures (e.g. `ZRef`, `ZQueue`) were elegant, they were used rarely. There was also some cost associated with polymorphism, such as errors, readability, and maintainability.

Therefore, we simplified these data structures by specializing them in their more monomorphic versions without significant loss of features:

| ZIO 1.x (removed)                    | ZIO 2.x               |
|--------------------------------------|-----------------------|
| `ZRef[+EA, +EB, -A, +B]`             | `Ref[A]`              |
| `ZTRef[+EA, +EB, -A, +B]`            | `TRef[A]`             |
| `ZRefM[-RA, -RB, +EA, +EB, -A, +B]`  | `Ref.Synchronized[A]` |
| `ZQueue[-RA, -RB, +EA, +EB, -A, +B]` | `Queue[A]`            |
| `ZHub[-RA, -RB, +EA, +EB, -A, +B]`   | `Hub[A]`              |

## Ref

ZIO 2.x unifies `Ref` and `RefM`. `RefM` becomes a subtype of `Ref` that has additional capabilities (i.e. the ability to perform effects within the operations) at some cost to performance:

| ZIO 1.x | ZIO 2.x             |
|---------|---------------------|
| `RefM`  | `Ref.Synchronized`  |

As the `RefM` is renamed to `Ref.Synchronized`; now the `Synchronized` is a subtype of `Ref`. This change allows a `Ref.Synchronized` to be used anywhere a `Ref` is currently being used.

To perform the migration, after renaming these types to the newer ones (e.g. `RefM` renamed to `Ref.Synchronized`) we should perform the following method renames:

| ZIO 1.x                 | ZIO 2.x                                |
|-------------------------|----------------------------------------|
| `RefM#dequeueRef`       | `zio.stream.SubscriptionRef#changes`   |
| `RefM#getAndUpdate`     | `Ref.Synchronized#getAndUpdateZIO`     |
| `RefM#getAndUpdateSome` | `Ref.Synchronized#getAndUpdateSomeZIO` |
| `RefM#modify`           | `Ref.Synchronized#modifyZIO`           |
| `RefM#modifySome`       | `Ref.Synchronized#modifySomeZIO`       |
| `RefM#update`           | `Ref.Synchronized#updateZIO`           |
| `RefM#updateAndGet`     | `Ref.Synchronized#updateAndGetZIO`     |
| `RefM#updateSome`       | `Ref.Synchronized#updateSomeZIO`       |
| `RefM#updateSomeAndGet` | `Ref.Synchronized#updateSomeAndGetZIO` |

## Semaphore and TSemaphore

There is a slight change on `TSemaphore#withPermit` method. In ZIO 2.x, instead of accepting `STM` values, it accepts only `ZIO` values and returns the `ZIO` value.

| `withPermit` | Input          | Output         |
|--------------|----------------|----------------|
| ZIO 1.x      | `STM[E, B]`    | `STM[E, B]`    |
| ZIO 2.x      | `ZIO[R, E, A]` | `ZIO[R, E, A]` |

## Queue

In ZIO 2.x, the `Queue` uses `Chunk` consistently with other ZIO APIs like ZIO Streams. This will avoid unnecessary conversions between collection types, particularly for streaming applications where streams use `Chunk` internally, but bulk take operations previously returned a `List` on `Queue`.

Here is a list of affected APIs: `takeAll`, `takeUpTo`, `takeBetween`, `takeN`, `unsafePollAll`, `unsafePollN`, and `unsafeOfferAll`. Let's see an example:

ZIO 1.x:

```scala
val taken: UIO[List[Int]] = for {
  queue <- Queue.bounded[Int](100)
  _     <- queue.offer(10)
  _     <- queue.offer(20)
  list  <- queue.takeUpTo(5)
} yield list
```

ZIO 2.x:

```scala
val taken: UIO[Chunk[Int]] = for {
  queue <- Queue.bounded[Int](100)
  _     <- queue.offer(10)
  _     <- queue.offer(20)
  chunk <- queue.takeUpTo(5)
} yield chunk
```

## ZIO Test

### ZSpec

The `ZSpec` data type has been renamed to `Spec`.

| ZIO 1.x | ZIO 2.x |
|---------|---------|
| `ZSpec` | `Spec`  |

So without any special effort, whenever we use `ZSpec` we should change it to `Spec`, e.g.:

```diff
- val myspec: ZSpec[Any, Nothing] =
+ val myspec: Spec[Any, Nothing] =
  test("my spec") {
    assertTrue(true)
  }
```

### Sharing Layers between Specs
In ZIO 1.x, in order to share costly layers between specs, you needed a "Master" spec that would invoke test code from individual Spec classes.
In ZIO 2.x, sharing layers is much simpler. 
Use `ZIOSpec[YourSharedType]` and plug your layer into the `bootstrap` field.

```scala

class SharedService()

object Layers {
  val sharedLayer = 
    ZLayer.succeed(new SharedService())
}

object UseSharedLayerA extends ZIOSpec[SharedService]{
  override def spec =
    test("use the shared layer in test A") {
      assertCompletes
    }

  override def bootstrap= Layers.sharedLayer
}

object UseSharedLayerB extends ZIOSpec[SharedService]{
  override def spec =
    test("use the shared layer in test B") {
      assertCompletes
    }

  override def bootstrap = Layers.sharedLayer
}
```
Then use the standard
```bash
sbt test
```
in order to run the tests.

Note - If you assign the results of a function call to `bootstrap`, like this:

```scala

class SharedService()

object LayerBuilder {
  def createLayer() = ZLayer.succeed(new SharedService())
}

object NotUsingSharedLayerA extends ZIOSpec[SharedService]{
  override def spec =
    test("use the shared layer in test A") {
      assertCompletes
    }

  override def bootstrap = LayerBuilder.createLayer()
}

object NotUsingSharedLayerB extends ZIOSpec[SharedService]{
  override def spec =
    test("use the shared layer in test B") {
      assertCompletes
    }

  override def bootstrap = LayerBuilder.createLayer()
}
```

These Specs will *not* share the same instance, as they are different references.

### Smart Constructors

By introducing smart constructors, we do not longer have the `testM` function to create effectful test suits. Instead, we should use the `test` function:

ZIO 1.x:

```scala
suite("Ref") {
  testM("updateAndGet") {
    val result = Ref.make(0).flatMap(_.updateAndGet(_ + 1))
    assertM(result)(Assertion.equalTo(1))
  }
}
```

ZIO 2.x:

```scala
suite("Ref") {
  test("updateAndGet") {
    val result = Ref.make(0).flatMap(_.updateAndGet(_ + 1))
    assertZIO(result)(Assertion.equalTo(1))
  }
}
```

In ZIO 2.x, when creating a test suite, it's not important whether we are testing pure or effectful code. `testM` was removed, and we can unifromly use `test` in both cases.

### Unification of `Assertion` and `AssertionM`

In ZIO 2.x, `Assertion` and `AssertionM` were unified to a single type, `Assertion`, so `AssertionM` was removed. In ZIO 2.x, instead of writing effectful assertions (`AssertionM`) and then asserting workflows, we should perform workflows and then simply assert the result of the workflow.

Assume we have written the following test in ZIO 1.x:

```scala
testM("Effectful Assertion ZIO 1.x") {
  def myEffectfulAssertion[Int](reference: Int): AssertionM[Int] = ???

  val sut = ZIO.effect(???)
  assertM(sut)(effectfulAssertion(5))
}
```

We can extract effectful operations from the effectful assertion and then perform the assertion like below:

```scala
test("Effectful Assertion ZIO 2.x") {
  def myAssertion[Int](reference: Int): Assertion[Int] = ???
  
  val res = for {
    sut <- ZIO.effect(???)
    res <- extractedOperations(sut)
  } yield assert(res)(myAssertion(5))
}
```

### Smart Assertion

ZIO 2.x introduced a new test method, named `assertTrue` which allows us to assert an expected behavior using ordinary Scala expressions that return `Boolean` values instead of specialized assertion operators.

So instead of writing following test assertions:

```scala
val list   = List(1, 2, 3, 4, 5)
val number = 3
val option = Option.empty[Int]

suite("ZIO 1.x Test Assertions")(
  test("contains")(assert(list)(Assertion.contains(5))),
  test("forall")(assert(list)(Assertion.forall(Assertion.assertion("even")(actual => actual % 2 == 0)))),
  test("less than")(assert(number)(Assertion.isLessThan(0))),
  test("isSome")(assert(option)(Assertion.equalTo(Some(3))))
)
```

We can write them like this:

```scala
suite("ZIO 2.x SmartAssertions")(
  test("contains")(assertTrue(list.contains(5))),
  test("forall")(assertTrue(list.forall(_ % 2 == 0))),
  test("less than")(assertTrue(number < 0)),
  test("isSome")(assertTrue(option.get == 3))
)
```

Smart Assertions are extremely expressive, so when a test fails:
- They highlight the exact section of the syntax with the path leading up to the left-hand side of the assertion that causes the failure.
- They have the strong and nice diffing capability which shows where our expectation varies.
- When using partial functions in test cases there is no problem with the happy path, but if something goes wrong, it is a little annoying to find what went wrong. But smart assertions are descriptive, e.g. when we call `Option#get` to an optional value that is `None` the test fails with a related error message: `Option was None`
- They have lots of domain specific errors that talk to us in a language that we understand.

### Compositional Specs

In ZIO 1.x, we cannot compose specs directly, although we can combine all children's specs via the suite itself:

```scala
val fooSuite = suite("Foo")(fooSpec)
val barSuite = suite("Bar")(barSpec)
val bazSuite = suite("Baz")(bazSpec)

val bigSuite = suite("big suite")(fooSuite, barSuite, bazSuite)
```

Now in ZIO 2.x, we can compose two suites using _binary composition operator_ without having to unnecessarily nest them inside another suite just for purpose of composition:

```scala
val bigSuite = fooSuite + barSuite + bazSuite
```

## ZIO Streams

ZIO Streams 2.x does not include any significant API changes. Almost the same code we have for ZIO Stream 1.x will continue working, so we don't need to relearn any APIs. Even though a good level of source compatibility has been maintained, some API elements have been changed.

So far, before ZIO 2.0, ZIO Stream has included three main abstractions:
1. **`ZStream`** — represents the source of elements
2. **`ZSink`** — represents consumers of elements that can be composed together to create composite consumers
3. **`ZTransducer`** — represents generalized stateful and effectful stream processing

![ZIO Streams 1.x](/img/assets/zio-streams-1.x.svg)

In ZIO 2.0, we added an underlying abstraction called `Channel`. Channels are underlying both `Stream` and `Sink`, so it's an abstraction that unifies everything in ZIO Streams.

![ZChannel](/img/assets/zio-streams-zchannel.svg)

Channels are nexuses of I/O operations, which support both reading and writing:

- A `Channel` can write some elements to the _output_, and it can terminate with some sort of _done_ value. The `Channel` uses this _done_ value to notify the downstream `Channel` that its emission of elements finished. In ZIO 2.x, `ZStream` is encoded as the output side of a `Channel`.

- A `Channel` can read from its input, and it can also terminate with some sort of _done_ value that is an upstream result. A `Channel` has an _input type_, and an _input done type_. The `Channel` uses the _done_ value to determine when the upstream `Channel` finishes its emission. In ZIO 2.x, `ZSink` is encoded as the input side of a `Channel`.

So we can say that streams are the output side and sinks are the input side of a `Channel`. What about the middle part? In ZIO 1.x, this used to be known as the `ZTransducer`. Transducers were great for writing high-performance codecs (e.g. compression). They were really just a specialization of sinks. We have added transducers because things were not sufficiently efficient using sinks. If we were to write streaming codecs using sinks, they could be quite slow.

In ZIO 2.x, we removed the transducers. Instead, we realized we need something else for the middle part, and now it's called a `Pipeline` in ZIO 2.x. Pipelines accept a stream as input and return the transformed stream as output.

![ZIO Streams 2.x](/img/assets/zio-streams-2.x.svg)

Pipelines are basically an abstraction for composing a bunch of operations together that can be later applied to a stream. For example, we can create a pipeline that reads bytes, decodes them to a UTF-8 string, splits that into lines, and then splits on commas. That is a very simple CSV parsing pipeline which we can later use by piping other streams into it. 

| ZIO Streams 1.x | ZIO Streams 2.x                  |
|-----------------|----------------------------------|
|                 | `ZChannel`                       |
| `ZStream`       | `ZStream` (backed by `ZChannel`) |
| `ZSink`         | `ZSink` (backed by `ZChannel`)   |
| `ZTransducer`   | `ZPipeline`                      |

## ZIO Schedules

`Schedule.jittered` in ZIO 1 is equivalent to `Schedule.jittered(0.0, 1.0)`. In ZIO 2 this changed to `Schedule.jittered(0.8, 1.2)`.

In ZIO 1 the average throughput of the updated schedule was twice the original schedule. In ZIO 2 the average throughput of the updated schedule is the same as the original schedule.

The new behavior is more in line with the definition of 'jitter' as used in signal processing.
Even so, [research](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) shows that `Schedule.jittered(0.0, 1.0)` is better for retrying.

## ZIO Services

There are two significant changes in ZIO Services:

1. All ZIO services moved to the `zio` package:

    | ZIO 1.x                 | ZIO 2.x                      |
    |-------------------------|------------------------------|
    | `zio.blocking.Blocking` | [Removed](#blocking-service) |
    | `zio.clock.Clock`       | `zio.Clock`                  |
    | `zio.console.Console`   | `zio.Console`                |
    | `zio.random.Random`     | `zio.Random`                 |
    | `zio.system.System`     | `zio.System`                 |

    And their live implementations renamed and moved to a new path:

    | ZIO 1.x                    | ZIO 2.x                   |
    |----------------------------|---------------------------|
    | `zio.Clock.Service.live`   | `zio.Clock.ClockLive`     |
    | `zio.Console.Service.live` | `zio.Console.ConsoleLive` |
    | `zio.System.Service.live`  | `zio.System.SystemLive`   |
    | `zio.Random.Service.live`  | `zio.Random.RandomLive`   |

2. In ZIO 2.0 all type aliases like `type Foo = Has[Foo.Service]` were removed. So when accessing services instead of `ZIO.service[Foo.Service]` we now do `ZIO.service[Foo]`.

### Blocking Service

Since there is rarely a need to use a separate blocking thread pool, ZIO 2.0 created _one global blocking pool_, and removed the Blocking service from `ZEnv` and the built-in services.

All blocking operations were moved to the `ZIO` data type:

| ZIO 1.x                   | ZIO 2.x |
|---------------------------|---------|
| `zio.blocking.Blocking.*` | `ZIO.*` |

With some renaming:

| ZIO 1.x (`zio.blocking.Blocking.*`) | ZIO 2.x (`ZIO.*`)               |
|-------------------------------------|---------------------------------|
| `effectBlocking`                    | `ZIO.attemptBlocking`           |
| `effectBlockingCancelable`          | `ZIO.attemptBlockingCancelable` |
| `effectBlockingIO`                  | `ZIO.attemptBlockingIO`         |
| `effectBlockingInterrupt`           | `ZIO.attemptBlockingInterrupt`  |

Now we have all the blocking operations under the `ZIO` data type as below:
- `ZIO.attemptBlocking`
- `ZIO.attemptBlockingCancelable`
- `ZIO.attemptBlockingIO`
- `ZIO.attemptBlockingInterrupt`
- `ZIO.blocking`
- `ZIO.blockingExecutor`

We can also provide a user-defined blocking executor in ZIO 2.x with the `Runtime#withBlockingExecutor` operator that constructs a new `Runtime` with the specified blocking executor.

### Clock Service

There is a slight change in the Clock service; the return value of the `currentDateTime`, and `localDateTime` methods changed from `IO` to `UIO`, so they do not longer throw `DateTimeException`:

| Method Name       | Return Type (ZIO 1.x) | Return Type (ZIO 2.x) |
|-------------------|-----------------------|-----------------------|
| `currentDateTime` | `IO[OffsetDateTime]`  | `UIO[OffsetDateTime]` |
| `localDateTime`   | `IO[LocalDateTime]`   | `UIO[LocalDateTime]`  |

In ZIO 2.0, without changing any API, the _retrying_, _repetition_, and _scheduling_ logic moved into the `Clock` service.

Working with these three time-related APIs, always made us require `Clock` as our environment. So by moving these primitives into the `Clock` service, now we can directly call them via the `Clock` service. This change solves a common anti-pattern in ZIO 1.0, whereby a middleware that uses `Clock` via this retrying, repetition, or scheduling logic must provide the `Clock` layer on every method invocation:

```scala
trait Journal {
  def append(log: String): ZIO[Any, Throwable, Unit]
}

trait Logger {
  def log(line: String): UIO[Unit]
}

case class JournalLoggerLive(clock: Clock, journal: Journal) extends Logger {
  override def log(line: String): UIO[Unit] = {
    for {
      current <- clock.currentDateTime
      _ <- journal.append(s"$current--$line")
        .retry(Schedule.exponential(2.seconds))
        .provideEnvironment(ZEnvironment(clock))
        .orDie
    } yield ()
  }
}
```

In ZIO 2.0, we can implement the logger service with a better ergonomic:

```scala
case class JournalLoggerLive(clock: Clock, journal: Journal) extends Logger {
  override def log(line: String): UIO[Unit] = {
    for {
      current <- clock.currentDateTime
      _       <- journal.append(s"$current--$line").retry(Schedule.exponential(2.seconds)).orDie
    } yield ()
  }
}
```

Note that the ZIO API didn't change, but the `Clock` trait became a bigger one, having more clock-related methods.

### Console Service

Method names in the _Console_ service were renamed to more readable names:

| ZIO 1.x       | ZIO 2.x          |
|---------------|------------------|
| `putStr`      | `print`          |
| `putStrErr`   | `printError`     |
| `putStrLn`    | `printLine`      |
| `putStrLnErr` | `printLineError` |
| `getStrLn`    | `readLine`       |

## Other New Features

### Smart Constructors

Every data type in ZIO (`ZIO`, `ZStream`, etc.) has a variety of constructor functions that are designed to _up convert_ some weaker type into the target type. Typically, these converter functions are named `fromXYZ`, e.g. `ZIO.fromEither`, `ZStream.fromZIO`, etc.

While these are precise, ZIO 2.0 provides the `ZIO.from` constructor which can intelligently choose the most likely constructor based on the input type. So instead of writing `ZIO.fromEither(Right(3))` we can easily write `ZIO.from(Right(3))`. Let's try some of them:

```scala

ZIO.fromOption(Some("Ok!"))
ZIO.from(Some("Ok!"))

ZIO.fromEither(Right(3))
ZIO.from(Right(3))

ZIO.fromFiber(Fiber.succeed("Ok!"))
ZIO.from(Fiber.succeed("Ok!"))

ZStream.fromIterable(List(1,2,3)) 
ZStream.from(List(1,1,3))

ZStream.fromChunk(Chunk(1,2,3))
ZStream.from(Chunk(1,2,3))

ZStream.fromIterableZIO(ZIO.succeed(List(1,2,3)))
ZStream.from(ZIO.succeed(List(1,2,3)))
```

### ZState

`ZState` is a new data type that ZIO 2.0 introduced:

```scala
sealed trait ZState[S] {
  def get: UIO[S]
  def set(s: S): UIO[Unit]
  def update(f: S => S): UIO[Unit]
}
```

We can `set`, `get`, and `update` the state which is part of the ZIO environment using `ZIO.setState`, `ZIO.getState`, and `ZIO.updateState` operations:

```scala

object ZStateExample extends zio.ZIOAppDefault {
  final case class MyState(counter: Int)

  val app = for {
    _     <- ZIO.updateState[MyState](state => state.copy(counter = state.counter + 1))
    count <- ZIO.getStateWith[MyState](_.counter)
    _     <- Console.printLine(count)
  } yield count

  def run = app.provide(ZState.initial(MyState(0)))
}
```

### Hub

`Hub` is a new concurrent data structure like `Queue`. While `Queue` solves the problem of _distributing_ messages to multiple consumers, `Hub` solves the problem of _broadcasting_ the same message to multiple consumers.

![Hub](/img/assets/hub.svg)

Here is an example of broadcasting messages to multiple subscribers:

```scala
for {
  hub <- Hub.bounded[String](requestedCapacity = 2)
  s1 = hub.subscribe
  s2 = hub.subscribe
  _ <- ZIO.scoped {
    s1.zip(s2).flatMap { case (left, right) =>
      for {
        _ <- hub.publish("Hello from a hub!")
        _ <- left.take.flatMap(Console.printLine(_))
        _ <- right.take.flatMap(Console.printLine(_))
      } yield ()
    }
  }
} yield ()
```

Visit the [Hub](../../reference/concurrency/hub.md) page to learn more about it.

### ZIO Aspects

We introduced `ZIOAspect` which enables us to modify the existing `ZIO` effect with some additional aspects like debugging, tracing, retrying, and logging:

```scala
val myApp: ZIO[Random, Nothing, String] =
  ZIO.ifZIO(
    Random.nextIntBounded(10) @@ ZIOAspect.debug map (_ % 2 == 0)
  )(
    onTrue = ZIO.succeed("Hello!"), 
    onFalse = ZIO.succeed("Good Bye!")) @@ ZIOAspect.debug @@ ZIOAspect.logged("result")
    
// Sample Output:     
// 2
// Hello!
// timestamp=2021-09-05T15:32:56.705901Z level=INFO thread=#2 message="result: Hello!" file=ZIOAspect.scala line=74 class=zio.ZIOAspect$$anon$4 method=apply
```

### Debugging

ZIO 2.x introduces the `debug` method that is useful when we want to print something to the console for debugging purposes without introducing additional environmental requirements or error types:

```scala
val myApp: ZIO[Random, Nothing, String] =
  ZIO
    .ifZIO(
      Random.nextIntBounded(10) debug("random") map (_ % 2 == 0)
    )(
      onTrue = ZIO.succeed("Hello!"),
      onFalse = ZIO.succeed("Good Bye!")
    )
    .debug("result")
// Sample Output
// random: 2
// result: Hello!
``` 

### Logging

ZIO 2.x supports a lightweight built-in logging facade that standardizes the interface for logging functionality. So it doesn't replace existing logging libraries, but also we can plug it into one of the existing logging backends.

We can easily log using the `ZIO.log` function:

```scala
ZIO.log("Application started!")
```

To log with a specific log-level, we can use the `ZIO.logLevel` combinator:

```scala
ZIO.logLevel(LogLevel.Warning) {
  ZIO.log("The response time exceeded its threshold!")
}
```
Or we can use the following functions directly:

* `ZIO.logDebug`
* `ZIO.logError`
* `ZIO.logFatal`
* `ZIO.logInfo`
* `ZIO.logWarning`

```scala
ZIO.logError("File does not exist: ~/var/www/favicon.ico")
```

It also supports logging spans:

```scala
ZIO.logSpan("myspan") {
  ZIO.sleep(1.second) *> ZIO.log("The job is finished!")
}
```

ZIO Logging calculates the running duration of that span, and includes that in the logging data associating to its span label.

### Compile-time Execution Tracing

ZIO 1.x's execution trace is not as useful as it could be, because it contains tracing information for internal ZIO operators that is not very helpful, when trying to understand, where an error occurred.

Let's say we have the following application in ZIO 1.x:

```scala

object TracingExample extends zio.App {

  def doSomething(input: Int): ZIO[Console, String, Unit] =
    for {
      _ <- console.putStrLn(s"Do something $input").orDie // line number 8
      _ <- ZIO.fail("Boom!")
      _ <- console.putStrLn("Finished my job").orDie
    } yield ()

  def myApp: ZIO[Console, String, Unit] =
    for {
      _ <- console.putStrLn("Hello!").orDie
      _ <- doSomething(5)
      _ <- console.putStrLn("Bye Bye!").orDie
    } yield ()

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.exitCode
}
```

The output would be something like this:

```
Hello!
Do something 5
Fiber failed.
A checked error was not handled.
Boom!

Fiber:Id(1634884059941,1) was supposed to continue to:
  a future continuation at TracingExample$.myApp(TracingExample.scala:16)
  a future continuation at zio.ZIO.exitCode(ZIO.scala:606)

Fiber:Id(1634884059941,1) execution trace:
  at TracingExample$.doSomething(TracingExample.scala:8)
  at zio.ZIO.orDieWith(ZIO.scala:1118)
  at zio.ZIO.refineOrDieWith(ZIO.scala:1497)
  at zio.console.package$Console$Service$.putStrLn(package.scala:44)
  at zio.console.package$.putStrLn(package.scala:88)
  at TracingExample$.myApp(TracingExample.scala:15)
  at zio.ZIO.orDieWith(ZIO.scala:1118)
  at zio.ZIO.refineOrDieWith(ZIO.scala:1497)
  at zio.console.package$Console$Service$.putStrLn(package.scala:44)
  at zio.console.package$.putStrLn(package.scala:88)

Fiber:Id(1634884059941,1) was spawned by:

Fiber:Id(1634884059516,0) was supposed to continue to:
  a future continuation at zio.App.main(App.scala:59)
  a future continuation at zio.App.main(App.scala:58)

Fiber:Id(1634884059516,0) ZIO Execution trace: <empty trace>

Fiber:Id(1634884059516,0) was spawned by: <empty trace>
```

The execution trace is informative, but it doesn't lead us to the exact point, where the failure happened. It's a little hard to see what is going on here. 

Let's rewrite the previous example in ZIO 2.x:

```scala

object TracingExample extends ZIOAppDefault {

  def doSomething(input: Int): ZIO[Any, String, Unit] =
    for {
      _ <- Console.printLine(s"Do something $input").orDie
      _ <- ZIO.fail("Boom!") // line number 8
      _ <- Console.printLine("Finished my job").orDie
    } yield ()

  def myApp: ZIO[Any, String, Unit] =
    for {
      _ <- Console.printLine("Hello!").orDie
      _ <- doSomething(5)   // line number 15
      _ <- Console.printLine("Bye Bye!").orDie
    } yield ()

  def run = myApp
}
```

The output is more descriptive than in ZIO 1.x. It is similar to a Java stacktrace:

```
Hello!
Do something 5
timestamp=2021-12-19T08:25:09.372926403Z level=ERROR thread=#zio-fiber-1639902309 message="Exception in thread "zio-fiber-1639902309" java.lang.String: Boom!
	at zio.examples.TracingExample.doSomething(TracingExample.scala:8)
	at zio.examples.TracingExample.myApp(TracingExample.scala:15)"
```

As we see, the first line of the execution trace points to the exact location in the source code which causes the failure (`ZIO.fail("Boom!")`) that is line number 8.

In ZIO 2.x, tracing is not optional, and unlike in ZIO 1.x, it is impossible to disable async tracing, either globally, or for specific effects. ZIO now always generates async stack traces, and it is impossible to turn this feature off, either at the global level or at the level of individual effects. Since nearly all users were running ZIO with tracing turned on, this change should have minimal impact on ZIO applications.

Another improvement about ZIO tracing is its performance. Tracing in ZIO 1.x slows down the application performance by two times. In ZIO 1.x, we wrap and unwrap every combinator at runtime to be able to trace the execution. While it is happening on the runtime, it takes a lot of allocations which all need to be garbage collected afterwards adding a huge amount of complexity at runtime.

Some users turned off tracing to achieve better performance, but that results in losing the ability to trace the application, when something breaks.

In ZIO 2.x, we moved execution tracing from the runtime to the compile-time. This is done by capturing tracing information from the source code at compile time using macros. Most tracing information is pre-allocated at startup, and never needs to be garbage collected. As a result, we end up with much better performance in execution tracing.

---

## ZIO Quickstart: Building GraphQL Web Service

This quickstart shows how to build a GraphQL web service using ZIO. It uses

- [ZIO HTTP](https://ziohttp.com/) for the HTTP server
- [Caliban](https://ghostdogpr.github.io/caliban/) for the GraphQL

Creating GraphQL API using Caliban is an easy process. We can define our data models using case classes and ADTs. Then Caliban can derive the whole GraphQL schema from these data models.

## Running The Example

First, open the console and clone the [ZIO Quickstarts](https://github.com/zio/zio-quickstarts) project using `git` (or you can simply download the project) and then change the directory:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-graphql-webservice
```

Once you are inside the project directory, run the application:

```bash
sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Testing The Quickstart

In this project, we have defined models of our employees with their names and roles. Then using GraphQL annotations, we asked Caliban to derive the GraphQL schema from these models.

So we can query all the employees that are software developers using the GraphQL query:

```graphql
query {
  employees(role: SoftwareDeveloper) {
    name
    role
  }
}
```

To run this query, we can use any of the GraphGL clients or use the following curl command:

```bash
curl 'http://localhost:8088/api/graphql' --data-binary '{"query":"query{\n employees(role: SoftwareDeveloper){\n name\n role\n}\n}"}'
```

The response will be as follows:

```json
{
  "data": {
    "employees": [
      {
        "name": "Maria",
        "role": "SoftwareDeveloper"
      },
      {
        "name": "Peter",
        "role": "SoftwareDeveloper"
      }
    ]
  }
}
```

---

## ZIO Quickstart: Hello World

This is the simplest quickstart for a ZIO application. You can download and run it very quickly. This will give you an idea of how to write similar apps.

## Running The Example

First, open the console and clone the [ZIO Quickstarts](https://github.com/zio/zio-quickstarts) project using `git` (or you can simply download the project) and then change the directory:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-hello-world
```

Once you are inside the project directory, run the application:

```bash
sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Testing The Quickstart

When we prompt the `sbt run` the sbt search for the executable class and will find the `zio.dev.quickstart.MainApp` which contains the following code:

```scala

object MainApp extends ZIOAppDefault {
  def run = Console.printLine("Hello, World!")
}
```

The `MainApp` is the main object of our application which extends the `ZIOAppDefault` trait. This trait provides the `main` function which can be invoked from IDEs or the command-line, e.g. the `sbt run`.

This example is the simplest, example we can run. Let's change it so that we ask the name of the user and print a hello along with the user's name:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      _    <- Console.print("Please enter your name: ")
      name <- Console.readLine
      _    <- Console.printLine(s"Hello, $name!")
    } yield ()
}
```

In this example, we used `for-comprehension` to compose `print`, `readLine` and `printLine` workflows. We can think of it as the following `flatMap` operations:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    Console.print("Please enter your name: ")
      .flatMap { _ =>
        Console.readLine
          .flatMap { name =>
            Console.printLine(s"Hello, $name!")
          }
      }
}
```

---

## ZIO Quickstart: Building RESTful Web Service

This quickstart shows how to build a RESTful web service using ZIO. It uses

- [ZIO HTTP](https://ziohttp.com/) for the HTTP server
- [ZIO JSON](https://zio.dev/zio-schema/) for schema generation and JSON serialization
- [ZIO Quill](https://zio.dev/zio-quill/) for type-safe queries on the JDBC database

## Running The Example

First, open the console and clone the [zio-quickstarts](https://github.com/zio/zio-quickstarts) project using `git` (or you can simply download the project) and then change the directory:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-restful-webservice
```

Once you are inside the project directory, run the application:

```bash
sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Testing The Quickstart

In this quickstart, we will build a RESTful web service that has the following Http apps:

- **Greeting App**— shows how to write a basic Http App.
- **Download App**— shows how to work with files, headers, and status codes and also streaming data.
- **Counter App**— shows how to have a stateful web service and how to use the ZIO environment for Http Apps.
- **User App**— shows how to have a stateful web service to register and manage users.

The most important part of this quickstart is learning how to build an `Route` data type that is used to build the HTTP apps.

It is a data type that models a route in an HTTP application, just like the `ZIO` data type that models ZIO workflows.

We can say that `Route[Env, Err]` is a function that takes a `Request` and returns a `ZIO[Env, Err, Response]` if it matches to the providing route pattern. To put it another way, `HTTP[R, E, A, B]` is an HTTP application that:

- Accepts an `Request` and returns `Response`
- Uses the `Env` from the environment
- Will fail with `Err` if there is an error

We can write many routes and then serve them using the `Server.serve` method.

### 1. Greeting App

The Greeting App is a simple Http App that returns a greeting message. First, let's see how this app is defined:

```scala
object GreetingRoutes {
  def apply(): Routes[Any, Nothing] = ???
}
```

Routes in the `GreetingRoutes` don't require any services from the environment (`Any`) and won't fail (`Nothing`).

It has three routes, and we are going to test them one by one:

1. When we build and run this quickstart, there is a greeting app that we can access using the following endpoint:

```bash
GET http://localhost:8080/greet
```

Let's try to access this endpoint using curl and see what we get:

```bash
user@host ~> curl -i localhost:8080/greet
HTTP/1.1 200 OK
content-type: text/plain
content-length: 12

Hello World!⏎
```

2. We have another endpoint that pattern matches the `/greet/:name` request:

```bash
GET http://localhost:8080/greet/:name
```

Using this endpoint, we can greet a user by its name:

```bash
user@host ~> curl -i http://localhost:8080/greet/John
HTTP/1.1 200 OK
content-type: text/plain
content-length: 10

Hello John!⏎
```

3. Finally, we have a third endpoint that extracts the names from the query parameters:

```bash
GET http://localhost:8080/greet?name=:name
```

Let's try to request this endpoint and see what we get:

```bash
user@host ~> curl -i "http://localhost:8080/greet?name=John"
HTTP/1.1 200 OK
content-type: text/plain
content-length: 11

Hello John!⏎
```

It also works for more than one query params:

```bash
user@host ~> curl -i "http://localhost:8080/greet?name=Jane&name=John"
HTTP/1.1 200 OK
content-type: text/plain
content-length: 21

Hello Jane and John!⏎
```

### 2. Download App

The next example shows how to download a file from the server. First, let's look at the type of the `DownloadRoutes`:

```scala
object DownloadRoutes {
  def apply(): Routes[Any, Nothing] = ???
}
```

It is a `Routes` that doesn't require any environment, and won't fail. Let's try to access this endpoint using curl and see what we get:

1. Our first endpoint is `/download` which downloads a file from the server:

```bash
GET http://localhost:8080/download
```

If we try to request this endpoint using curl, we will see the following output:

```bash
user@host ~> curl -i http://localhost:8080/download
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=file.txt
transfer-encoding: chunked

line number 1
1, 2, 3, 4, 5
line number 3
end of file
```

Also, if we try to access this URL from the browser, the browser will prompt us to download the file with `file.txt` as the name.

2. The second endpoint is an example of downloading a big file when we want to stream the chunks of the file to the client:

```bash
GET http://localhost:8080/download/stream
```

When we try to access this endpoint using curl, we will see the following output:

```bash
curl -i http://localhost:8080/download/stream
HTTP/1.1 200 OK
transfer-encoding: chunked

1
2
3
...
```

We have scheduled some delays between each line to simulate downloading a big file. So when we run the above `curl` command, we can see that the content of the file will be downloaded gradually.

### 3. Counter App

The next example shows how we can have a stateful web service. Let's look at the type of the `counterApp`:

```scala
object CounterRoutes {
  def apply(): Routes[Ref[Int], Nothing] = ???
}
```

This is a `Routes` that requires a `Ref[Int]` as an environment, it cannot fail. This counter increments every time we access the `/up` endpoint and decrements every time we access the `/down` endpoint:

```bash
GET http://localhost:8080/up
Get http://localhost:8080/down
```

Let's try to access the `up` endpoint 100 times and then access the `down` endpoint 25 times:

```bash
user@host ~> for i in {1..100}; do curl http://localhost:8080/up; echo -n ' '; done;
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
user@host ~> for i in {1..25}; do curl http://localhost:8080/down; echo -n ' '; done;
99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75
```

We can see that the state of the counter is maintained between requests. In this example, we used the ZIO environment to store the access and store the state of the counter.

### 4. User App

The `UserRoutes` is a `Routes` with the following definition:

```scala
object UserRoutes {
  def apply(): Routes[UserRepo, Response] = ???
}
```

It requires a `UserRepo` service from the ZIO environment, it can fail with `Response`. In this example, we use the in-memory version of the `UserRepo` service called `InmemoryUserRepo`.

:::note
Please note that the `Routes` errors of type `Response` are handled with a proper response that displays an error message to the end user.
:::

This app has three endpoints:

```bash
POST http://localhost:8080/users -d '{"name": "John", "age": 30}'
GET  http://localhost:8080/users
GET  http://localhost:8080/users/:id
```

Let's try to register a new user:

```bash
user@host ~> curl -i http://localhost:8080/users -d '{"name": "John", "age": 35}'
HTTP/1.1 200 OK
content-type: text/plain
content-length: 36

f0f319ea-404d-4a55-abd0-41bee4ce887e
```

Now, we can get any registered user by its id:

```bash
user@host ~> curl -i http://localhost:8080/users/f0f319ea-404d-4a55-abd0-41bee4ce887e
HTTP/1.1 200 OK
content-type: application/json
content-length: 24

{"name":"John","age":35}
```

While this app is stateful, it is not persistent. We just provided the in-memory version of the `UserRepo` service called `InmemoryUserRepo`:

```scala
Server.serve(
  GreetingRoutes() ++ DownloadRoutes() ++ CounterRoutes() ++ UserRoutes()
).provide(
  Server.defaultWithPort(8080),
  ZLayer.fromZIO(Ref.make(0)),
  InmemoryUserRepo.layer
)
```

To make it persistent, we can provide the `PersistentUserRepo` service instead:

```scala
Server.serve(
  GreetingRoutes() ++ DownloadRoutes() ++ CounterRoutes() ++ UserRoutes()
).provide(
  Server.defaultWithPort(8080),
  ZLayer.fromZIO(Ref.make(0)),
  PersistentUserRepo.layer
)
```

Now, if we register a new user, the user will be persisted and if the application is restarted, the user will be available.

---

## Tutorial: How to Build a GraphQL Web Service

Having GraphQL APIs enables the clients the ability to query the exact data they need. This powerful feature makes GraphQL more flexible than RESTful APIs.

Instead of having endpoints for our resources, the GraphQL API only provides a set of types and fields in terms of schemas. The client can ask for this schema, and that will help the client to know what kind of data they can expect from the server, and finally, the client can use the schema to build its queries.

GraphQL supports three types of operations: **queries**, **mutations**, and **subscriptions**. In this tutorial, we are going to learn the foundation of GraphQL using _queries_. Once we learned that, we can easily write two other types of operations.

The corresponding source code for this tutorial is available on [GitHub](https://github.com/zio/zio-quickstarts). If you haven't read the [ZIO Quickstart: Building GraphQL Web Service](../quickstarts/graphql-webservice.md) yet, we recommend you to read it first and download and run the source code, before reading this tutorial.

## Installation

In this tutorial, we will build a GraphQL API using _Caliban_, and in order to serve it, we will use the _ZIO HTTP_ library. So let's install the necessary dependencies by putting the following lines in the _build.sbt_ file:

```sbt
libraryDependencies ++= Seq(
  "dev.zio"               %% "zio"              % "2.1.7",
  "com.github.ghostdogpr" %% "caliban-quick"    % "2.8.1"
)
```

Now, we are ready to jump into the next section.

## Creating a GraphQL Schema

When working with GraphQL, we need to define a _schema_, which defines what kind of data with which types we can query. In GraphQL, schemas are defined in their type system. For example, assume we have a data type called _Employee_, which had defined in scala as the following:

```scala
case class Employee(name: String, role: Role)

sealed trait Role
object Role {
  case object SoftwareDeveloper       extends Role
  case object SiteReliabilityEngineer extends Role
  case object DevOps                  extends Role
}
```

It has two fields, _name_, and _role_, and the _role_ field is a _sealed trait_, which means that it can only be one of the three values, _SoftwareDeveloper_, _SiteReliabilityEngineer_, or _DevOps_.

If we want to define such a data type in GraphQL, we have something like this:

```graphql
type Employee {
  name: String!
  role: Role!
}

enum Role {
  SoftwareDeveloper
  SiteReliabilityEngineer
  DevOps
}
```

After defining the schema for our data types, the next step is to define queries that can be performed on the data. For example, if we want to query all the employees with a specific role, we can do that like this:

```graphql
type Queries {
  employees(role: Role!): [Employee!]!
}
```

Similarly, if we want to query an employee by its name, we can define that like this:

```graphql
type Queries {
  employee(name: String!): Employee
}
```

Fortunately, we are not required to define the schema in GraphQL manually. Instead, we can use the _Caliban_ library which has a built-in facility to generate the schema from our data types defined in Scala:

```scala
case class EmployeesArgs(role: Role)
case class EmployeeArgs(name: String)

case class Queries(
    employees: EmployeesArgs => List[Employee],
    employee: EmployeeArgs => Option[Employee]
)
```

## Running the GraphQL Server

After defining all the queries, in order to serve the GraphQL API, we need to perform the following steps:

1. Create a `GraphQLInterpreter` instance, which is a wrapper around the _GraphQL API_. It allows us to add some middleware around the query execution.
2. Create an `HttpApp` instance from the `GraphQLInterpreter` instance. We can do this by using the `ZHttpAdapter.makeHttpService` defined in the `caliban-zio-http` module.
3. Serve the resulting `HttpApp` instance using the `Server.start` method of the _ZIO HTTP_ module.

```scala

object MainApp extends zio.ZIOAppDefault {

  private val employees = List(
    Employee("Alex", Role.DevOps),
    Employee("Maria", Role.SoftwareDeveloper),
    Employee("James", Role.SiteReliabilityEngineer),
    Employee("Peter", Role.SoftwareDeveloper),
    Employee("Julia", Role.SiteReliabilityEngineer),
    Employee("Roberta", Role.DevOps)
  )

  override def run =
    graphQL(
      RootResolver(
        Queries(
          args => employees.filter(e => args.role == e.role),
          args => employees.find(e => e.name == args.name)
        )
      )
    ).runServer(
      port = 8088,
      apiPath = "/api/graphql",
      graphiqlPath = Some("/api/graphiql"),
    )

}
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Effectful Queries

In the previous section, we used an in-memory data structure to store the data. But, in real-world applications we usually want to perform some kind of effectful queries to retrieve the data from the database. In such cases, we can use queries that return `ZIO` values:

```diff
case class Queries(
-    employees: EmployeesArgs => List[Employee],
+    employees: EmployeesArgs => ZIO[UserRepo, Throwable, List[Employee]],
-    employee: EmployeeArgs => Option[Employee]
_    employee: EmployeeArgs => ZIO[UserRepo, Throwable, Option[Employee]]
)
```

As we see, each query is a function that takes some arguments and returns a `ZIO` workflow.

## Running the GraphQL Client

In this project, we have defined models of our employees with their names and roles. Then using GraphQL annotations, we asked Caliban to derive the GraphQL schema from these models.

So we can query all the employees that are software developers using the GraphQL query:

```graphql
query {
  employees(role: SoftwareDeveloper) {
    name
    role
  }
}
```

To run this query, we can use any of the GraphGL clients or use the following curl command:

```bash
curl 'http://localhost:8088/api/graphql' --data-binary '{"query":"query{\n employees(role: SoftwareDeveloper){\n name\n role\n}\n}"}'
```

The response will be as follows:

```json
{
  "data": {
    "employees": [
      {
        "name": "Maria",
        "role": "SoftwareDeveloper"
      },
      {
        "name": "Peter",
        "role": "SoftwareDeveloper"
      }
    ]
  }
}
```

## Conclusion

In this tutorial, we learned the basic elements of writing GraphQL web services, using the _Caliban_ library. Caliban has great documentation, which can be found [here](https://ghostdogpr.github.io/caliban/). We can learn more about this project by visiting its website.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) project.

---

## Tutorial: How to Build a RESTful Web Service

ZIO provides good support for building RESTful web services. Using _Service Pattern_, we can build web services that are modular and easy to test and maintain. On the other hand, we have several powerful official and community libraries that can help us to work with JSON data types, and databases and also work with HTTP protocol.

In this tutorial, we will learn how to build a RESTful web service using ZIO. The corresponding source code for this tutorial is available on [GitHub](https://github.com/zio/zio-quickstarts). If you haven't read the [ZIO Quickstart: Building RESTful Web Service](../quickstarts/restful-webservice.md) yet, we recommend you read it first and download and run the source code, before reading this tutorial.

## Installation

We need to add the following dependencies to our project:

```scala
libraryDependencies ++= Seq(
  "dev.zio"  %% "zio-http" % "3.0.0-RC6"
)
```

For this tutorial, we will be using the _[ZIO HTTP](https://ziohttp.com/)_ library, which is a library for building HTTP applications using ZIO.

## Introduction to The `Route` Data Type

Before we start to build a RESTful web service, we need to understand the `Route` data type. It is a data type that models a route in an HTTP application.

We can think of the `Route[Env, Err]` as a description of an HTTP route that accepts a `Request` that matches the route pattern and returns a `Response`. It can use the environment of type `Env` and may fail with an error of type `Err`.

A simple `Route` can be defined as follows:

```scala

val helloRoute: Route[Any, Nothing] =
  Method.GET / "hello" -> handler(Response.text("Hello, world!"))
```

We can say that `Route[Any, Nothing]` is a function that takes a `Request` and returns a `Response`. It doesn't require any services from the environment and won't fail.

Having multiple routes, we can collect them into a single `Routes` data type:

```scala

val routes: Routes[Any, Nothing] =
  Routes(
    Method.GET / "hello" -> handler(Response.text("Hello, world!")),
    Method.GET / "greet" -> handler(Response.text("Hello, ZIO!"))
  )
```

Finally, we can serve the routes using the `Server.serve` method:

```scala

object MainApp extends ZIOAppDefault {
  def run = Server.serve(routes).provide(Server.default)
}
```

## Modeling Http Applications

Let's try to model some HTTP applications using the `Route` data type. So first, we are going to learn some basic `Route` constructors and how to combine them to build more complex HTTP applications.

### Creation of a `Handler`

The `Handler.succeed` constructor creates a `Handler` that always returns a successful response:

```scala

val app: Handler[Any, Nothing, Any, Response] =
  Handler.succeed(Response.text("Hello, world!"))
```

We have the same constructor for failures called `Http.fail`. It creates a `Handler` application that always returns a failed response:

```scala

val app: Handler[Any, Response, Any, Nothing] =
  handler(ZIO.fail(Response.internalServerError("Something went wrong")))
```

We can also create a `Handler` form a function. The `Handler.fromFunction` constructor takes a total function of type `A => B` and then creates a `Handler` that accepts an `A` and returns a `B`:

```scala

val app: Handler[Any, Nothing, Int, Double] = Handler.fromFunction[Int](_ / 2.0)
```

Handlers can be effectual. We have a couple of constructors that can be used to create a `Handler` that are effectual:

- `Http.fromZIO`
- `Http.fromStream`
- `Http.fromFunctionZIO`
- `Http.fromFile`

There are lots of other constructors, to learn more about them, please refer to the [`Handler`](https://ziohttp.com/reference/handler) page in the ZIO HTTP documentation.

### Combining Handlers

The `Handler` data type is composable like `ZIO`. We can create new complex `Handler` by combining existing simple ones by using `flatMap`, `zip`, `andThen`, `orElse`, and `++` methods:

```scala

val a           : Handler[Any, Nothing, Int, Double]    = ???
val b           : Handler[Any, Nothing, Double, String] = ???
def c(i: Double): Handler[Any, Nothing, Long, String]   = ???

val d = a >>= c // a flatMap c (combine two handlers sequentially)
val f = a >>> b // a andThen b (pipe output of the a handler to input of the b handler)
val h = a <> b  // a orElse b  (run a, if it fails, run b)
```

## Built-in `Request` and `Response` Data Types

Until now, we have learned how to create a `Handler` with some simple request and response types, e.g. `String` and `Int` in a `Handler[Any, Nothing, String, Int]`. But, in real life, when we want to deal with HTTP requests and responses, we need to have a more complex type for the request and response.

ZIO HTTP provides a type `Request` for HTTP requests and a type `Response` for HTTP responses. It has a built-in decoder for `Request` and an encoder for `Response`. So we don't need to worry about the details of how requests and responses are decoded and encoded.

The `Response` type has a default `apply` constructor in its companion object that takes a status, headers, and, HTTP data to create a `Response`:

```scala
object Response {
  def apply[R, E](
    status: Status = Status.Ok,
    headers: Headers = Headers.empty,
    data: HttpData = HttpData.Empty
  ): Response = ???
}
```

Other than the default constructor, we have several helper methods to create a `Response`. Here are some of them:

1. **`Response.ok`**: Creates a successful response with 200 status code.
2. **`Response.text("Hello World")`**: Creates a successful response with 200 status code and a body of `Hello World`.
3. **`Response.status(Status.BadRequest)`**: Creates a response with a status code of 400.
4. **`Response.html("Hello World")`**: Creates a successful response with 200 status code and an HTML body of `Hello World`.
5. **`Response.redirect("/")`**: Creates a successful response that redirects to the root path.

On the other hand, we do not need to create a `Request` instead, we need to pattern-match incoming requests to decompose them and determine the appropriate action to take.

Each incoming request can be extracted into two parts using pattern matching:

- HTTP Method (GET, POST, PUT, etc.)
- Path (e.g. /, /greeting, /download)

Let's see an example of how to pattern match on incoming requests:

```scala

val httpApp: Route[Any, Nothing] =
  Method.GET / "greet" / string("name") ->
    handler { (name: String, _: Request) =>
      Response.text(s"Hello $name!")
    }
```

Using this DSL we only access the method and path of the incoming request. If we need to access the query string, the body, and more, we need to use the following DSL:

```scala

val httpApp: Route[Any, Response] =
  Method.GET / "greet" ->
    handler { (req: Request) =>
      if (req.url.queryParams.nonEmpty)
        ZIO.succeed(Response.text(s"Hello ${req.url.queryParams("name").mkString(" and ")}!"))
      else
        ZIO.fail(Response.badRequest("Missing query parameter 'name'"))
    }
```

Until now, we have learned how to create `Route` and `Routes` applications that handle HTTP requests. In the next section, we will learn how to create HTTP servers that can serve HTTP routes.

## Creating HTTP Server

To start an HTTP server, the ZIO HTTP requires a `Routes` of type `Routes[Env, Response]` and returns an effect that requires a `Server` from the environment and never produces a value and never fails:

```scala
object Server {
  def server[R](
    http: Routes[Env, Response]
  ): ZIO[R with Server, Nothing, Nothing] = ???
}
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Greeting App

First, we need to define a request handler that will handle `GET` requests to the `/greet` path:

```scala

object GreetingRoutes {
  def apply(): Routes[Any, Response] =
    Routes(
      // GET /greet?name=:name
      Method.GET / "greet" -> handler { (req: Request) =>
        if (req.url.queryParams.nonEmpty)
          ZIO.succeed(
            Response.text(
              s"Hello ${req.url.queryParams("name").map(_.mkString(" and "))}!"
            )
          )
        else
          ZIO.fail(Response.badRequest("The name query parameter is missing!"))
      },

      // GET /greet
      Method.GET / "greet" -> handler(Response.text(s"Hello World!")),

      // GET /greet/:name
      Method.GET / "greet" / string("name") -> handler {
        (name: String, _: Request) =>
          Response.text(s"Hello $name!")
      }
    )

}
```

In the above example, we have defined three routes:

- The first case matches a request with a path of `/greet` and a query parameter `name`.
- The second case matches a request with a path of `/greet` with no query parameters.
- The third case matches a request with a path of `/greet/:name` and extracts the `name` from the path.

Next, we need to create a server for `GreetingRoutes`:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    Server.serve(GreetingRoutes()).provide(Server.default)
}
```

Now, we have three endpoints in our server. We can test the server according to the steps mentioned in the corresponding [quickstart](../quickstarts/restful-webservice.md).

Note that if we have written other routes along with `GreetingRoutes`, such as `DownloadRoutes`, `CounterRoutes`, and `UserRoutes`, we can combine them together and start a server for that routes:

```scala

Server.serve(
  GreetingRoutes() ++ DownloadRoutes() ++ CounterRoutes() ++ UserRoutes()
).provide(Server.default)
```

## Conclusion

In this tutorial, we have learned the basic building blocks of writing HTTP servers. We learned how to write routes and handlers. And finally, we saw how to create an HTTP server that can handle HTTP applications.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) project.

---

## Tutorial: How to Create a Custom Logger for a ZIO Application?

## Introduction

As we have seen in the [previous tutorial](enable-logging-in-a-zio-application.md), ZIO has a variety of built-in logging facilities. Also, it has a default logger that can be used to print log messages to the console. When we go to production, we may want to use a different logger with a customized configuration. For example, we may want to log to a file or a database instead of the console.

In this tutorial, we are going to see how we can create a custom logger for a ZIO application.

## Running the Examples

In [this article](enable-logging-in-a-zio-application.md), we enabled logging for `UserApp` http application. In this tutorial, we are going to create a custom logger for the `UserApp`.

To run the code, clone the repository and checkout the [ZIO Quickstarts](http://github.com/zio/zio-quickstarts) project:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-restful-webservice-custom-logger
```

And finally, run the application using sbt:

```bash
$ sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Creating a Custom Logger

To create a new logger for the ZIO application, we need to create a new `ZLogger` object. The `ZLogger` is a trait that defines the interface for a ZIO logger. The default logger has implemented this trait through the `ZLogger.default` object.

```scala

val logger: ZLogger[String, Unit] =
  new ZLogger[String, Unit] {
    override def apply(
      trace: Trace,
      fiberId: FiberId,
      logLevel: LogLevel,
      message: () => String,
      cause: Cause[Any],
      context: FiberRefs,
      spans: List[LogSpan],
      annotations: Map[String, String]
    ): Unit =
      println(s"${java.time.Instant.now()} - ${logLevel.label} - ${message()}")
  }
```

So then, we can remove all the default loggers and replace them with our custom logger:

```scala

object MainApp extends ZIOAppDefault {

  val logger: ZLogger[String, Unit] =
    new ZLogger[String, Unit] {
      override def apply(
        trace: Trace,
        fiberId: FiberId,
        logLevel: LogLevel,
        message: () => String,
        cause: Cause[Any],
        context: FiberRefs,
        spans: List[LogSpan],
        annotations: Map[String, String]
      ): Unit =
        println(s"${java.time.Instant.now()} - ${logLevel.label} - ${message()}")
    }

  override val bootstrap = Runtime.removeDefaultLoggers ++ Runtime.addLogger(logger)

  def run =
    for {
      _ <- ZIO.log("Application started!")
      _ <- ZIO.log("Another log message.")
      _ <- ZIO.log("Application stopped!")
    } yield ()
}
```

By running this application, the log messages will be printed like this:

```bash
2022-06-04T13:49:19.554648Z - INFO - Application started!
2022-06-04T13:49:19.567854Z - INFO - Another log message.
2022-06-04T13:49:19.568831Z - INFO - Application stopped!
```

## Using SLF4J Logger in a ZIO Application

So far, we learned how to write a custom logger for a ZIO application. Now, in this section, we want to add SLF4J Logging support to the `UserApp` we have developed in the [Restful Web Service](../quickstarts/restful-webservice.md) quickstart.

SLF4J is a logging facade that decouples our application code from any underlying logging implementation. To enable SLF4J logging for a ZIO application, we need to implement the `ZLogger` trait using SLF4J. Fortunately, the ZIO Logging project has done this for us.

So we can simply add this library to our project by adding the following dependencies to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-logging"       % "2.0.0"
libraryDependencies += "dev.zio" %% "zio-logging-slf4j" % "2.0.0"
```

Now we can use the `SLF4J.sl4j` layer to enable SLF4J logging:

```scala

object MainApp extends ZIOAppDefault {
  override val bootstrap = SLF4J.slf4j(LogFormat.colored)

  def run = ZIO.log("Application started!")
}
```

Let's run the application and see the output:

```bash
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
```

Oops! The SLF4J failed to find any binding in the classpath. To fix this, we need to add an SLF4J binding to our classpath.

## Adding a Simple SLF4J Binding to the Classpath

The SLF4J has a simple binding that can be used by adding the `slf4j-simple` dependency to our `build.sbt` file:

```scala
libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.7.36"
```

And then we can run the application again:

```bash
[ZScheduler-Worker-7] INFO zio-slf4j-logger - timestamp=2022-06-04T19:36:43.768256+04:30 level=INFO thread=zio-fiber-6 message="Application started!"
```

It works! Now, our ZIO application uses SLF4J for its logging backend.

Similarly, we can bind our application to any other logging framework by adding the appropriate dependency to our `build.sbt` file:

- `slf4j-log4j12`
- `slf4j-reload4j`
- `slf4j-jdk14`
- `slf4j-nop`
- `slf4j-jcl`
- `logback-classic`

To switch to another logging framework, we need to provide one of the above dependencies instead of `slf4j-simple`. In the next section, we will learn how to switch to the `reload4j` logging framework.

## Switching to the Reload4j Logging Framework

To use the `reload4j` logging framework, we need to add the following dependencies to our `build.sbt` file:

```diff
- libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.7.36"
+ libraryDependencies += "org.slf4j" % "slf4j-reload4j" % "1.7.36"
```

Now we can configure our logger by adding the `log4j.properties` to the resources directory:

```
log4j.rootLogger = Info, consoleAppender
log4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.consoleAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n
```

By customizing the `ConversionPattern` we can control the format of the log messages.

## Switching to the Logback Logging Framework

In the same way, we can switch to the `logback-classic` logging framework by adding the following dependencies to our `build.sbt` file:

```diff
- libraryDependencies += "org.slf4j"      % "slf4j-reload4j"  % "1.7.36"
+ libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.11"
```

Then we can configure our logger by adding the `logback.xml` to the resources directory:

```xml
<?xml version="1.0" encoding="UTF-8"?>

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <root level="info">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>
```

## Conclusion

In this article, we have learned how to create a custom logger for a ZIO application. We also covered how to add SLF4J logging support instead of default ZIO logging.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) project.

---

## Tutorial: How to Debug a ZIO Application?

## Introduction

Writing applications using functional programming helps us to write a base code that is less error-prone and more predictable. However, we often make mistakes when developing applications. Even though we have written lots of tests, we might have missed some areas of our code that could have caused errors. Finally, in the middle of one night, the alarm starts calling and paging us to take the right action for the error in production. This is where debugging comes in. It is a process of finding the root cause of the error and then fixing it. Sometimes this process takes a large amount of time and effort.

In this article, we are going to learn how to debug a ZIO application. We will start with the simplest example of a ZIO application and then move to the more complex ones.

## Debugging an Ordinary Scala Application

Before talking about debugging functional effects, we need to understand how to debug an ordinary Scala application. In scala, one simple way to debug a code is to use `print` statements to print the intermediate values of the computation to the console.

Assume we have the following fibonacci function:

```scala
def fib(n: Int): Int = {
  @annotation.tailrec
  def go(n: Int, a: Int, b: Int): Int =
    if (n == 0) a
    else go(n - 1, b, a + b) 
  go(n, 0, 1)
}
```

The implementation of this function is correct, but for pedagogical purposes, let's debug it by printing the intermediate values of the computation:

```scala
def fib(n: Int): Int = {
  @annotation.tailrec
  def go(n: Int, a: Int, b: Int): Int =
    if (n == 0) {
      println(s"final result: $a")
      a
    } else {
      println(s"go(${n - 1}, $b, ${a + b})")
      go(n - 1, b, a + b)
    }
  println(s"go($n, 0, 1)")
  go(n, 0, 1)
}
```

Now if we call `fib(3)`, we will see the following output:

```
go(3, 0, 1)
go(2, 1, 1)
go(1, 1, 2)
go(0, 2, 3)
final result: 2
```

The `print` statements are the easiest way for lazy programmers to debug their code. However, they are not the most efficient way to debug code.

## Debugging a ZIO Application Using `debug` Effect

When we use functional effects like `ZIO`, we are creating the description of the computation that we want to run. For example, assume we have the following code:

```scala

val effect: ZIO[Any, Nothing, Unit] = ZIO.succeed(3).map(_ * 2)
```

The `effect` itself is a description of the computation that we want to run. So we can't use print statements to debug effects directly. For example, if we write `println(effect)`, we will get something like this:

```scala
OnSuccess(<empty>.MainApp.effect(MainApp.scala:4),Sync(<empty>.MainApp.effect(MainApp.scala:4),MainApp$$$Lambda$23/0x00000008000bc440@44a3ec6b),zio.ZIO$$Lambda$25/0x00000008000ba040@71623278)
```

This is not the expected output. We want to see the result of the computation, not the description of the computation. Why did this happen? Because we haven't run the computation yet.

So keep in mind that, unlike the ordinary scala print statements, we can't use print statements directly to debug functional effects, unless we unsafely run the computation:

```scala

val effect: ZIO[Any, Nothing, Int] = 
  ZIO.succeed(3).map(_ * 2)

val executedEffect: Int =
  Unsafe.unsafe { implicit unsafe =>
    Runtime.default.unsafe.run(effect).getOrThrowFiberFailure()
  }
  
println(s"executedEffect: $executedEffect")
```

This will print the result of the computation. But, this is not the idiomatic way to debug functional effects.

Simple _print statements_ are not composable with ZIO applications. So we can't use them to debug ZIO applications easily. So instead of print statements, we should use ZIO effects to debug ZIO applications. We can use `Console.printLine` effect to debug ZIO applications, but the ZIO itself has a specialized effect called `ZIO.debug`/`ZIO#debug` which allows us to print intermediate values easily.

For example, assume we have written the Fibonacci function using the `ZIO` data type:

```scala

def fib(n: Int): ZIO[Any, Nothing, Int] = {
  if (n <= 1) ZIO.succeed(n)
  else fib(n - 1).zipWith(fib(n - 2))(_ + _)
}
```

We can debug this program by utilizing the `ZIO#debug` effect:

```scala

def fib(n: Int): ZIO[Any, Nothing, Int] = {
  if (n <= 1) ZIO.succeed(n).debug(s"fib($n) = $n")
  else {
    fib(n - 1)
      .zipWith(fib(n - 2))(_ + _)
      .debug(s"fib($n) = fib(${n - 1}) + fib(${n - 2})")
  }
}
```

If we run the above program, we will see the following output:

```
fib(1) = 1: 1
fib(0) = 0: 0
fib(2) = fib(1) + fib(0): 1
fib(1) = 1: 1
fib(3) = fib(2) + fib(1): 2
fib(1) = 1: 1
fib(0) = 0: 0
fib(2) = fib(1) + fib(0): 1
fib(4) = fib(3) + fib(2): 3
fib(1) = 1: 1
fib(0) = 0: 0
fib(2) = fib(1) + fib(0): 1
fib(1) = 1: 1
fib(3) = fib(2) + fib(1): 2
fib(5) = fib(4) + fib(3): 5
```

The `ZIO#debug` effect taps into the called function and prints its output. It doesn't change the result of a computation. We also can use `ZIO.debug` to print any arbitrary message.

We have the same effect as the following:

```scala

ZStream
  .fromIterable(1 to 3)
  .debug("before map")
  .map(_ * 2)
  .debug("after map")
  .runDrain
```

The output is:

```
before map: 1
after map: 2
before map: 2
after map: 4
before map: 3
after map: 6
```

## Debugging Using a Debugger

Debugging using print statements is usable in some cases, and sometimes it is not performant. Another way to debug a code is to use a debugger. A debugger is a program that allows us to step through the code and see the intermediate values of the computation. Some IDEs like IntelliJ IDEA or Visual Studio Code have built-in debuggers. We can use these to debug our code, whether we are debugging a functional effect or an ordinary scala program.

To learn how to use a debugger in each of the IDEs, we can look at the following links:
- [IntelliJ IDEA](https://www.jetbrains.com/help/idea/run-debug-and-test-scala.html)
- [Visual Studio Code](https://code.visualstudio.com/docs/editor/debugging)

## Conclusion

In this article we discussed how to debug functional effects using `debug` effect and also debuggers. We saw that debugging with functional effects can be even easier than debugging ordinary scala programs.

---

## Tutorial: How to Deploy a ZIO Application Using Docker?

## Introduction

Docker is a tool that allows us to package, ship, and run our applications in an isolated environment called a container. Using Docker, we can simplify the deployment process by isolating our applications in their own container and abstracting them from the host environment.

In this tutorial, we are going to learn how to build a Docker image for our ZIO application and then how to deploy it. Instead of writing the `Dockerfile` from scratch, we will use the _[sbt-native-packager](https://github.com/sbt/sbt-native-packager)_ to build our Docker image.

## Running The Examples

In [this quickstart](../quickstarts/restful-webservice.md), we developed a web service containing 4 different HTTP Applications. Now in this article, we want to dockerize this web application.

To access the code examples, you can clone the [ZIO Quickstarts](http://github.com/zio/zio-quickstarts) project:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-restful-webservice-dockerize
```

Once you are inside the project directory, run the application:

```bash
sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Prerequisites

Before we can dockerize our web service, we need to [download and install Docker](https://docs.docker.com/get-docker/). So we assume that the reader has already installed Docker.

## Adding SBT Native Packager Plugin

The sbt-native-packager is an sbt plugin that enables us an easy way to package the application as a docker image and deploy that as a docker container.

First, we need to add the plugin to our `project/plugins.sbt` file:

```
addSbtPlugin("com.github.sbt" % "sbt-native-packager" % "1.9.9")
```

Now it's time to enable the `JavaAppPackaging` and `DockerPlugin` plugins. So we need to add the following lines in the `build.sbt` file:

```
enablePlugins(JavaAppPackaging)
enablePlugins(DockerPlugin)
```

## Building The Docker Image

The `DockerPlugin` plugin of _sbt-native-packager_ is responsible for configuring and building the docker image. We can run the following command to build the docker image:

```bash
$ sbt docker:publishLocal
```

After the docker image is built, we can run the `docker images` command to see the list of images that are currently available in the local docker registry:

```bash
$ docker images
REPOSITORY                          TAG     IMAGE ID       CREATED        SIZE
zio-quickstart-restful-webservice   0.1.0   c9ae81ee8fa6   17 hours ago   558MB
```

Note that, to see the generated `Dockerfile` we can use the `docker:stage` command:

```bash
$ sbt docker:stage
```

The `Dockerfile` will be generated in the `target/docker/stage` directory.

## Deploying The Docker Image

Now we can create a new container from this image by using the `docker run` command:

```bash
$ docker run -p 80:800 zio-quickstart-restful-webservice:0.1.0
```

Using the `-p` flag, we can specify the port that the container will listen to. As the web service is running on port `8080`, we bind this port to the host port `80`. Therefore, we can access the web service from the host machine through the port `80`:

```bash
$ curl -i "http://localhost/greet?name=Jane&name=John"
HTTP/1.1 200 OK
content-type: text/plain
content-length: 20

Hello Jane and John!
```

## Configuring The Docker Image

By default, the `sbt-native-packager` plugin will build the docker image using some predefined settings. So without any configuration we can use the `sbt docker:publish` or `sbt docker:publishLocal` commands to build and publish the docker image to the remote or local docker registry.

However, it is possible to configure the docker image, and it has lots of options to configure. We can find the list of available options in the [sbt-native-packager documentation](https://www.scala-sbt.org/sbt-native-packager/formats/docker.html#configuration).

### Exposing Container Ports

For example, when we build a docker image, we can specify which ports the container will listen to, by using the `EXPOSE` instruction in the `Dockerfile`. In the similar way, we can expose the ports using _sbt-native-packager_, by using the `dockerExposedPorts` setting in the `build.sbt` file:

```scala
dockerExposedPorts := Seq(8080)
```

Now, when we build the docker image and create a container from it, the new container has the port `8080` exposed. So when we run the `docker ps` command, we can see that the new container has the port `8080` exposed under the `PORTS` column:

```bash
$ docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED         STATUS         PORTS      NAMES
29982b053379   zio-quickstart-restful-webservice:0.1.0   "/opt/docker/bin/zio…"   3 seconds ago   Up 2 seconds   8080/tcp   bold_liskov
```

### Publishing The Docker Image to a Remote Registry

In a CI/CD pipeline, we might want to publish the docker image to a remote registry other than the local registry. We can do this by configuring the `dockerUsername` and `dockerRepository` settings in the `build.sbt` file:

```scala
dockerUsername := sys.props.get("docker.username")
dockerRepository := sys.props.get("docker.registry")
```

Now, we can use the following command to publish the docker image to the remote registry:

```bash
$ export DOCKER_USERNAME=<username>  // e.g: johndoe
$ export DOCKER_REGISTRY=<registry>  // e.g: docker.io
$ sbt -Ddocker.username=$NAMESPACE -Ddocker.registry=$DOCKER_REGISTRY docker:publish
```

## Conclusion

In this tutorial, we learned how to build a docker image using _sbt-native-packager_, and how to deploy the docker image to the local or remote Docker registry.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) on Github.

---

## Tutorial: How to Enable Logging in a ZIO Application

## Introduction

ZIO has built-in support for logging. This tutorial will show you how to enable logging for a ZIO application.

## Running The Examples

In [this quickstart](../quickstarts/restful-webservice.md), we developed a web service containing 4 different HTTP Applications, but we haven't enabled logging for them. In this tutorial, we are going to enable logging for the `UserApp` we have developed in this quickstart.

To access the code examples, you can clone the [ZIO Quickstarts](http://github.com/zio/zio-quickstarts) project:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-restful-webservice-logging
```

And finally, run the application using sbt:

```bash
$ sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Default Logger

ZIO has a default logger that prints any log messages that are equal, or above the `Level.Info` level:

```scala

object MainApp extends ZIOAppDefault {
  def run = ZIO.log("Application started")
}
```

This will print the following message to the console:

```
timestamp=2022-06-01T09:43:08.848398Z level=INFO thread=#zio-fiber-6 message="Application started" location=zio.examples.MainApp.run file=MainApp.scala line=4
```

If we want to include the `Cause` in the log message, we can use the `ZIO.logCause` which takes the message and also the cause:

```scala modc:compile-only

object MainApp extends ZIOAppDefault {

  def run =
    ZIO
      .dieMessage("Boom!")
      .foldCauseZIO(
        cause => ZIO.logErrorCause("application stopped working due to an unexpected error", cause),
        _ => ZIO.unit
      )
}
```

Here is the output which includes the cause:

```
timestamp=2022-06-02T05:18:04.131876Z level=ERROR thread=#zio-fiber-6 message="application stopped working due to an unexpected error" cause="Exception in thread "zio-fiber-6" java.lang.RuntimeException: Boom!
at zio.examples.MainApp.run(MainApp.scala:9)
at zio.examples.MainApp.run(MainApp.scala:10)" location=zio.examples.MainApp.run file=MainApp.scala line=11
```

## Supported Log Levels

To distinguish importance of log messages from each other, ZIO supports the following log levels. The default log level is `Info`, so when we use the `ZIO.log` or `ZIO.logCause` methods, the log message will be logged at the `Info` level. For other log levels, we can use any of the `ZIO.log*` or `ZIO.log*Cause` methods:

| LogLevel | Value        | Log Message    | Log with Cause      |
| -------- | ------------ | -------------- | ------------------- |
| All      | Int.MinValue |                |                     |
| Fatal    | 50000        | ZIO.logFatal   | ZIO.logFatalCause   |
| Error    | 40000        | ZIO.logError   | ZIO.logErrorCause   |
| Warning  | 30000        | ZIO.logWarning | ZIO.logWarningCause |
| Info     | 20000        | ZIO.logInfo    | ZIO.logInfoCause    |
| Debug    | 10000        | ZIO.logDebug   | ZIO.logDebugCause   |
| Trace    | 0            | ZIO.logTrace   | ZIO.logTraceCause   |
| None     | Int.MaxValue |                |                     |

As we said earlier, the default logger prints log messages equal or above the `Info` level. So, if we run the following code, only the `Info`, `Warning`, and `Error` and the `Fatal` log messages will be printed:

```scala

object MainApp extends ZIOAppDefault {
  def run =
    for {
      _ <- ZIO.logFatal("Fatal")
      _ <- ZIO.logError("Error")
      _ <- ZIO.logWarning("Warning")
      _ <- ZIO.logInfo("Info")
      _ <- ZIO.logDebug("Debug")
      _ <- ZIO.logTrace("Trace")
    } yield ()
}
```

The output will look like the following:

```
timestamp=2022-06-01T10:16:26.623633Z level=FATAL thread=#zio-fiber-6 message="Fatal" location=zio.examples.MainApp.run file=MainApp.scala line=6
timestamp=2022-06-01T10:16:26.638771Z level=ERROR thread=#zio-fiber-6 message="Error" location=zio.examples.MainApp.run file=MainApp.scala line=7
timestamp=2022-06-01T10:16:26.640827Z level=WARN thread=#zio-fiber-6 message="Warning" location=zio.examples.MainApp.run file=MainApp.scala line=8
timestamp=2022-06-01T10:16:26.642260Z level=INFO thread=#zio-fiber-6 message="Info" location=zio.examples.MainApp.run file=MainApp.scala line=9
```

## Overriding Log Levels

As we mentioned, the default log level for ZIO.log and ZIO.logCause is `Info`. We can use these two methods to log various part of our workflow, and then finally we can wrap the whole workflow with our desired log level:

```scala

ZIO.logLevel(LogLevel.Debug) {
  for {
    _ <- ZIO.log("Workflow Started.")
    _ <- ZIO.log("Running task 1.")
    _ <- ZIO.log("Running task 2.")
    _ <- ZIO.log("Workflow Completed.")
  } yield ()
}
```

## UserApp: Add Logging

In this section, we are going to log all the HTTP requests coming to the `UserApp`, and then in each step, when we are handling a request we will log the result, whether is successful or not.

We demonstrate this for the "POST /users" endpoint. This process is the same for all the other endpoints:

```scala

Routes(
  // POST /users -d '{"name": "John", "age": 35}'
  Method.POST / "users" -> handler { (req: Request) =>
    (for {
      body <- req.body.asString
      _ <- ZIO.logInfo(s"POST /users -d $body")
      u = body.fromJson[User]
      r <- u match {
        case Left(e) =>
          ZIO.logErrorCause(s"Failed to parse the input", Cause.fail(e))
            .as(Response.text(e).status(Status.BadRequest))
        case Right(u) =>
          UserRepo.register(u)
            .foldCauseZIO(
              failure =>
                ZIO.logErrorCause(s"Failed to register user", Cause.fail(failure))
                  .as(Response.status(Status.InternalServerError)),
              success =>
                ZIO.logInfo(s"User registered: $success")
                  .as(Response.text(success))
            )
      }
    } yield r) @@ logSpan("register-user") @@ logAnnotateCorrelationId(req)
  }
)
```

## Logging Spans

ZIO supports logging spans. A span is a logical unit of work that is composed of a start and end time. The start time is when the span is created and the end time is when the span is completed. The span is useful for measuring the time it takes to execute a piece of code. To create a new span, we can use the `ZIO.logSpan` function as follows:

```scala

ZIO.logSpan("span name") {
  // do some work
  // log inside the span
  // do some more work
  // another log inside the span
}
```

For example, assume we have a function that takes the `username` and returns the profile picture of the user. We can wrap the whole function in a new span called "get-profile-picture" and log inside the span:

```scala

case class User(id: String, name: String, profileImage: String)

def getProfilePicture(username: String) =
  ZIO.logSpan("get-profile-picture") {
    for {
      _    <- ZIO.log(s"Getting information of $username from the UserService")
      user <- ZIO.succeed(User("1", "john", "john.png"))
      _    <- ZIO.log(s"Downloading profile image ${user.profileImage}")
      img  <- ZIO.succeed(Array[Byte](1, 2, 3))
      _    <- ZIO.log("Profile image downloaded")
    } yield img
  }
```

If we run this code with `getProfilePicture("john")`, the output will look like the following:

```
timestamp=2022-06-01T13:59:40.779263Z level=INFO thread=#zio-fiber-6 message="Getting information of john from the UserService" get-profile-picture=6ms location=zio.examples.MainApp.getProfilePicture file=MainApp.scala line=11
timestamp=2022-06-01T13:59:40.793804Z level=INFO thread=#zio-fiber-6 message="Downloading profile image john.png" get-profile-picture=20ms location=zio.examples.MainApp.getProfilePicture file=MainApp.scala line=13
timestamp=2022-06-01T13:59:40.795677Z level=INFO thread=#zio-fiber-6 message="Profile image downloaded" get-profile-picture=22ms location=zio.examples.MainApp.getProfilePicture file=MainApp.scala line=15
```

Any logs inside the span region will be logged with the span name and the duration from the start of the span.

## Multiple Spans

We can also create multiple spans and log inside them:

```scala

ZIO.logSpan("span1") {
  for {
    _ <- ZIO.log("log inside span1")
    _ <- ZIO.logSpan("span2") {
      ZIO.log("log inside span1 and span2")
    }
    _ <- ZIO.log("log inside span1")
  } yield ()
}
```

## UserApp: Logging Spans

To measure the time taken to process the request at different points of the code, we can wrap any workflow with `ZIO.logSpan`. In the `UserApp` example, we wrote a workflow that handles the registration of a new user. We can wrap the workflow in a span and log inside the span:

```scala

Routes(
  // POST /users -d '{"name": "John", "age": 35}'
  Method.POST / "users" ->
    handler(ZIO.logSpan("register-user") {
      ??? // registration workflow
    })
)
```

As we need the same for all other endpoints, we introduced a new ZIO Aspect called `LogAspect.logSpan` which can be applied to any ZIO workflow. Let's see how it is implemented and how it works:

```scala

object LogAspect {
  def logSpan(
      label: String
  ): ZIOAspect[Nothing, Any, Nothing, Any, Nothing, Any] =
    new ZIOAspect[Nothing, Any, Nothing, Any, Nothing, Any] {
      override def apply[R, E, A](zio: ZIO[R, E, A])(
        implicit trace: Trace
      ): ZIO[R, E, A] =
        ZIO.logSpan(label)(zio)
    }
}
```

To apply this aspect to a ZIO workflow, we can use the `@@` operator:

```scala
val workflow = ZIO.unit
```

```scala
workflow @@ LogAspect.logSpan("register-user")
```

Now, let's run the web service and post a request to the `/users` endpoint and create a new user and see the logs:

```bash
$ curl -i http://localhost:8080/users -d '{"name": "John", "age": 42}'
```

And the logs:

```
timestamp=2022-06-03T09:42:15.590135Z level=INFO thread=#zio-fiber-16 message="POST /users -d {"name": "John", "age": 42}" register-user=16ms location=dev.zio.quickstart.users.UserApp.apply.applyOrElse file=UserApp.scala line=22
timestamp=2022-06-03T09:42:15.748359Z level=INFO thread=#zio-fiber-16 message="User registered: 24c4ed63-ecc2-41fb-ac0e-5cbf22f187f6" register-user=174ms location=dev.zio.quickstart.users.UserApp.apply.applyOrElse file=UserApp.scala line=35
```

In the above logs, we can see that the `register-user` span is created and the time taken to process the request is logged.

```bash
register-user=16ms
register-user=174ms
```

## Annotating Logs

The default log message is a string, containing a series of key-value pairs. It contains the following information:

- `timestamp`: the time when the log is created
- `level`: the log level
- `thread`: the thread name (fiber name)
- `message`: the log message
- `cause`: the cause of the log
- `location`: the location of the source code where the log is created (filename)
- `line`: the line number of the source code where the log is created

Other than these key-value pairs, we can also annotate the log message with other customized information using the `ZIO.logAnnotate` function. This is especially useful when we are writing multiple services and we want to correlate logs between them using the `CorrelationId`.

For example, when we are handling an HTTP request, we can annotate the log message with the user's id:

```scala

object MainApp extends ZIOAppDefault {
  // We use random delay to simulate interleaving of operations in real world
  def randomDelay = Random.nextIntBounded(1000).flatMap(t => ZIO.sleep(t.millis))

  def run =
    ZIO.foreachParDiscard(Chunk("UserA", "UserB", "UserC")) { user =>
      ZIO.logAnnotate("user-id", user) {
        for {
          _ <- randomDelay
          _ <- ZIO.log("fetching user from database")
          _ <- randomDelay
          _ <- ZIO.log("downloading user's profile picture")
        } yield ()
      }
    }
}
```

The output will look as follows:

```
timestamp=2022-06-01T16:31:00.058151Z level=INFO thread=#zio-fiber-9 message="fetching user from database" location=zio.examples.MainApp.run file=MainApp.scala line=14 user-id=UserC
timestamp=2022-06-01T16:31:00.413569Z level=INFO thread=#zio-fiber-7 message="fetching user from database" location=zio.examples.MainApp.run file=MainApp.scala line=14 user-id=UserA
timestamp=2022-06-01T16:31:00.525170Z level=INFO thread=#zio-fiber-9 message="downloading user's profile picture" location=zio.examples.MainApp.run file=MainApp.scala line=16 user-id=UserC
timestamp=2022-06-01T16:31:00.807873Z level=INFO thread=#zio-fiber-8 message="fetching user from database" location=zio.examples.MainApp.run file=MainApp.scala line=14 user-id=UserB
timestamp=2022-06-01T16:31:00.830572Z level=INFO thread=#zio-fiber-7 message="downloading user's profile picture" location=zio.examples.MainApp.run file=MainApp.scala line=16 user-id=UserA
timestamp=2022-06-01T16:31:00.839411Z level=INFO thread=#zio-fiber-8 message="downloading user's profile picture" location=zio.examples.MainApp.run file=MainApp.scala line=16 user-id=UserB
```

As we can see, the `user-id` with its value is annotated to each log message.

## UserApp: Logging Correlation Ids

To add a Correlation ID to the logs, we should first extract the `X-Correlation-ID` header from the request and use it as the Correlation ID.

As this is a common pattern along with all other endpoints, we created a new ZIO Aspect called `LogAspect.logAnnotateCorrelationId` which can be applied to any ZIO workflow:

```scala

object LogAspect {
  def logAnnotateCorrelationId(
      req: Request
  ): ZIOAspect[Nothing, Any, Nothing, Any, Nothing, Any] =
    new ZIOAspect[Nothing, Any, Nothing, Any, Nothing, Any] {
      override def apply[R, E, A](
          zio: ZIO[R, E, A]
      )(implicit trace: Trace): ZIO[R, E, A] =
        correlationId(req).flatMap(id => ZIO.logAnnotate("correlation-id", id)(zio))

      def correlationId(req: Request): UIO[String] =
        ZIO
          .succeed(req.headers.get("X-Correlation-ID"))
          .flatMap(id => Random.nextUUID.map(uuid => id.getOrElse(uuid.toString)))
    }
}
```

Now, we can apply this aspect to any ZIO workflow:

```scala
workflow @@ LogAspect.logAnnotateCorrelationId(req)
```

By applying this aspect to a ZIO workflow, all logs inside that workflow will be annotated with the `correlation-id` annotation:

Let's run the web service and post a request to the `/users` endpoint and create a new user and see the logs:

```bash
$ curl -i -H "X-Correlation-ID: f798d2f2-abf2-46ff-b3f4-ae1888256706" \
      http://localhost:8080/users -d '{"name": "John", "age": 42}'
```

Here are the logs:

```scala
timestamp=2022-06-03T10:13:18.334468Z level=INFO thread=#zio-fiber-32 message="POST /users -d {"name": "John", "age": 42}" register-user=1ms location=dev.zio.quickstart.users.UserApp.apply.applyOrElse file=UserApp.scala line=22 correlation-id=f798d2f2-abf2-46ff-b3f4-ae1888256706
timestamp=2022-06-03T10:13:18.335034Z level=INFO thread=#zio-fiber-32 message="User registered: ec02143a-8030-4c70-a110-a497617c5c72" register-user=2ms location=dev.zio.quickstart.users.UserApp.apply.applyOrElse file=UserApp.scala line=35 correlation-id=f798d2f2-abf2-46ff-b3f4-ae1888256706
```

We can see that both log lines have the same `correlation-id` annotation.

## Conclusion

In this tutorial, we learned how to enable logging in a ZIO application and how to use the built-in logging facilities of ZIO without requiring any additional dependencies.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) on Github.

---

## Tutorial: How to Encode and Decode JSON Data?

## Introduction

In this article, we will cover how to encode and decode JSON data.

## Running Examples

To access the code examples, you can clone the [ZIO Quickstarts](http://github.com/zio/zio-quickstarts) project:

```bash 
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-encode-decode-json-data
```

To run all tests, execute the following command:

```bash
$ sbt Test/runMain dev.zio.quickstart.JsonSpec
```

## What is ZIO JSON?

ZIO JSON is a library that provides facilities for writing efficient JSON encoders and decoders. In this article, we will use this library to work with JSON data. To learn more about that, please refer to the [ZIO JSON](https://zio.github.io/zio-json/) documentation.

## Adding Dependencies

To use ZIO JSON, we need to add the following dependency to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-json" % "0.3.0-RC10"
```

## JsonEncoder and JsonDecoder

The `JsonEncoder` and `JsonDecoder` are the two main types in ZIO JSON. They are used to encode and decode JSON data. Let's see how they are defined (with simplified syntax):

```scala
trait JsonDecoder[A] {
  def decodeJson(str: CharSequence): Either[String, A]
}

trait JsonEncoder[A] {
  def encodeJson(a: A): CharSequence
}
```

We can say for a type `A`:
- A **decoder** is a function that takes a `CharSequence` and returns a `Right` with the decoded value or a `Left` with an error message.
- An **encoder** is a function that takes a value of type `A` and returns a `CharSequence` that represents the encoded value (JSON string).

If we provide an instance of `JsonDecoder` and `JsonEncoder` for a type `A`, we can encode and decode JSON data of that type.

## Built-in Decoders and Encoders

### Simple Values

The ZIO JSON library provides a default implementation for most of the primitive types like `Int`, `String`, `Boolean`, etc.

Let's start test some simple examples:

```scala

test("decode from string") {
  val json    = "\"John Doe\""
  val decoded = JsonDecoder[String].decodeJson(json)

  assertTrue(decoded == Right("John Doe"))
}

test("decode from int") {
  val json    = "123"
  val decoded = JsonDecoder[Int].decodeJson(json)

  assertTrue(decoded == Right(123))
}
```

### Higher-kinded Types

It also supports higher-kinded types like `List` and `Option`:

```scala

test("decode from optional value") {
  val json = "null"
  val decoded = JsonDecoder[Option[Int]].decodeJson(json)
  assertTrue(decoded == Right(None))
} +
test("decode from array of ints") {
  val json    = "[1, 2, 3]"
  val decoded = json.fromJson[Array[Int]]

  assert(decoded)(isRight(equalTo(Array(1, 2, 3))))
}
```

## How to Define Custom Decoder/Encoder?

### Writing From Scratch

To have a new instance we implement the `JsonEncoder` and `JsonDecoder` interfaces for a type `A`.

For example, if we have a type `Person` we can create instances of `JsonEncoder` and `JsonDecoder` for this type as below:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val encoder: JsonEncoder[Person] =
    new JsonEncoder[Person] {
      override def unsafeEncode(a: Person, indent: Option[Int], out: Write): Unit = ???
    }
  implicit val decoder: JsonDecoder[Person] =
    new JsonDecoder[Person] {
      override def unsafeDecode(trace: List[JsonError], in: RetractReader): Person = ???
    }
}
```

Writing encoders and decoders from scratch is a complicated task and is not recommended for regular usage. So we don't deep into it furthermore.

### Automatic Derivation of Codecs (macros)

By using macro utilities, we can derive the instances of `JsonEncoder` and `JsonDecoder` for a case class using `DeriveJsonDecoder.gen[A]` and `DeriveJsonEncoder.gen[A]` macros:

```scala

test("automatic derivation for case classes") {
  case class Person(name: String, age: Int)
  object Person {
    implicit val decoder: JsonDecoder[Person] = DeriveJsonDecoder.gen[Person]
    implicit val encoder: JsonEncoder[Person] = DeriveJsonEncoder.gen[Person]
  }

  assertTrue((Person("John", 42).toJson == "{\"name\":\"John\",\"age\":42}")
    && ("{\"name\":\"John\",\"age\":42}".fromJson[Person] == Right(Person("John", 42)))
  )
}
```

Let's try a more complex example. Assume we have a data type `Fruit` that is written as follows:

```scala
sealed trait Fruit extends Product with Serializable
case class Banana(curvature: Double) extends Fruit
case class Apple (poison: Boolean)   extends Fruit
```

We can generate encoder and decoder for this ADT using the macros:

```scala

object Fruit {
  implicit val decoder: JsonDecoder[Fruit] =
    DeriveJsonDecoder.gen[Fruit]

  implicit val encoder: JsonEncoder[Fruit] =
    DeriveJsonEncoder.gen[Fruit]
}
```

So then we can have the following tests:

```scala

test("decode from custom adt") {
  val json =
    """
      |[
      |  {
      |    "Apple": {
      |      "poison": false
      |    }
      |  },
      |  {
      |    "Banana": {
      |      "curvature": 0.5
      |    }
      |  }
      |]
      |""".stripMargin

  val decoded = json.fromJson[List[Fruit]]
  assertTrue(decoded == Right(List(Apple(false), Banana(0.5))))
} +
test("roundtrip custom adt") {
  val fruits = List(Apple(false), Banana(0.5))
  val json = fruits.toJson
  val roundTrip = json.fromJson[List[Fruit]]
  assertTrue(roundTrip == Right(fruits))
}
```

### Mapping Existing Codecs to Complex Types

If we have `JsonDecoder[A]` we can map its **output** to `JsonDecoder[B]` by providing a function `A => B` as an argument to `map` operation:

```scala
trait JsonDecoder[A] {
  def map[B](f: A => B): JsonDecoder[B]
} 
```

Example:

```scala

test("mapping decoders") {
  case class Person(name: String, age: Int)
  object Person {
    implicit val decoder = JsonDecoder[(String, Int)].map { case (name, age) => Person(name, age) }
  }

  val person = "[\"John\",42]".fromJson[Person]

  assertTrue(person == Right(Person("John", 42)))
}
```

If we have `JsonEncoder[A]` we can map its **input** by providing a function  of typ `B => A` to `contramap` operator to create a new `JsonEncoder[B]`:

```scala
trait JsonEncoder[A] {
  def contramap[B](f: B => A): JsonEncoder[B]
}
```

Example:

```scala

test("mapping encoders (contramap)") {
  case class Person(name: String, age: Int)
  object Person {
    implicit val encoder: JsonEncoder[Person] =
      JsonEncoder[(String, Int)].contramap((p: Person) => (p.name, p.age))
  }

  val json = Person("John", 42).toJson

  assertTrue(json == "[\"John\",42]")
}
```

## Conclusion

In this section we have covered the basics of JSON encoding and decoding. We have also seen how to create custom codecs for complex types. 

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) on Github.

---

## Tutorial: How to Gracefully Shutdown ZIO Applications?

## Introduction

Graceful shutdown is a critical aspect of building robust and reliable software applications. It ensures that an application terminates smoothly, allowing it to complete necessary operations, release resources, and exit in a controlled manner.

In this article, we will explore how to achieve a graceful shutdown in ZIO applications, leveraging the power of the ZIO functional effect system.

## What is Graceful Shutdown?

Graceful shutdown refers to the process of stopping an application in a controlled and orderly manner.

Graceful shutdowns are applicable in various scenarios where we need to ensure the proper and controlled termination of an application or service. Some common use cases for graceful shutdowns include:

- **Server Applications**: When stopping a server application, such as a web server, API server, or message broker, it's essential to gracefully handle incoming requests, complete ongoing operations, and release resources before shutting down. Graceful shutdowns prevent data loss, ensure clean client disconnections, and maintain service availability for existing requests.

- **Database Connections**: Graceful shutdowns are important when closing database connections to ensure that pending transactions are committed or rolled back, and resources are released. This helps maintain data integrity and prevent blocking or deadlocking issues caused by abrupt connection closures.

- **Background Processes and Workers**: Graceful shutdowns are relevant for long-running background processes, worker threads, or scheduled tasks. They provide a mechanism to stop these processes gracefully, complete their current work items, and release any acquired resources before exiting.

- **Service-oriented Architectures**: In a service-oriented architecture, where multiple services communicate with each other, graceful shutdowns enable services to notify their dependencies, complete pending requests, and gracefully disconnect from other services before shutting down.

- **Streaming and Data Processing Applications**: Graceful shutdowns are crucial for streaming applications or data processing pipelines. They provide a mechanism to complete in-flight data processing, flush buffers, and ensure data consistency before stopping the application.

In summary, graceful shutdowns are relevant in a wide range of scenarios where we need to ensure the proper completion of ongoing operations, release resources, and maintain system integrity before terminating an application or service. They play a crucial role in ensuring data consistency, preventing resource leaks, and providing a smooth user experience during application shutdown.

## Releasing Resources Before App Termination

To ensure that all resources are released before our application terminates, we need to ensure that we have managed all resources using `ZIO.acquireRelease*` operations. It ensures that resources are properly acquired when needed and released when they are no longer required. In their release action, we should perform any clean-up operation needed to make sure our termination process is graceful.

```scala
trait ZIO[-R, +E, +A] {
  def acquireRelease[R, R1, E, A](
      acquire: => ZIO[R, E, A]
    )(
      release: A => ZIO[R1, Nothing, Any]
    ): ZIO[R with R1 with Scope, E, A]
}
```

Using `acquireRelease` method, involve another data type called `Scope`. For further details, please refer to the [`Scope`](../../reference/resource/scope.md) page on the documentation project.

## Adding Finalizer Before App Termination

By using `ensuring`/`onExit` in our ZIO application, we are explicitly specifying the clean-up logic that needs to be executed regardless of the outcome (success, failure, or interruption) of the ZIO effect.

The overall process is like this:

```scala

object GracefulShutdownExample extends ZIOAppDefault {

  def appLogic = {
    // Your application logic goes here
    ZIO.unit
  }

  def cleanup = {
    // Perform cleanup operations here
    ZIO.unit
  }

  override def run = appLogic.ensuring(cleanup)
}
```

We can categorize these finalizers into two categories:

1. If we don't require the exit status of effect, we can use `ZIO#ensuring`

```scala
trait ZIO[-R, +E, +A] {
  final def ensuring[R1 <: R](finalizer: => URIO[R1, Any]): ZIO[R1, E, A]
}
```

2. If we need the exit status of effect, we can use `ZIO#onExit`

```scala
trait ZIO[-R, +E, +A] {
  final def onExit[R1 <: R](cleanup: Exit[E, A] => URIO[R1, Any]): ZIO[R1, E, A]
}
```

## Graceful Shutdown and Resource Management in ZIO Applications

When building a ZIO application that integrates with external libraries such as ZIO HTTP, ZIO kafka, ZIO Redis, or any other resourceful ZIO libraries, it's common to rely on those libraries' built-in resource management utilities, often implemented using ZIO's `acquireRelease` or similar mechanisms.

When we introduce an additional clean-up task using ZIO's ensuring operator within our ZIO application, we not only ensure that the clean-up task will be executed regardless of whether the entire app succeeds, fails, or is interrupted, but we also ensure that all the resources managed by those external libraries will be properly initialized and released. This ensures that our application can gracefully shut down while handling any in-flight incoming requests or ongoing operations with the integrated libraries.

This allows us to coordinate the release of resources, complete pending operations, and ensure the stability and consistency of our application during the shutdown process.

## Conclusion

The combination of ZIO's ensuring operator and the resource management utilities provided by the integrated libraries ensures that the resources acquired by those libraries will be released properly. This helps prevent resource leaks, maintain data integrity, and provide a graceful shutdown experience for our ZIO application, including handling ongoing requests and tasks.

---

## Tutorial: How to Make a ZIO Application Configurable?

## Introduction

One of the most common requirements for writing an application is to be able to configure it, especially when we are writing cloud-native applications.

In this tutorial, we will start with a simple ZIO application and then try to make it configurable using the ZIO Config library.

## Prerequisites

We will use the [ZIO Quickstart: Restful Web Service](../quickstarts/restful-webservice.md) as our ground project. So make sure you have downloaded and tested it before you start this tutorial.

## Problem

We have a web service that does not allow us to configure the host and port of the service:

```bash
git clone https://github.com/zio/zio-quickstarts.git
cd zio-quickstarts/zio-quickstart-restful-webservice
sbt run
```

The output is:

```
Server started on http://localhost:8080
```

We want to be able to configure the host and port of the service so that before running the application, we specify the host and port of the service.

In this article, we will see how we can make our application configurable using ZIO Config.

## Step 1: Define the Configuration Data Types (ADTs)

In this example our configuration data type is a case class that contains two fields:

```scala
case class HttpServerConfig(host: String, port: Int, nThreads: Int)
```

## Step 2: Define the Configuration Descriptor

Next, we need to define the configuration descriptor that describes the configuration data type. The best practice is to define the configuration descriptor in the companion object of the configuration data type:

```scala

object HttpServerConfig {
  implicit val config: Config[HttpServerConfig] =
    deriveConfig[HttpServerConfig].nested("HttpServerConfig")
}
```

## Step 3: Accessing Configuration Data using `ZIO.config`

By utilizing the `ZIO.config[HttpServerConfig]` function, we can obtain access to the configuration information that has been read by the current `ConfigProvider`:

```scala

ZIO.config[HttpServerConfig].flatMap { config =>
  ??? // Do something with the configuration
}
```

The above code is a ZIO effect that will access the `HttpServerConfig` configuration data and then by using flatMap, we can do something with it, for example, we can print it:

```scala

val workflow: ZIO[Any, Exception, Unit] =
  ZIO.config[HttpServerConfig].flatMap { config =>
    Console.printLine(
      "Application started with following configuration:\n" +
        s"\thost: ${config.host}\n" +
        s"\tport: ${config.port}"
    )
  }
```

Let's run the above workflow and see the output:

```scala

case class HttpServerConfig(host: String, port: Int)

object HttpServerConfig {
  implicit val config: Config[HttpServerConfig] = deriveConfig[HttpServerConfig].nested("HttpServerConfig")
}

object MainApp extends ZIOAppDefault {

  val workflow: Task[Unit] =
    ZIO.config[HttpServerConfig].flatMap { config =>
      Console.printLine(
        "Application started with following configuration:\n" +
          s"\thost: ${config.host}\n" +
          s"\tport: ${config.port}"
      )
    }

  def run = workflow
}
```

When try to run the above code, we will see the following output:

```log
timestamp=2023-04-01T14:00:13.902065Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-4" zio.Config$Error$And: ((((Missing data at HttpServerConfig.host: Expected HTTPSERVERCONFIG_HOST to be set in the environment) or (Missing data at HttpServerConfig.host: Expected HttpServerConfig.host to be set in properties)) and ((Missing data at HttpServerConfig.nThreads: Expected HTTPSERVERCONFIG_NTHREADS to be set in the environment) or (Missing data at HttpServerConfig.nThreads: Expected HttpServerConfig.nThreads to be set in properties))) and ((Missing data at HttpServerConfig.port: Expected HTTPSERVERCONFIG_PORT to be set in the environment) or (Missing data at HttpServerConfig.port: Expected HttpServerConfig.port to be set in properties)))
        at dev.zio.quickstart.MainApp.run(MainApp.scala:35)"
```

The above error is because we have not provided any configuration to the application. By default, ZIO will try to read configuration data from the application properties or environment variables.

So let's try to provide them as environment variables and see what happens:

```bash
HTTPSERVERCONFIG_HOST=localhost HTTPSERVERCONFIG_PORT=8080 HTTPSERVERCONFIG_NTHREADS=0 sbt "runMain dev.zio.quickstart.MainApp"
```

Now we can see this output:

```
Application started with following configuration:
	host: localhost
	port: 8080
```

Great! We have ZIO application that can access the configuration data. It works! Now, let's apply the same approach to our RESTful Web Service.

```scala title="documentation/guides/tutorials/make-a-zio-app-configurable/src/main/scala/dev/zio/quickstart/MainApp.scala" 
package dev.zio.quickstart

object MainApp extends ZIOAppDefault {
  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.setConfigProvider(
      ConfigProvider.fromResourcePath()
    )

  private val serverConfig: ZLayer[Any, Config.Error, Server.Config] =
    ZLayer
      .fromZIO(
        ZIO.config[HttpServerConfig](HttpServerConfig.config).map { c =>
          Server.Config.default.binding(c.host, c.port)
        }
      )

  private val nettyConfig: ZLayer[Any, Config.Error, NettyConfig] =
    ZLayer
      .fromZIO(
        ZIO.config[HttpServerConfig](HttpServerConfig.config).map { c =>
          NettyConfig.default.maxThreads(c.nThreads)
        }
      )

  def run = {
    (Server
      .install(
        GreetingRoutes() ++ DownloadRoutes() ++ CounterRoutes() ++ UserRoutes()
      )
      .flatMap(port =>
        Console.printLine(s"Started server on port: $port")
      ) *> ZIO.never)
      .provide(
        serverConfig,
        nettyConfig,
        Server.live,

        // A layer responsible for storing the state of the `counterApp`
        ZLayer.fromZIO(Ref.make(0)),

        // To use the persistence layer, provide the `PersistentUserRepo.layer` layer instead
        InmemoryUserRepo.layer
      )
  }
}
```

Until now, we made our RESTful web service configurable to be able to use its config from the ZIO environment with a simple configuration layer.

Now let's move on to the next step: reading configuration data from HOCON files by utilizing custom `ConfigProvider`s.

## Step 3: Reading Configuration Data From HOCON Files

ZIO Config library provides various ways read configuration data from different sources, e.g.:
- HOCON files
- JSON files
- YAML files
- XML files

In this tutorial, we will use the HOCON files. [HOCON](https://github.com/lightbend/config) is config format which is superset of JSON developed by Lightbend.

### Adding ZIO Config Dependencies

We should add the following dependencies to our `build.sb` file:

```scala
libraryDependencies += "dev.zio" %% "zio-config"          % "4.0.2"
libraryDependencies += "dev.zio" %% "zio-config-typesafe" % "4.0.2"
libraryDependencies += "dev.zio" %% "zio-config-magnolia" % "4.0.2"
```

### Defining the HOCON Configuration File

We can define our configuration inside `application.conf` file in the `resources` directory:

```json
# application.conf

HttpServerConfig {
  # The port to listen on.
  port = 8080
  port = ${?PORT}

  # The hostname to listen on.
  host = "localhost"
  host = ${?HOST}

  nThreads = 0
  nThreads = ${?N_THREADS}
}
```

HOCON supports substitutions, so in the above configuration, we can use the environment variables `?PORT` and `?HOST` to substitute the values. We also provide a default value for the port and host.

### Changing the Default ConfigProvider to HOCON Provider

To be able to read the configuration data from the HOCON files, we can use the `TypesafeConfigProvider` to read the configuration data from the `application.conf` file:

```scala

Runtime.setConfigProvider(
  TypesafeConfigProvider.fromResourcePath()
)
```

Then we should change the default `ConfigProvider` to the new one by using `Runtime.setConfigProvider` layer:

```scala title="documentation/guides/tutorials/make-a-zio-app-configurable/src/main/scala/dev/zio/quickstart/MainApp.scala" 
package dev.zio.quickstart

object MainApp extends ZIOAppDefault {
  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.setConfigProvider(
      ConfigProvider.fromResourcePath()
    )

  private val serverConfig: ZLayer[Any, Config.Error, Server.Config] =
    ZLayer
      .fromZIO(
        ZIO.config[HttpServerConfig](HttpServerConfig.config).map { c =>
          Server.Config.default.binding(c.host, c.port)
        }
      )

  private val nettyConfig: ZLayer[Any, Config.Error, NettyConfig] =
    ZLayer
      .fromZIO(
        ZIO.config[HttpServerConfig](HttpServerConfig.config).map { c =>
          NettyConfig.default.maxThreads(c.nThreads)
        }
      )

  def run = {
    (Server
      .install(
        GreetingRoutes() ++ DownloadRoutes() ++ CounterRoutes() ++ UserRoutes()
      )
      .flatMap(port =>
        Console.printLine(s"Started server on port: $port")
      ) *> ZIO.never)
      .provide(
        serverConfig,
        nettyConfig,
        Server.live,

        // A layer responsible for storing the state of the `counterApp`
        ZLayer.fromZIO(Ref.make(0)),

        // To use the persistence layer, provide the `PersistentUserRepo.layer` layer instead
        InmemoryUserRepo.layer
      )
  }
}
```

## Step 4: Running The Application

Now, if we run the application, it will start the server using the configuration defined in the `application.conf` file with its default values:

```
$ sbt run
Server started on port: 8080
```

We can set the `HOST` and `PORT` environment variables to override the default values:

```scala
$ HOST=localhost PORT=8081 sbt run
Server started on port: 8081
```

## Conclusion

This tutorial covered how to use ZIO Config to read configuration data from HOCON files and configure our application. We haven't covered all the features of the ZIO Config library. To learn more about this library please visit the [ZIO Config documentation](https://zio.github.io/zio-config/).

The complete working example of this tutorial is available on the `configurable-app` branch of our [ZIO Quickstart: Building RESTful Web Service](https://github.com/zio/zio-quickstart-restful-webservice/tree/configurable-app) quickstart on GitHub.

---

## Tutorial: How to Monitor a ZIO Application Using ZIO's Built-in Metric System?

## Introduction

ZIO has a built-in metric system that allows us to monitor the performance of our application. This is very useful for debugging and tuning our application. In this tutorial, we are going to learn how to add metrics to our application and then how to connect our application to one of the metric backends, e.g. Prometheus.

## Running the Example

To access the code examples, you can clone the [ZIO Quickstarts](http://github.com/zio/zio-quickstarts) project:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-restful-webservice-metrics
```

And finally, run the application using sbt:

```bash
$ sbt run
```

Alternatively, to enable hot-reloading and prevent port binding issues, you can use:

```bash
sbt reStart
```

:::note
If you encounter a "port already in use" error, you can use `sbt-revolver` to manage server restarts more effectively. The `reStart` command will start your server and `reStop` will properly stop it, releasing the port.

To enable this feature, we have included `sbt-revolver` in the project. For more details on this, refer to the [ZIO HTTP documentation on hot-reloading](https://ziohttp.com/installation#hot-reload-changes-watch-mode).
:::

## Trying a Simple Example

Before going to apply the metrics in our application, let's try a simple example:

```scala

object MainApp extends ZIOAppDefault {
  private val count = Metric.counterInt("fib_call_total").fromConst(1)

  def fib(n: Int): ZIO[Any, Nothing, Int] =
    if (n <= 1) ZIO.succeed(1)
    else for {
      a <- fib(n - 1) @@ count
      b <- fib(n - 2) @@ count
    } yield a + b

  def run =
    for {
      i <- Console.readLine("Please enter a number to calculate fibonacci: ").mapAttempt(_.toInt)
      n <- fib(i) @@ count
      _ <- Console.printLine(s"fib($i) = $n")
      c <- count.value
      _ <- ZIO.debug(s"number of fib calls to calculate fib($i): ${c.count}")
    } yield ()
}
```

In this example, we are going to calculate the Fibonacci number for a given number. We also count the number of times we call the `fib` function using the `count` metric. Finally, we will print the value of the metric as a debug message.

This is a pedagogical example of how to use metrics. In real life, we will probably want to poll the metrics using a web API and feed them to a monitoring system, e.g. Prometheus. In the following sections, we will learn how to do that by applying the metrics to our RESTful web service.

## Built-in ZIO HTTP Metrics

ZIO HTTP has built-in support for metrics. We can attach metrics middleware to our HTTP application using the `@@` syntax:

```scala

object UserRoutes {

  def apply(): Routes[UserRepo, Response] =
    Routes(
      Method.GET / "users" -> handler {
        UserRepo.users.foldZIO(
          e =>
            ZIO
              .logError(s"Failed to retrieve users. $e") *>
              ZIO.fail(Response.internalServerError("Cannot retrieve users!")),
          users =>
            ZIO
              .log(
                s"Retrieved users successfully: response length=${users.length}"
              )
              .as(Response(body = Body.from(users)))
        )
      }
    ) @@ Middleware.metrics()
}
```

The `metrics` middleware is attached to all the routes in the `UserRoutes`. Currently, it only counts the number of requests to the `/users` endpoint. We can add more routes to the `UserRoutes` and all of them will be counted by the `metrics` middleware.

After adding the metrics to routes, it is time to serve the metrics as a RESTful API. Before that, let's add the required dependencies to our project.

## Dependencies

In the following sections, we are going to utilize the `zio-metrics-connector` module from the ZIO ZMX project and also provide metrics as a REST API. So let's add the following dependency to our project:

```scala
libraryDependencies += "dev.zio" %% "zio-metrics-connectors"            % "2.5.6"
libraryDependencies += "dev.zio" %% "zio-metrics-connectors-prometheus" % "2.5.6"
```

This module provides various connectors for metrics backend, e.g. Prometheus.

## Serving Prometheus Metrics

The following snippet shows how to provide an HTTP endpoint that exposes the metrics as a REST API for Prometheus:

```scala

object PrometheusPublisherRoutes {
  def apply(): Routes[PrometheusPublisher, Nothing] = {
    Routes(
      Method.GET / "metrics" ->
        handler(
          ZIO.serviceWithZIO[PrometheusPublisher](_.get.map(Response.text))
        )
    )
  }
}
```

Next, we need to add the `PrometheusPublisherRoutes` HTTP App to our application:

```scala

object MainApp extends ZIOAppDefault {
  private val metricsConfig = ZLayer.succeed(MetricsConfig(1.seconds))

  def run =
    Server.serve(
      GreetingRoutes() ++ DownloadRoutes() ++ CounterRoutes() ++ UserRoutes() ++ PrometheusPublisherRoutes()
    ).provide(
      Server.default,

      // An layer responsible for storing the state of the `counterApp`
      ZLayer.fromZIO(Ref.make(0)),

      // To use the persistence layer, provide the `PersistentUserRepo.layer` layer instead
      InmemoryUserRepo.layer,

      // configs for metric backends
      metricsConfig,

      // The prometheus reporting layer
      prometheus.publisherLayer,
      prometheus.prometheusLayer,
    )
}
```

## Testing the Metrics

Now that we have the metrics as a REST API, we can test them. Let's run the application and then send some requests to the application as below:

```bash
$ curl -i http://localhost:8080/users -d '{"name": "John", "age": 42}'
$ curl -i http://localhost:8080/users -d '{"name": "Jane", "age": 43}'
$ curl -i http://localhost:8080/users
```

If we fetch the metrics from the "/metrics" endpoint, we will see the metrics in the Prometheus format, like below:

```bash
$ curl -i http://localhost:8080/metrics
HTTP/1.1 200 OK
content-type: text/plain
date: Tue, 30 Apr 2024 18:58:26 GMT
content-length: 4801

# TYPE http_concurrent_requests_total gauge
# HELP http_concurrent_requests_total
http_concurrent_requests_total{method="GET",path="/users",} 0.0 1714503503829
# TYPE http_concurrent_requests_total gauge
# HELP http_concurrent_requests_total
http_concurrent_requests_total{method="POST",path="/users",} 0.0 1714503503829
# TYPE http_request_duration_seconds histogram
# HELP http_request_duration_seconds
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.005",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.01",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.025",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.05",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.075",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.1",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.25",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.5",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="0.75",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="1.0",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="2.5",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="5.0",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="7.5",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="10.0",} 2.0 1714503503829
http_request_duration_seconds_bucket{method="POST",path="/users",status="200",le="+Inf",} 2.0 1714503503829

http_request_duration_seconds_sum{method="POST",path="/users",status="200",} 0.100570365 1714503503829
http_request_duration_seconds_count{method="POST",path="/users",status="200",} 2.0 1714503503829
http_request_duration_seconds_min{method="POST",path="/users",status="200",} 0.00120463 1714503503829
http_request_duration_seconds_max{method="POST",path="/users",status="200",} 0.099365735 1714503503829
# TYPE http_request_duration_seconds histogram
# HELP http_request_duration_seconds
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.005",} 0.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.01",} 0.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.025",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.05",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.075",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.1",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.25",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.5",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="0.75",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="1.0",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="2.5",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="5.0",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="7.5",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="10.0",} 1.0 1714503503829
http_request_duration_seconds_bucket{method="GET",path="/users",status="200",le="+Inf",} 1.0 1714503503829

http_request_duration_seconds_sum{method="GET",path="/users",status="200",} 0.017157212 1714503503829
http_request_duration_seconds_count{method="GET",path="/users",status="200",} 1.0 1714503503829
http_request_duration_seconds_min{method="GET",path="/users",status="200",} 0.017157212 1714503503829
http_request_duration_seconds_max{method="GET",path="/users",status="200",} 0.017157212 1714503503829
# TYPE http_requests_total counter
# HELP http_requests_total
http_requests_total{method="POST",path="/users",status="200",} 2.0 1714503503829
# TYPE http_requests_total counter
# HELP http_requests_total
http_requests_total{method="GET",path="/users",status="200",} 1.0 1714503503829⏎
```

Now that we have the metrics as a REST API, we can add this endpoint to our Prometheus server to fetch the metrics periodically.

## Conclusion

In this tutorial, we have learned how to define metrics and apply them to our application. We have also learned how to provide the metrics as a REST API which then can be polled by a Prometheus server.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) on GitHub.

---

## Tutorial: How to Run Our First ZIO Project With VSCode?

ZIO is a _type-safe_ library for building asynchronous and concurrent applications. The Scala compiler can catch a lot of errors at compile time since it is type-safe. This ensures that our code is mostly correct, and it prevents lots of bugs from creeping into the runtime. So having a type-safe editor that highlights errors will improve our development experience. VSCode with the power of the _Metals_ extension provides us with a type-safe editor for writing Scala programs.

In this tutorial, we will walk through how to run our first ZIO project with VSCode. We will start by creating a new project and then importing that into the VSCode workspace.

## Requirements

In this article, we assume that we have already downloaded and installed the following tools:
- [SBT build tool](https://www.scala-sbt.org/download.html)
- [VSCode](https://code.visualstudio.com/)

## Creating a new ZIO project

1. First, we need to create a new directory where we should put our project files:

```bash
mkdir my-zio-project
cd my-zio-project
```

2. Next, we need to create a new `build.sbt` file:

```bash
touch build.sbt
```

Inside this file, let's add the following lines which include the version of Scala language, the name of our project, and the dependencies we want to use:

```scala
scalaVersion := "2.13.10"
name := "my-zio-project"
version := "0.0.1"

libraryDependencies ++= Seq(
  "zio" %% "zio" % "2.0.5"
)
```

3. Also, we need to specify the SBT version we want to use in the `project/build.properties` file:

```bash
mkdir project
touch project/build.properties
```

Now, let's add the following lines to the `project/build.properties` file:

```
sbt.version = 1.8.0
```

4. We are ready to open our project in VsCode. We can do this by opening the `my-zio-project` directory from the `File > Open Folder` menu.

5. When we open a project that has a `build.sbt` file, the _Metals_ extension will automatically detect the project and prompt us with the following message:

![VSCode](img/vscode.jpg)

6. By clicking on the `Import Build` button, the _Metals_ will try to download the dependencies from the `build.sbt` file and then import the whole project into the workspace.

7. After that, we are ready to write our first ZIO program by creating a new file called `MainApp.scala` inside the `src/main/scala` directory:

```bash
$ mkdir src/main/scala
$ touch src/main/scala/MainApp.scala
```

Let's add the following lines to our `MainApp.scala` file:

```scala

object MainApp extends ZIOAppDefault {
  def run = 
    for {
      name <- Console.readLine("What is your name? ")
      _    <- Console.printLine(s"Hello, $name!")
    } yield ()
}
```

Now, we can run our program by clicking on the _run_ button above the `MainApp` object.

After we have imported our project using _Metals_, the editor can highlight syntax errors, type-check our code, and run the program. Additionally, it provides code completion that helps us choose the right method. Another handy feature is the ability to jump to the definition of methods and classes just by _Ctrl + Click_ing on them.

## Diagnosing the Build

Sometime, the build may fail. To diagnose the build, the first place to look is the _Metals_ output, which can be found in the _Output Window_ in the VSCode. The output window can be opened by typing _Toggle Output_ in the command palette (`Ctrl + Shift + P`).

The _Metals_ also provide a build diagnostics panel called _Metals Doctor_ which can be opened by typing _Metals: Run doctor_ in the command palette. This command will show us the status of the build and if there are any errors, it will guide us on how to fix them.

## Conclusion

In this tutorial, we have learned how to run a ZIO project with VSCode using the Metals extension. VSCode along with the Metals extension provide lots of handy features that improve the productivity of developers. To learn more about this, we recommend reading the [Metals documentation for VSCode editor](https://scalameta.org/metals/docs/editors/vscode/).

---

## Tutorial: How to Run Our First ZIO Project With IntelliJ IDEA?

## Introduction

IntelliJ IDEA is a popular IDE for Java developers. It is a powerful tool for developing Java applications. Fortunately, not only does it support the Java language, but it also supports many other languages, including Scala.

## Prerequisites

We assume that the reader has already installed IntelliJ IDEA. To download the latest version of IntelliJ IDEA, visit [IntelliJ IDEA download page](https://www.jetbrains.com/idea/download/).

## Installing Scala and ZIO plugins

To add the Scala support for IntelliJ IDEA, we need to install the [Scala plugin](https://plugins.jetbrains.com/plugin/scala). This plugin supports projects that are written in Scala along with the SBT build tool like the ZIO projects.

Besides the Scala plugin, in this tutorial we will also use the [ZIO plugin](https://plugins.jetbrains.com/plugin/zio), to create a new ZIO project from predefined templates.

Both of these plugins are available in the IntelliJ IDEA marketplace. We should follow these steps:
1. Open the IntelliJ IDEA settings by pressing the _Ctrl+Alt+S_ key combination.
2. Click on the _Plugins_ tab.
3. Click on the _Marketplace_ tab.
4. Search for the _Scala_ plugin and click on the _Install_ button.
5. Search for the _ZIO for IntelliJ_ plugin and click on the _Install_ button.
6. Finally, we might need to restart IntelliJ IDEA.

## Creating a New Project

To create a new ZIO project, we will use the _ZIO for IntelliJ_ plugin. This plugin provides a new project template that we can use to create a new ZIO project.

First, click on the _File > New > Project ..._ menu. This will open a dialog that lets us choose the project type:

![IntelliJ IDEA new project dialog](img/intellij-idea-new-project-1.png)

Then, we can choose the _ZIO project (with sbt)_ project type. It will create a new ZIO project with the sbt build tool:

![IntelliJ IDEA new project dialog](img/intellij-idea-new-project-2.png)

It asks us to choose the project name, its location, the ZIO version and other related settings. After entering the project details, we can click on the _Create_ button:

It will create a new ZIO project and open it in the IntelliJ IDEA. From here, we can start working on the project.

## Running the Application

IntelliJ IDEA provides a green run button beside any classes that is executable. So if we open the `src/main/scala/Main.scala` file, we can see the run button. By clicking on that we can see that the IntelliJ IDEA start to compile and run the `Main` class.

> **_Note:_** 
> 
> At the time of writing this tutorial, the _ZIO for IntelliJ_ doesn't provide a right template for ZIO 2.x projects. So let's write our own Main class:
>
> ```scala
> import zio._
> 
> object Main extends ZIOAppDefault {
>   def run = Console.printLine("Hello, world!")
> }
> ```

## Running ZIO Tests

IntelliJ IDEA can run tests in the `test` directory. Let's create a new test class `ExampleSpec` and add it to the `src/test/scala` directory:

```scala

object ExampleSpec extends ZIOSpecDefault {
  def spec =
    suite("spec")(
      test("test1") {
        assertTrue(true)
      },
      test("test2") {
        assertTrue(false)
      }
    )
}
```

The IntelliJ IDEA detects the tests and places a run button beside them. So we can click on the run button, and it will run the tests:

![IntelliJ IDEA test output](img/intellij-idea-test-runner-output.png)

## Conclusion

In this tutorial, we have learned how to create a new ZIO project in IntelliJ IDEA. Then we learned how to run ZIO applications and ZIO tests in IntelliJ IDEA.

---

## Caliban

[Caliban](https://ghostdogpr.github.io/caliban/) is a purely functional library for creating GraphQL servers and clients in Scala.

## Introduction

Key features of Caliban
- **Purely Functional** — All interfaces are pure and types are referentially transparent.
- **Type Safety** — Schemas are type safe and derived at compile time.
- **Minimal Boilerplate** — No need to manually define a schema for every type in your API.
- **Excellent Interoperability** — Out-of-the-box support for major HTTP server libraries, effect types, JSON libraries, and more.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "com.github.ghostdogpr" %% "caliban"       % "2.8.1"
libraryDependencies += "com.github.ghostdogpr" %% "caliban-quick" % "2.8.1" // Optional: HTTP routes via ZIO HTTP
```

Caliban also have lots of optional modules to interoperate with other various libraries:

```scala
libraryDependencies += "com.github.ghostdogpr" %% "caliban-http4s"     % "2.8.1" // routes for http4s
libraryDependencies += "com.github.ghostdogpr" %% "caliban-akka-http"  % "2.8.1" // routes for akka-http
libraryDependencies += "com.github.ghostdogpr" %% "caliban-play"       % "2.8.1" // routes for play
libraryDependencies += "com.github.ghostdogpr" %% "caliban-finch"      % "2.8.1" // routes for finch
libraryDependencies += "com.github.ghostdogpr" %% "caliban-cats"       % "2.8.1" // interop with cats effect
libraryDependencies += "com.github.ghostdogpr" %% "caliban-monix"      % "2.8.1" // interop with monix
libraryDependencies += "com.github.ghostdogpr" %% "caliban-tapir"      % "2.8.1" // interop with tapir
libraryDependencies += "com.github.ghostdogpr" %% "caliban-federation" % "2.8.1" // interop with apollo federation
```

## Example

First, to define Caliban API, we should define data models using case classes and ADTs. Then the Caliban can derive the whole GraphQL schema from these data models:

```scala

sealed trait Role

object Role {
  case object SoftwareDeveloper       extends Role
  case object SiteReliabilityEngineer extends Role
  case object DevOps                  extends Role
}

case class Employee(
    name: String,
    role: Role
)

case class EmployeesArgs(role: Role)
case class EmployeeArgs(name: String)

case class Queries(
    @GQLDescription("Return all employees with specific role")
    employees: EmployeesArgs => List[Employee],
    @GQLDescription("Find an employee by its name")
    employee: EmployeeArgs => Option[Employee]
)

object CalibanExample extends zio.ZIOAppDefault {

  val employees = List(
    Employee("Alex", Role.DevOps),
    Employee("Maria", Role.SoftwareDeveloper),
    Employee("James", Role.SiteReliabilityEngineer),
    Employee("Peter", Role.SoftwareDeveloper),
    Employee("Julia", Role.SiteReliabilityEngineer),
    Employee("Roberta", Role.DevOps)
  )

  override def run =
    graphQL(
      RootResolver(
        Queries(
          args => employees.filter(e => args.role == e.role),
          args => employees.find(e => e.name == args.name)
        )
      )
    ).runServer(
      port = 8088,
      apiPath = "/api/graphql",
      graphiqlPath = Some("/api/graphiql"),
    )
}
```

Now let's query all software developers using GraphQL query language:

```graphql
query{
  employees(role: SoftwareDeveloper){
    name
    role
  }
}
```

Here is the _curl_ request of this query:

```bash
curl 'http://localhost:8088/api/graphql' --data-binary '{"query":"query{\n employees(role: SoftwareDeveloper){\n name\n role\n}\n}"}'
```

And the response:

```json
{
  "data" : {
    "employees" : [
      {
        "name" : "Maria",
        "role" : "SoftwareDeveloper"
      },
      {
        "name" : "Peter",
        "role" : "SoftwareDeveloper"
      }
    ]
  }
}
```

## Resources

- [Zymposium - Caliban](https://www.youtube.com/watch?v=mzqsXklbmfM&t=2849s) by Adam Fraser, Kit Langton and Pierre Ricadat — We will be building full-stack applications with the special guest international man of mystery Pierre Ricadat. Pierre will be walking us through Caliban, a GraphQL library in Scala that just had its 1.0 release. We'll be seeing how easy it can be to upgrade an existing application to use GraphQL with the power of Caliban!

---

## Distage

[Distage](https://izumi.7mind.io/distage/) is a compile-time safe, transparent, and debuggable Dependency Injection framework for pure FP Scala.

## Introduction

By using _Distage_ we can auto-wire all components of our application.
- We don't need to manually link components together
- We don't need to manually specify the order of allocation and allocation of dependencies. This will be derived automatically from the dependency order.
- We can override any component within the dependency graph.
- It helps us to create different configurations of our components for different use cases.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "io.7mind.izumi" %% "distage-core" % "1.0.8"
```

## Example

In this example we create a `RandomApp` comprising two `Random` and `Logger` services. By using `ModuleDef` we _bind_ services to their implementations:

```scala

trait Random {
  def nextInteger: UIO[Int]
}

final class ScalaRandom extends Random {
  override def nextInteger: UIO[Int] =
    ZIO.effectTotal(scala.util.Random.nextInt())
}

trait Logger {
  def log(name: String): Task[Unit]
}

final class ConsoleLogger extends Logger {
  override def log(line: String): Task[Unit] = {
    val timeStamp = LocalDateTime.now()
    ZIO.effect(println(s"$timeStamp: $line"))
  }
}

final class RandomApp(random: Random, logger: Logger) {
  def run: Task[Unit] = for {
    random <- random.nextInteger
    _ <- logger.log(s"random number generated: $random")
  } yield ()
}

object DistageExample extends zio.App {
  def RandomAppModule: ModuleDef = new ModuleDef {
    make[Random].from[ScalaRandom]
    make[Logger].from[ConsoleLogger]
    make[RandomApp] // `.from` is not required for concrete classes
  }
  
  val resource: Lifecycle[Task, Locator] = Injector[Task]().produce(
    plan = Injector[Task]().plan(
      bindings = RandomAppModule,
      activation = Activation.empty,
      roots = Roots.target[RandomApp]
    )
  )

  val myApp: Task[Unit] = resource.use(locator => locator.get[RandomApp].run)

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.exitCode
}
```

---

## Fhir-indexer

[Fhir-indexer](https://github.com/royashcenazi/fhir-indexer) is a ZIO based library for fetching FHIR resources fast and easy.

## Introduction

Welcome to the FHIR Indexing with ZIO and HAPI FHIR repository! This open-source project aims to provide a robust and efficient solution for indexing FHIR (Fast Healthcare Interoperability Resources) based APIs. The project leverages the power of the ZIO framework to handle concurrency and asynchronicity, while wrapping around the widely-used HAPI FHIR library for working with FHIR resources.

## Features

- **Asynchronous and Concurrent Indexing**: The project takes advantage of ZIO's powerful concurrency model to efficiently index FHIR resources from multiple APIs simultaneously.

- **Modular and Extensible Design**: The codebase is structured in a modular way, allowing easy extension and customization to fit different FHIR-based API implementations.

- **Authentication Support**: The library supports various authentication mechanisms for accessing secure FHIR endpoints, ensuring data privacy and security.

- **Resource Filtering and Transformation**: Users can define custom filters and transformations to tailor the indexing process according to their specific requirements.

## Installation

### SBT

`libraryDependencies += "io.github.royashcenazi" % "fhir-indexer" % "v0.0.2"`

### Maven

```xml
   <dependency>
        <groupId>io.github.royashcenazi</groupId>
        <artifactId>fhir-indexer</artifactId>
        <version>v0.0.2</version>
    </dependency>
```

## Example

```scala
object ZioApp extends ZIOAppDefault {

  class ConditionIndexer(override val hapiFhirClient: FHIRHapiClient) extends ApiIndexer {
    override type ProcessResult = Unit

    override def indexApi[T <: IParam](searchMetadata: RequestMetadata[T]): ZIO[Any, Throwable, Unit] = {
      for {
        data <- createPagedApiIndexingFlow[Encounter, T](searchMetadata)
        _ <- ZIO.log(s"Processed api: ${getClass.getSimpleName} with ${data.size} entries")
      } yield ()
    }
  }

  object ConditionIndexer {
    val layer: ZLayer[FHIRHapiClient, Nothing, ConditionIndexer] = ZLayer {
      for {
        hapiClient <- ZIO.service[FHIRHapiClient]
      } yield new ConditionIndexer(hapiClient)
    }
  }

  def myApp: ZIO[ConditionIndexer, Nothing, Exit[Throwable, Unit]] = {
    val requestMetadata = RequestMetadata[TokenClientParam]("")
    for {
      conditionsService <- ZIO.service[ConditionIndexer]
      zio = conditionsService.indexApi[TokenClientParam](requestMetadata)
    } yield {
      Unsafe.unsafe { implicit unsafe =>
        Runtime.default.unsafe.run(zio)
      }
    }
  }

  object FhirConfigTest {
    val layer: zio.ZLayer[Any, Nothing, FhirIndexingConfig] =
      zio.ZLayer.succeed(FhirIndexingConfig(url = "",
        authUrl ="/oauth2/token",
        clientId = "",
        secret = ""))
  }

  override def run: ZIO[Any, Throwable, Exit[Throwable, Unit]] = {
    myApp.debug("example")
      .provide(
        Client.default,
        FhirConfigTest.layer,
        FhirAuthClientImpl.layer(),
        FHIRHapiClientImpl.layer(),
        ConditionIndexer.layer
      )
  }
}
```

Happy indexing!

---

## ZIO Ecosystem Community Libraries

In this section we are going to introduce some of the most important libraries that have first-class ZIO support from the community.

If you know a useful library that has first-class ZIO support, please consider [submitting a pull request](https://github.com/zio/zio/pulls) to add it to this list.

<DocCardList />

---

## LogStage

[LogStage](https://izumi.7mind.io/logstage/) is a zero-cost structural logging framework for Scala & Scala.js.

## Introduction

Some key features of _LogStage_:

1. LogStage extracts structure from ordinary string interpolations in your log messages with zero changes to code.
2. LogStage uses macros to extract log structure, it is faster at runtime than a typical reflective structural logging frameworks
3. Log contexts
4. Console, File, and SLF4J sinks included, File sink supports log rotation,
5. Human-readable output and JSON output included,
6. Method-level logging granularity. Can configure methods com.example.Service.start and com.example.Service.doSomething independently,
7. Slf4J adapters: route legacy Slf4J logs into LogStage router

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
// LogStage core library
libraryDependencies += "io.7mind.izumi" %% "logstage-core" % "1.0.8"
```

There are also some optional modules:

```scala
libraryDependencies ++= Seq(
  // Json output
  "io.7mind.izumi" %% "logstage-rendering-circe" % "1.0.8",
  // Router from Slf4j to LogStage
  "io.7mind.izumi" %% "logstage-adapter-slf4j" % "1.0.8",
  // LogStage integration with DIStage
  "io.7mind.izumi" %% "distage-extension-logstage" % "1.0.8",
  // Router from LogStage to Slf4J
  "io.7mind.izumi" %% "logstage-sink-slf4j " % "1.0.8",
)
```

## Example

Let's try a simple example of using _LogStage_:

```scala

object LogStageExample extends zio.App {
  val myApp = for {
    _ <- log.info("I'm logging with logstage!")
    userId = UUIDGen.getTimeUUID()
    _ <- log.info(s"Current $userId")
    _ <- log.info("I'm logging within the same fiber!")
    f <- log.info("I'm logging within a new fiber!").fork
    _ <- f.join
  } yield ()

  val loggerLayer: ULayer[Has[LogIO2[IO]]] =
    ZLayer.succeed(LogZIO.withFiberId(IzLogger()))

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.provide(loggerLayer).exitCode
}
```

The output of this program would be something like this:

```
I 2021-07-26T21:27:35.164 (LogStageExample.scala:8)  …mpty>.LogStageExample.myApp [14:zio-default-async-1] fiberId=Id(1627318654646,1) I'm logging with logstage!
I 2021-07-26T21:27:35.252 (LogStageExample.scala:10)  <.LogStageExample.myApp.8 [14:zio-default-async-1] fiberId=Id(1627318654646,1) Current userId=93546810-ee32-11eb-a393-11bc5b145beb
I 2021-07-26T21:27:35.266 (LogStageExample.scala:11)  <.L.myApp.8.10 [14:zio-default-async-1] fiberId=Id(1627318654646,1) I'm logging within the same fiber!
I 2021-07-26T21:27:35.270 (LogStageExample.scala:12)  <.L.m.8.10.11 [16:zio-default-async-2] fiberId=Id(1627318655269,2) I'm logging within a new fiber!
```

---

## MUnit ZIO

[MUnit ZIO](https://github.com/poslegm/munit-zio) is an integration library between MUnit and ZIO.

## Introduction

[MUnit](https://scalameta.org/munit/) is a Scala testing library that is implemented as a JUnit runner. It has _actionable errors_, so the test reports are colorfully pretty-printed, stack traces are highlighted, error messages are pointed to the source code location where the failure happened.

The MUnit ZIO enables us to write tests that return `ZIO` values without needing to call any unsafe methods (e.g. `Runtime#unsafeRun`).

## Installation

In order to use this library, we need to add the following lines in our `build.sbt` file:

```scala
libraryDependencies += "org.scalameta" %% "munit" % "0.7.27" % Test
libraryDependencies += "com.github.poslegm" %% "munit-zio" % "0.0.2" % Test
```

If we are using a version of sbt lower than 1.5.0, we will also need to add:

```scala
testFrameworks += new TestFramework("munit.Framework")
```

## Example

Here is a simple MUnit spec that is integrated with the `ZIO` effect:

```scala

class SimpleZIOSpec extends ZSuite {
  testZ("1 + 1 = 2") {
    for {
      a <- ZIO(1)
      b <- ZIO(1)
    }
    yield assertEquals(a + b, 2)
  }
}
```

---

## Rezilience

[Rezilience](https://github.com/svroonland/rezilience) is a ZIO-native library for making resilient distributed systems.

## Introduction

Rezilience is a ZIO-native fault tolerance library with a collection of policies for making asynchronous systems more resilient to failures inspired by Polly, Resilience4J, and Akka. It does not have external library dependencies other than ZIO.

It comprises these policies:
- **CircuitBreaker** — Temporarily prevent trying calls after too many failures
- **RateLimiter** — Limit the rate of calls to a system
- **Bulkhead** — Limit the number of in-flight calls to a system
- **Retry** — Try again after transient failures
- **Timeout** — Interrupt execution if a call does not complete in time

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "nl.vroste" %% "rezilience" % "<version>"
```

## Example

Let's try an example of writing _Circuit Breaker_ policy for calling an external API:

```scala

object CircuitBreakerExample extends zio.App {

  def callExternalSystem: ZIO[Console, String, Nothing] =
    putStrLn("External service called, but failed!").orDie *>
      ZIO.fail("External service failed!")

  val myApp: ZIO[Console with Clock, Nothing, Unit] =
    CircuitBreaker.withMaxFailures(
      maxFailures = 10,
      resetPolicy = Schedule.exponential(1.second),
      onStateChange = (state: State) =>
        ZIO(println(s"State changed to $state")).orDie
    ).use { cb =>
      for {
        _ <- ZIO.foreach_(1 to 10)(_ => cb(callExternalSystem).either)
        _ <- cb(callExternalSystem).catchAll(errorHandler)
        _ <- ZIO.sleep(2.seconds)
        _ <- cb(callExternalSystem).catchAll(errorHandler)
      } yield ()
    }

  def errorHandler: CircuitBreakerCallError[String] => URIO[Console, Unit] = {
    case CircuitBreaker.CircuitBreakerOpen =>
      putStrLn("Circuit breaker blocked the call to our external system").orDie
    case CircuitBreaker.WrappedError(error) =>
      putStrLn(s"External system threw an exception: $error").orDie
  }
  
  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.exitCode
}
```

---

## Scala k8s

[Scala k8s](https://github.com/hnaderi/scala-k8s) is a Kubernetes client, data models and typesafe manifest generation for scala, scalajs, and scala native.

## Introduction
This library provides a full blown extensible client that you can use to interact directly with kubernetes API server, to create operators or accomplish other automation tasks, also you can use it to create or manipulate manifests in scala.

### Design principles
- As extensible as possible
- As dependency free as possible
- As Un-opinionated as possible
- Provide seamless integrations
- All specs are generated from the spec directly and will be in sync with kubernetes all the time

## Installation

This library is currently available for Scala binary versions 2.12, 2.13 and 3.2 on JVM/JS/Native.  
This library is architecured in a microkernel fashion and all the main kubernetes stuff are implemented/generated in pure scala, and integration modules are provided separately.  
main modules are:

- `objects` raw kubernetes objects, which has no dependency
- `client` raw kubernetes client and requests, requests can also be extended in user land easily!

Latest version [![Latest version](https://index.scala-lang.org/hnaderi/scala-k8s/scala-k8s-objects/latest-by-scala-version.svg?style=flat-square)](https://index.scala-lang.org/hnaderi/scala-k8s/scala-k8s-objects)

``` scala
libraryDependencies ++= Seq(
  "dev.hnaderi" %% "scala-k8s-objects" % "[VERSION]", // JVM, JS, Native ; raw k8s objects
  "dev.hnaderi" %% "scala-k8s-client" % "[VERSION]", // JVM, JS, Native ; k8s client kernel and requests
  )
```

The following integrations are currently available:  

```scala
libraryDependencies ++= Seq(
  "dev.hnaderi" %% "scala-k8s-http4s" % "[VERSION]", // JVM, JS, Native ; http4s and fs2 integration
  "dev.hnaderi" %% "scala-k8s-zio" % "[VERSION]", // JVM ; ZIO native integration using zio-http and zio-json 
  "dev.hnaderi" %% "scala-k8s-sttp" % "[VERSION]", // JVM, JS, Native ; sttp integration using jawn parser
  "dev.hnaderi" %% "scala-k8s-circe" % "[VERSION]", // JVM, JS ; circe integration
  "dev.hnaderi" %% "scala-k8s-json4s" % "[VERSION]", // JVM, JS, Native; json4s integration
  "dev.hnaderi" %% "scala-k8s-spray-json" % "[VERSION]", // JVM ; spray-json integration
  "dev.hnaderi" %% "scala-k8s-play-json" % "[VERSION]", // JVM ; play-json integration
  "dev.hnaderi" %% "scala-k8s-zio-json" % "[VERSION]", // JVM, JS ; zio-json integration
  "dev.hnaderi" %% "scala-k8s-jawn" % "[VERSION]", // JVM, JS, Native ; jawn integration
  "dev.hnaderi" %% "scala-k8s-manifests" % "[VERSION]", // JVM ; yaml manifest generation
  "dev.hnaderi" %% "scala-k8s-scalacheck" % "[VERSION]" // JVM, JS, Native; scalacheck instances
)
```
This project also has a [sbt integration](https://github.com/hnaderi/sbt-k8s) you can use to generate manifests.

## Example

We can define any kubernetes object:

```scala

val config = ConfigMap(
  metadata = ObjectMeta(
    name = "example",
    namespace = "staging",
    labels = Map(
      Labels.name("example"),
      Labels.instance("one")
    )
  ),
  data = DataMap(
    "some config" -> "some value",
    "config file" -> Data.file("config.json")
  ),
  binaryData = DataMap.binary(
    "blob" -> Data.file("blob.dat"),
    "blob2" -> Paths.get("blob2.bin"),
    "other inline data" -> "some other data"
  )
)
```

or use `kubectl` style helpers:

```scala
val config2 = ConfigMap(
  data = DataMap.fromDir(new File("path/to/data-directory"))
)
```

And we can connect to our cluster and send requests:

```scala

// This example uses `kubectl proxy` to simplify authentication
val client = ZIOKubernetesClient.make("http://localhost:8001")
val nodes = ZIOKubernetesClient.send(APIs.nodes.list())
```

Runnable example:
```scala

//NOTE run `kubectl proxy` before running this example
object ZIOExample extends ZIOAppDefault {
  private val app = ZIO.scoped {
    for {
      n <- ZIOKubernetesClient.send(APIs.nodes.list())
      _ <- ZIO.foreach(n.items.map(_.metadata.flatMap(_.name)))(Console.printLine(_))
    } yield ()
  }

  override def run =
    app.provide(
      Client.default,
      ZIOBackend.make,
      ZIOKubernetesClient.make("http://localhost:8001")
    )

}
```

---

## Tamer


 [Tamer](https://github.com/laserdisc-io/tamer) is a multi-functional Kafka connector for producing data based on [ZIO Kafka](https://github.com/zio/zio-kafka).

 ### Introduction

 Tamer is a completely customizable source connector that produces to Kafka. It ships with preconfigured modules for SQL, cloud storage and REST API, but you can provide your own functions and Tamer will take care of the rest.

 ### Installation

 Depending on the source you have at hand you can add the correct dependency in your `build.sbt`:

 ```scala
 libraryDependencies += "io.laserdisc" %% "tamer-db"                % "0.16.1"
 libraryDependencies += "io.laserdisc" %% "tamer-oci-objectstorage" % "0.16.1"
 libraryDependencies += "io.laserdisc" %% "tamer-rest"              % "0.16.1"
 libraryDependencies += "io.laserdisc" %% "tamer-s3"                % "0.16.1"
 ```

 ### Example

 Let's say you have a inventory DB that's compatible with [Doobie](https://github.com/tpolecat/doobie), you can get all of your items with just a few lines of code:

 ```scala

 case class Row(id: String, name: String, description: Option[String], modifiedAt: Instant)
     extends tamer.db.Timestamped(modifiedAt)

 object Row {
   implicit val rowJsonCodec = DeriveJsonCodec.gen[Row]
 }

 object DatabaseSimple extends zio.App {
   // Here we'll go with zio-json codec, you can use avro, circe and jsoniter
   // out-of-the box or plug yours!
   implicit val stateKeyJsonCodec = DeriveJsonCodec.gen[tamer.Tamer.StateKey]
   implicit val windowJsonCodec = DeriveJsonCodec.gen[tamer.db.Window]

   val program: RIO[ZEnv, Unit] = tamer.db.DbSetup
     .tumbling(window =>
       sql"""SELECT id, name, description, modified_at 
            |FROM users 
            |WHERE modified_at > ${window.from} AND modified_at <= ${window.to}""".stripMargin
         .query[Row]
     )(
       recordKey = (_, v) => v.id,
       from = Instant.parse("2020-01-01T00:00:00.00Z"),
       tumblingStep = 5.days
     )
     .runWith(dbLayerFromEnvironment ++ tamer.kafkaConfigFromEnvironment)

   override final def run(args: List[String]): URIO[ZEnv, ExitCode] =
     program.exitCode

   // If you have other codecs like circe in the classpath you have to disambiguate
   implicit lazy val stateKeyCodec: Codec[Tamer.StateKey] = Codec.optionalZioJsonCodec
   implicit lazy val windowCodec: Codec[tamer.db.Window] = Codec.optionalZioJsonCodec
   implicit lazy val stringCodec: Codec[String] = Codec.optionalZioJsonCodec
 }
 ```
 See full example [on the GitHub repo](https://github.com/laserdisc-io/tamer/blob/4e1a7646fb44041648d9aa3ba089decb81ebe487/example/src/main/scala/tamer/db/DatabaseSimple.scala)

---

## Tofu ZIO 2 Logging

[Tofu](https://docs.tofu.tf/) is a functional toolkit modules providing a comprehensive set of tools adressing
real-world problems through the prism of the tagless-final approach and functional programming in general.
Tofu consists of several independent modules, one of them is [Tofu Logging](https://docs.tofu.tf/docs/tofu.logging.home)
, which provides first-class [ZIO support](https://docs.tofu.tf/docs/tofu.logging.recipes.zio2).

Key features of Tofu Logging:

- **100% structured logging**: you can easily log json-s with nested objects, arrays, numeric and boolean fields.
- logging context: implemented on top of `FiberRef`.
- built upon [Logback](https://logback.qos.ch/),
  supports [logstash-logback-encoder](https://github.com/logfellow/logstash-logback-encoder).

## Installation

Add the following lines in your `build.sbt` file:

```scala
libraryDependencies += "tf.tofu" %% "tofu-zio2-logging" % "version"
```

For `Loggable` derivation (see below):

```scala
libraryDependencies += "tf.tofu" %% "tofu-logging-derivation" % "version"
```

And choose a logback layout: Tofu's own implementation or logstash-encoder. See
the [logback configuration](#logback-layout) section.

```scala
libraryDependencies += "tf.tofu" %% "tofu-logging-logstash-logback" % "version"
// OR:
libraryDependencies += "tf.tofu" %% "tofu-logging-layout" % "version"
```

## Quick start

To use Tofu with ZIO logging facade just add `TofuZLogger` to your app runtime:

```scala

object Main extends ZIOAppDefault {
  val program: UIO[Unit] = ZIO.log("Hello, ZIO logging!")

  override def run = {
    program.logSpan("full_app") @@ ZIOAspect.annotated("foo", "bar")
  }.provide(
    Runtime.removeDefaultLoggers,
    TofuZLogger.addToRuntime
  )
}
```

The log message will be:

```json
{
  "level": "INFO",
  "logger_name": "my.package.Main",
  "message": "Hello, ZIO logging!",
  "zSpans": {
    "full_app": 534
  },
  "zAnnotations": {
    "foo": "bar"
  }
}
```

* __logger_name__ is parsed from `zio.Trace` which contains the location where log method is called
* all `zio.LogSpan` are collected in the json object named __zSpans__
* all `zio.LogAnnotation` are collected in the json object named __zAnnotations__ (to avoid conflicts with Tofu
  annotations)

### ZLogAnnotation and TofuDefaultContext

`ZLogAnnotation` allows you to store typed, structured data on the default logging context (implemented on `FiberRef`).
It also solves another problem:
having a list of annotations in front of your eyes, you can easily make sure that the same names are not assigned to
different values (which can lead to an invalid json and corrupt your structured logs).
That's why we recommend to group all your log annotations in one place.

`TofuDefaultContext` is a service which allows to look up an element from the context added via `ZLogAnnotation`:

```scala
trait TofuDefaultContext {
  def getValue[A](key: LogAnnotation[A]): UIO[Option[A]]
}
```

It has two implementations:

- `TofuDefaultContext.layerZioContextOff: ULayer[TofuDefaultContext]`
- `TofuDefaultContext.layerZioContextOn: ULayer[TofuDefaultContext]`

It doesn't matter which one to use with ZIO Logging facade. The difference will be only when working
with Tofu's own loggers and using `TofuDefaultContext` as
a [ContextProvider](https://docs.tofu.tf/docs/tofu.logging.recipes.zio2#contextprovider)
(this feature is Tofu-specific, and we won't cover it here).

```scala
val httpCode: ZLogAnnotation[Int] = ZLogAnnotation.make("httpCode")

val program: UIO[Unit] = {
  for {
    _ <- ZIO.log("Hello, ZIO logging!")
    maybeCode <- ZIO.serviceWithZIO[TofuDefaultContext](_.getValue(httpCode)) // Some(204)
  } yield ()
}.provide(TofuDefaultContext.layerZioContextOn) @@ httpCode(204) @@ ZLogAnnotation.loggerName("MyLogger")
```

will produce:

```json
{
  "level": "INFO",
  "logger_name": "MyLogger",
  "message": "Hello, ZIO logging!",
  "httpCode": 204
}
```

You can change the logger name via `ZLogAnnotation.loggerName`.

`ZLogAnnotation.make[A](name: String)` implicitly requires a `Loggable[A]` instance.

### Loggable

`Loggable[A]` is a typeclass that describes how a value of some type can be logged.
Given an instance of `Loggable` for a type, a value of the type can be converted into the final internal representation
called `LoggedValue` and thus logged in a way that you provided.
There are multiple predefined ways to create an instance of `Loggable`, many of them can be found
in `tofu.logging.Loggable` object:

- provided instances for all primitive types, as well as stdlib's collections and collections from Cats
- `Loggable.empty` for no-op logging of value
- `Loggable.either` for logging either of A and B
- provided `java.time.*` instances

Of course, you can describe your `Loggable` instance yourself:

- by extending trait `DictLoggable` for multi-field objects
- using `Loggable[A]#contramap[B](f: B => A)` method
- using configurable auto derivation

Tofu has a macros that allows you to easily derive instances of `Loggable[YourClass]` for case classes or ADTs.
In additional, there are several annotations to configure generation of `Loggable`:

- `@hidden`: when applied to the field means "do not log\write this field"
- `@masked`: when applied to the field means "mask field value with given mode"
- `@unembed`: when applied to the field means "log subfields along with fields of owner"

```scala

case class User(
  id: Int,
  @hidden
  password: String,
  @masked(MaskMode.ForLength(3))
  login: String,
  godMode: Boolean = false
)

val user = User(100, "secret", "username")
val userAnnotation: ZLogAnnotation[User] = ZLogAnnotation.make("user")

val program: UIO[Unit] =
  ZIO.log("Hello, ZIO logging!") @@ userAnnotation(user)
```

The output of this program will be:

```json
{
  "level": "INFO",
  "logger_name": "my.package.Main",
  "message": "Hello, ZIO logging!",
  "user": {
    "id": 100,
    "login": "use*****",
    "godMode": false
  }
}
```

Read more on [the website](https://docs.tofu.tf/docs/tofu.logging.loggable).

## Logback layout

Tofu has a [logstash-logback-encoder](https://github.com/logfellow/logstash-logback-encoder) provider
named `tofu.logging.TofuLoggingProvider`.
Enter the following content into the `logback.xml` file to get JSON logs:

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<configuration>
    <appender name="logstash" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <pattern>
                    <pattern>
                        { "env": "prod" } <!-- optional static fields -->
                    </pattern>
                </pattern>
                <timestamp/>
                <logLevel/>
                <loggerName/>
                <message/>
                <provider class="tofu.logging.TofuLoggingProvider"/>
            </providers>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="logstash"/>
    </root>

</configuration>
```

Read more about logback layouts configuration on [the website](https://docs.tofu.tf/docs/tofu.logging.layouts).

---

## TranzactIO

[TranzactIO](https://github.com/gaelrenoux/tranzactio) is a ZIO wrapper for some Scala database access libraries, currently for [Doobie](https://github.com/tpolecat/doobie) and [Anorm](https://github.com/playframework/anorm).

## Introduction

Using functional effect database access libraries like _Doobie_ enforces us to use their specialized monads like `ConnectionIO` for _Doobie_. The goal of _TranzactIO_ is to provide seamless integration with these libraries to help us to stay in the `ZIO` world.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "io.github.gaelrenoux" %% "tranzactio" % "2.1.0"
```

In addition, we need to declare the database access library we are using. For example, for the next example we need to add following dependencies for _Doobie_ integration:

```scala
libraryDependencies += "org.tpolecat" %% "doobie-core" % "0.13.4"
libraryDependencies += "org.tpolecat" %% "doobie-h2"   % "0.13.4"
```

## Example

Let's try an example of simple _Doobie_ program:

```scala

object TranzactIOExample extends zio.App {

  val query: ZIO[Connection with Console, Throwable, Unit] = for {
    _ <- PersonQuery.setup
    _ <- PersonQuery.insert(Person("William", "Stewart"))
    _ <- PersonQuery.insert(Person("Michelle", "Streeter"))
    _ <- PersonQuery.insert(Person("Johnathon", "Martinez"))
    users <- PersonQuery.list
    _ <- putStrLn(users.toString)
  } yield ()

  val myApp: ZIO[zio.ZEnv, Throwable, Unit] =
    Database.transactionOrWidenR(query).provideCustom(services.database)

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.exitCode
}

case class Person(firstName: String, lastName: String)

object PersonQuery {
  def list: TranzactIO[List[Person]] = tzio {
    sql"""SELECT first_name, last_name FROM person""".query[Person].to[List]
  }

  def setup: TranzactIO[Unit] = tzio {
    sql"""
        CREATE TABLE person (
          first_name VARCHAR NOT NULL,
          last_name VARCHAR NOT NULL
        )
        """.update.run.map(_ => ())
  }

  def insert(p: Person): TranzactIO[Unit] = tzio {
    sql"""INSERT INTO person (first_name, last_name) VALUES (${p.firstName}, ${p.lastName})""".update.run
      .map(_ => ())
  }
}

object services {
  val datasource: ZLayer[Blocking, Throwable, Has[DataSource]] =
    ZLayer.fromEffect(
      blocking.effectBlocking {
        val ds = new JdbcDataSource
        ds.setURL(s"jdbc:h2:mem:mydb;DB_CLOSE_DELAY=10")
        ds.setUser("sa")
        ds.setPassword("sa")
        ds
      }
    )

  val database: ZLayer[Any, Throwable, doobie.Database.Database] =
    (Blocking.live >>> datasource ++ Blocking.live ++ Clock.live) >>> Database.fromDatasource
}
```

---

## ZIO AMQP

[ZIO AMQP](https://github.com/svroonland/zio-amqp) is a ZIO-based AMQP client for Scala.

## Introduction

ZIO AMQP is a ZIO-based wrapper around the RabbitMQ client. It provides a streaming interface to AMQP queues and helps to prevent us from shooting ourselves in the foot with thread-safety issues.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "nl.vroste" %% "zio-amqp" % "0.3.0"
```

## Example

First, let's create an instance of RabbitMQ:

```
docker run -d --name some-rabbit -p 5672:5672 -p 5673:5673 -p 15672:15672 rabbitmq:3-management
```

Then we need to create `my_exchange` and `my_queue` and bind the queue to the exchange via the RabbitMQ management dashboard (`localhost:15672`).

Now we can run the example below:

```scala

object ZIOAMQPExample extends ZIOAppDefault {

  val channel: ZIO[Scope, Throwable, Channel] = for {
    connection <- Amqp.connect(URI.create("amqp://localhost:5672"))
    channel    <- Amqp.createChannel(connection)
  } yield channel

  val myApp: ZIO[Any, Throwable, Unit] =
    ZIO.scoped {
      for {
        channel                            <- channel
        producer: ZIO[Any, Throwable, Long] =
          Random.nextUUID
            .flatMap(uuid => channel.publish(ExchangeName("my_exchange"), uuid.toString.getBytes).unit)
            .schedule(Schedule.spaced(1.seconds))

        consumer: ZIO[Any, Throwable, Unit] =
          channel
            .consume(queue = QueueName("my_queue"), consumerTag = ConsumerTag("my_consumer"))
            .mapZIO { record =>
              val deliveryTag = record.getEnvelope.getDeliveryTag
              Console.printLine(s"Received $deliveryTag: ${new String(record.getBody)}") *>
                channel.ack(DeliveryTag(deliveryTag))
            }
            .runDrain
        p                                  <- producer.fork
        c                                  <- consumer.fork
        _                                  <- p.zip(c).join
      } yield ()
    }

  override def run: ZIO[Environment with ZIOAppArgs with Scope, Any, Any] = myApp
}
```

---

## ZIO Apache Parquet

[ZIO Apache Parquet](https://github.com/grouzen/zio-apache-parquet) is a ZIO-powered Apache Parquet library.

## Introduction

ZIO Apache Parquet is a ZIO wrapper for [parquet-java](https://github.com/apache/parquet-java) and [hadoop](https://github.com/apache/hadoop) to easily read and write Parquet files without running a Spark cluster.

## Features

- **ZIO & ZIO Streams** — naturally integrates with ZIO ecosystem.
- **ZIO Schema** — type-safe filter predicates and codecs derivation.
- **Clean and simple API** — easy to use, no magic.

## Installation

```scala
libraryDependencies += "me.mnedokushev" %% "zio-apache-parquet-core" % "0.1.0"
```

## Example

A simple demo showcasing how to write a small chunk of data and read it applying a filter predicate.

```scala
//> using scala "3.5.0"
//> using dep me.mnedokushev::zio-apache-parquet-core:0.1.0

object Filtering extends ZIOAppDefault:

  case class MyRecord(a: Int, b: String, c: Option[Long])

  object MyRecord:
    // We need to provide field names using singleton types
    given Schema.CaseClass3.WithFields["a", "b", "c", Int, String, Option[Long], MyRecord] =
      DeriveSchema.gen[MyRecord]
    given SchemaEncoder[MyRecord]                                                          =
      Derive.derive[SchemaEncoder, MyRecord](SchemaEncoderDeriver.default)
    given ValueEncoder[MyRecord]                                                           =
      Derive.derive[ValueEncoder, MyRecord](ValueEncoderDeriver.default)
    given ValueDecoder[MyRecord]                                                           =
      Derive.derive[ValueDecoder, MyRecord](ValueDecoderDeriver.default)
    given TypeTag[MyRecord]                                                                =
      Derive.derive[TypeTag, MyRecord](TypeTagDeriver.default)

    // Define accessors to use them later in the filter predicate.
    // You can give any names to the accessors as we demonstrate here.
    val (id, name, age) = Filter[MyRecord].columns

  val data =
    Chunk(
      MyRecord(1, "bob", Some(10L)),
      MyRecord(2, "bob", Some(12L)),
      MyRecord(3, "alice", Some(13L)),
      MyRecord(4, "john", None)
    )

  val recordsFile = Path(Files.createTempDirectory("records")) / "records.parquet"

  override def run =
    (
      for {
        writer   <- ZIO.service[ParquetWriter[MyRecord]]
        reader   <- ZIO.service[ParquetReader[MyRecord]]
        _        <- writer.writeChunk(recordsFile, data)
        fromFile <- reader.readChunkFiltered(
                      recordsFile,
                      filter(
                        MyRecord.id > 1 `and` (
                          MyRecord.name =!= "bob" `or`
                            // Use .nullable syntax for optional fields.
                            MyRecord.age.nullable > 10L
                        )
                      )
                    )
        _        <- Console.printLine(fromFile)
      } yield ()
    ).provide(
      ParquetWriter.configured[MyRecord](),
      ParquetReader.configured[MyRecord]()
    )
  // Outputs:
  // Chunk(MyRecord(2,bob,Some(12)),MyRecord(3,alice,Some(13)),MyRecord(4,john,None))
```

See the full docs on the project's [website](https://mnedokushev.me/zio-apache-parquet/).

---

## ZIO EclipseStore

[ZIO EclipseStore](https://github.com/riccardomerolla/zio-eclipsestore) is a ZIO-based library for type-safe, efficient, and boilerplate-free access to [EclipseStore](https://github.com/eclipse-store/store).

## Introduction

ZIO EclipseStore provides a ZIO-powered interface to EclipseStore, offering a modern, functional approach to object graph persistence with type safety and composability.

## Features

- **Type-Safe API** — Leverage ZIO's type system for compile-time safety
- **Automatic Query Batching** — Batchable queries are optimized automatically
- **Parallel Execution** — Un-batchable queries execute in parallel with controlled concurrency
- **Typed Root Instances** — Declaratively describe and access root aggregates
- **Lifecycle Management** — Checkpoints, backups, and restarts via `LifecycleCommand`
- **Streaming Persistence** — Stream keys/values and batch updates with `putAll`/`persistAll`
- **GigaMap Module** — Advanced indexed maps with query DSL, CRUD, and persistence support
- **Resource Safety** — ZIO's resource management ensures proper cleanup
- **Custom Type Handlers & Blobs** — Register custom binary handlers for domain types and blobs
- **Backup Targets & Import/Export** — Built-in backup configuration (SQLite/SQL/S3/FTP), import/export helpers
- **Performance Tuning** — Configure channels, caches, compression, off-heap page store, encryption
- **Lazy Loading & Eager Storing** — Lazy references/collections plus eager-field semantics
- **ZIO Config Integration** — Load `EclipseStoreConfig` from HOCON/resources via zio-config
- **Effect-Oriented** — All operations are ZIO effects for composability

## Installation

```scala
libraryDependencies ++= Seq(
  "io.github.riccardomerolla" %% "zio-eclipsestore" % "1.0.5",
  "io.github.riccardomerolla" %% "zio-eclipsestore-gigamap" % "1.0.5", // Optional: for GigaMap module
  "io.github.riccardomerolla" %% "zio-eclipsestore-storage-sqlite" % "1.0.5" // Optional: for SQLite storage/backup
)
```

## Example

A simple demo showcasing batch operations, typed roots, and query execution:

```scala

object MyApp extends ZIOAppDefault:
  def run =
    val program = for
      // Batch store values
      _ <- EclipseStoreService.putAll(
        List("user:1" -> "Alice", "user:2" -> "Bob", "user:3" -> "Charlie")
      )
      
      // Retrieve values
      user <- EclipseStoreService.get[String, String]("user:1")
      _ <- ZIO.logInfo(s"User: ${user.getOrElse("not found")}")
      
      // Stream all values
      streamed <- EclipseStoreService.streamValues[String].runCollect
      _ <- ZIO.logInfo(s"All users: ${streamed.mkString(", ")}")
      
      // Work with typed roots
      favoritesDescriptor = RootDescriptor(
        id = "favorite-users",
        initializer = () => ListBuffer.empty[String]
      )
      favorites <- EclipseStoreService.root(favoritesDescriptor)
      _ <- ZIO.succeed(favorites.addOne("user:1"))
      _ <- EclipseStoreService.maintenance(LifecycleCommand.Checkpoint)
      
      // Execute multiple queries in batch
      queries = List(
        Query.Get[String, String]("user:1"),
        Query.Get[String, String]("user:2"),
        Query.Get[String, String]("user:3")
      )
      results <- EclipseStoreService.executeMany(queries)
      _ <- ZIO.logInfo(s"Batch query results: ${results.mkString(", ")}")
    yield ()
    
    program.provide(
      EclipseStoreConfig.temporaryLayer,
      EclipseStoreService.live
    )
```

See the full docs on the project's [GitHub repository](https://github.com/riccardomerolla/zio-eclipsestore).

---

## ZIO gRPC

[ZIO-gRPC](https://scalapb.github.io/zio-grpc/) lets us write purely functional gRPC servers and clients.

## Introduction

Key features of ZIO gRPC:
- **Functional and Type-safe** — Use the power of Functional Programming and the Scala compiler to build robust, correct and fully featured gRPC servers.
- **Support for Streaming** — Use ZIO's feature-rich `ZStream`s to create server-streaming, client-streaming, and bi-directionally streaming RPC endpoints.
- **Highly Concurrent** — Leverage the power of ZIO to build asynchronous clients and servers without deadlocks and race conditions.
- **Resource Safety** — Safely cancel an RPC call by interrupting the effect. Resources on the server will never leak!
- **Scala.js Support** — ZIO gRPC comes with Scala.js support, so we can send RPCs to our service from the browser.

## Installation

First of all we need to add following lines to the `project/plugins.sbt` file:

```scala
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.2")

libraryDependencies +=
  "com.thesamet.scalapb.zio-grpc" %% "zio-grpc-codegen" % "0.5.0"
```

Then in order to use this library, we need should add the following line in our `build.sbt` file:

```scala
PB.targets in Compile := Seq(
  scalapb.gen(grpc = true) -> (sourceManaged in Compile).value / "scalapb",
  scalapb.zio_grpc.ZioCodeGenerator -> (sourceManaged in Compile).value / "scalapb"
)

libraryDependencies ++= Seq(
  "io.grpc" % "grpc-netty" % "1.39.0",
  "com.thesamet.scalapb" %% "scalapb-runtime-grpc" % scalapb.compiler.Version.scalapbVersion
)
```

## Example

In this section, we are going to implement a simple server and client for the following gRPC _proto_ file:

```protobuf
syntax = "proto3";

option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";
option objc_class_prefix = "HLW";

package helloworld;

// The greeting service definition.
service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}
```

The hello world server would be like this:

```scala

object HelloWorldServer extends zio.App {

  val helloService: ZGreeter[ZEnv, Any] =
    (request: HelloRequest) =>
      putStrLn(s"Got request: $request") *>
        ZIO.succeed(HelloReply(s"Hello, ${request.name}"))

  val myApp = for {
    _ <- putStrLn("Server is running. Press Ctrl-C to stop.")
    _ <- ServerLayer
      .fromServiceList(
        ServerBuilder
          .forPort(9000)
          .addService(ProtoReflectionService.newInstance()),
        ServiceList.add(helloService))
      .build.useForever
  } yield ()

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.exitCode
}
```

And this is an example of using its client:

```scala

object HelloWorldClient extends zio.App {
  def myApp =
    for {
      r <- GreeterClient.sayHello(HelloRequest("World"))
      _ <- putStrLn(r.message)
    } yield ()

  val clientLayer =
    GreeterClient.live(
      ZManagedChannel(
        ManagedChannelBuilder.forAddress("localhost", 9000).usePlaintext()
      )
    )

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.provideCustom(clientLayer).exitCode
}
```

## Resources

- [Functional, Type-safe, Testable Microservices with ZIO gRPC](https://www.youtube.com/watch?v=XTkhxRTH1nE) by Nadav Samet (July 2020)

---

## ZIO K8s

[ZIO K8S](https://github.com/coralogix/zio-k8s) is an idiomatic ZIO client for the Kubernetes API.

## Introduction

This library provides a client for the full Kubernetes API as well as providing code generator support for custom resources and higher-level concepts such as operators, taking full advantage of the ZIO library.

Using ZIO K8S we can talk to the Kubernetes API that helps us to:
- Write an operator for our custom resource types
- Schedule some jobs in our cluster
- Query the cluster for monitoring purposes
- Write some cluster management tools

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "com.coralogix" %% "zio-k8s-client" % "1.3.3"
```

And then we need to choose the proper sttp backend:

```scala
"com.softwaremill.sttp.client3" %% "httpclient-backend-zio" % "3.1.1",
"com.softwaremill.sttp.client3" %% "slf4j-backend"          % "3.1.1"
```

Or the asynchronous version:

```scala
"com.softwaremill.sttp.client3" %% "async-http-client-backend-zio" % "3.1.1"
"com.softwaremill.sttp.client3" %% "slf4j-backend"                 % "3.1.1"
```

## Example

This is an example of printing the tail logs of a container:

```scala

object ZIOK8sLogsExample extends zio.App {
  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] = (args match {
    case List(podName) => tailLogs(podName, None)
    case List(podName, containerName) => tailLogs(podName, Some(containerName))
    case _ => console.putStrLnErr("Usage: <podname> [containername]")
  })
    .provideCustom(k8sDefault >>> Pods.live)
    .exitCode

  def tailLogs(podName: String,
               containerName: Option[String]
              ): ZIO[Pods with Console, K8sFailure, Unit] =
    pods
      .getLog(
        name = podName,
        namespace = K8sNamespace.default,
        container = containerName,
        follow = Some(true)
      )
      .tap { line: String =>
        console.putStrLn(line).ignore
      }
      .runDrain
}
```

## Resources

- [ZIO World - ZIO Kubernetes (ZIO K8S 1.0)](https://www.youtube.com/watch?v=BUMe2hGKjXA&t=31s) by Daniel Vigovszky (March 2020) — ZIO K8S 1.0, a new library by Daniel Vigovszky and Coralogix for working with Kubernetes, which includes support for the whole API, including event processors and compositional aspects for metrics and logging.

---

## ZIO Kinesis

[ZIO Kinesis](https://github.com/svroonland/zio-kinesis) is a ZIO-based AWS Kinesis client for Scala.

## Introduction

ZIO Kinesis is an interface to Amazon Kinesis Data Streams for consuming and producing data. This library is built on top of [ZIO AWS](https://github.com/zio/zio-aws), a library of automatically generated ZIO wrappers around AWS SDK methods.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

  ```scala
libraryDependencies += "nl.vroste" %% "zio-kinesis" % "0.20.0"
```

## Example

This is an example of consuming a stream from Amazon Kinesis:

```scala

object ZIOKinesisConsumerExample extends zio.App {
  val loggingLayer: ZLayer[Any, Nothing, Logging] =
    (Console.live ++ Clock.live) >>>
      Logging.console() >>>
      Logging.withRootLoggerName(getClass.getName)

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    Consumer
      .consumeWith(
        streamName = "my-stream",
        applicationName = "my-application",
        deserializer = Serde.asciiString,
        workerIdentifier = "worker1",
        checkpointBatchSize = 1000L,
        checkpointDuration = 5.minutes
      )(record => putStrLn(s"Processing record $record"))
      .provideCustom(Consumer.defaultEnvironment ++ loggingLayer)
      .exitCode
}
```

---

## ZIO NebulaGraph

[zio-nebula](https://github.com/nebula-contrib/zio-nebula) is a simple wrapper around [nebula-java](https://github.com/vesoft-inc/nebula-java/) for easier integration with [NebulaGraph](https://github.com/vesoft-inc/nebula) into Scala, ZIO applications.

## Introduction

- Supports all clients: Session Pool、Pool、Storage、Meta
- Support for configuring clients with typesafe config
- Other optimizations suitable for Scala pure functional
- Support Scala 3, Scala 2.13 and Scala 2.12

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "io.github.jxnu-liguobin" %% "zio-nebula" % <version>
```

There are the version correspondence between zio-nebula and nebula-java:

|  zio  | zio-nebula | nebula-java |
|:-----:|:----------:|:-----------:|
| 2.0.x |   0.0.x    |    3.6.0    |
| 2.0.x |   0.1.0    |    3.6.0    |
| 2.0.x |   0.1.1    |    3.6.1    |

## Example

Usually, we use a session client, which can be conveniently used in ZIO applications like this:

```scala

final class NebulaSessionClientExample(sessionClient: NebulaSessionClient) {

  def execute(stmt: String): ZIO[Any, Throwable, NebulaResultSet] = {
    // your business logic
    sessionClient.execute(stmt)
  }
}

object NebulaSessionClientExample {
  lazy val layer = ZLayer.fromFunction(new NebulaSessionClientExample(_))
}

object NebulaSessionClientMain extends ZIOAppDefault {

  override def run = (for {
    _ <- ZIO.serviceWithZIO[NebulaSessionClient](_.init()) // since 0.1.1, no need to call it manually. 
    _ <- ZIO.serviceWithZIO[NebulaSessionClientExample](
             _.execute("""
                         |INSERT VERTEX person(name, age) VALUES 
                         |'Bob':('Bob', 10), 
                         |'Lily':('Lily', 9),'Tom':('Tom', 10),
                         |'Jerry':('Jerry', 13),
                         |'John':('John', 11);""".stripMargin)
           )
    _ <- ZIO.serviceWithZIO[NebulaSessionClientExample](
             _.execute("""
                         |INSERT EDGE like(likeness) VALUES
                         |'Bob'->'Lily':(80.0),
                         |'Bob'->'Tom':(70.0),
                         |'Lily'->'Jerry':(84.0),
                         |'Tom'->'Jerry':(68.3),
                         |'Bob'->'John':(97.2);""".stripMargin)
           )
    _ <- ZIO.serviceWithZIO[NebulaSessionClientExample](
             _.execute("""
                         |USE test;
                         |MATCH (p:person) RETURN p LIMIT 4;
                         |""".stripMargin)
           )
  } yield ())
    .provide(
      Scope.default,
      NebulaSessionClientExample.layer,
      SessionClientEnv
    )

}
```

Add configuration in classpath:

```hocon
{
  graph {
    address = [
      {
        host = "127.0.0.1",
        port = 9669
      }
    ]
    auth {
      username = "root"
      password = "nebula"
    }
    spaceName = "test"
    reconnect = true
  }

  meta {
    address = [
      {
        host = "127.0.0.1",
        port = 9559
      }
    ]
    timeoutMills = 30000
    connectionRetry = 3
    executionRetry = 1
    enableSSL = false
  }

  storage {
    address = [
      {
        host = "127.0.0.1",
        port = 9559
      }
    ]
    timeoutMills = 30000
    connectionRetry = 3
    executionRetry = 1
    enableSSL = false
  }

  pool {
    timeoutMills = 60000
    enableSsl = false
    minConnsSize = 10
    maxConnsSize = 10
    intervalIdleMills = 100
    waitTimeMills = 100
  }
}
```

---

## ZIO Pulsar

[ZIO Pulsar](https://github.com/apache/pulsar) is the _Apache Pulsar_ client for Scala with ZIO and ZIO Streams integration.

## Introduction

ZIO Pulsar is a purely functional Scala wrapper over the official Pulsar client. Some key features of this library:

- **Type-safe** — Utilizes Scala type system to reduce runtime exceptions present in the official Java client.
- **Streaming-enabled** — Naturally integrates with ZIO Streams.
- **ZIO integrated** — Uses common ZIO primitives like ZIO effect and `ZManaged` to reduce the boilerplate and increase expressiveness.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file for _Scala 3_:

```scala
libraryDependencies += "com.github.jczuchnowski" %% "zio-pulsar" % "0.1"
```

## Example

First of all we need to create an instance of _Apache Pulsar_ and run that:

```
docker run -it \
  -p 6650:6650 \
  -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  --network pulsar \
  apachepulsar/pulsar:2.7.0 \
  bin/pulsar standalone
```

Now we can run the following example:

```scala

object StreamingExample extends zio.App {
  val topic = "my-topic"

  val producer: ZManaged[Has[PulsarClient], PulsarClientException, Unit] =
    for {
      sink <- Producer.make(topic, Schema.STRING).map(_.asSink)
      _ <- Stream.fromIterable(0 to 100).map(i => s"Message $i").run(sink).toManaged_
    } yield ()

  val consumer: ZManaged[Has[PulsarClient] with Blocking with Console, PulsarClientException, Unit] =
    for {
      builder <- ConsumerBuilder.make(Schema.STRING).toManaged_
      consumer <- builder
        .subscription(Subscription("my-subscription", SubscriptionType.Exclusive))
        .topic(topic)
        .build
      _ <- consumer.receiveStream.take(10).foreach { e =>
        consumer.acknowledge(e.getMessageId) *>
          putStrLn(new String(e.getData, StandardCharsets.UTF_8)).orDie
      }.toManaged_
    } yield ()

  val myApp =
    for {
      f <- consumer.fork
      _ <- producer
      _ <- f.join.toManaged_
    } yield ()

  def run(args: List[String]): URIO[ZEnv, ExitCode] =
    myApp
      .provideCustom(
        (Console.live ++ Clock.live) >+>
          PulsarClient.live("localhost", 6650)
      ).useNow.exitCode
}
```

## Resources

- [ZIO World - ZIO PULSAR](https://www.youtube.com/watch?v=tpwydDqQBmk) by Jakub Czuchnowski (March 2020) — A new library that offers a native, first-class ZIO experience for Pulsar, the Kafka competitor gaining traction for some use cases.

---

## ZIO Slick Interop

[ZIO Slick Interop](https://github.com/ScalaConsultants/zio-slick-interop) is a small library, that provides interop between Slick and ZIO.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "io.scalac" %% "zio-slick-interop" % "0.4.0"
```

## Example

To run this example we should also add the _HikariCP integration for Slick_ in our `build.sbt` file:

```scala
libraryDependencies += "com.typesafe.slick" %% "slick-hikaricp" % "3.3.3"
```

Here is a full working example of creating database-agnostic Slick repository:

```scala

case class Item(id: Long, name: String)

trait ItemRepository {
  def add(name: String): IO[Throwable, Long]

  def getById(id: Long): IO[Throwable, Option[Item]]

  def upsert(name: String): IO[Throwable, Long]
}

object ItemsTable {
  class Items(tag: Tag) extends Table[Item](
    _tableTag = tag,
    _tableName = "ITEMS"
  ) {
    def id = column[Long]("ID", O.PrimaryKey, O.AutoInc)

    def name = column[String]("NAME")

    def * = (id, name) <> ((Item.apply _).tupled, Item.unapply _)
  }

  val table = TableQuery[ItemsTable.Items]
}

object SlickItemRepository {
  val live: ZLayer[DatabaseProvider, Throwable, ItemRepository] =
    ZLayer.fromServiceM { db =>
      db.profile.flatMap { profile =>

        val initialize = ZIO.fromDBIO(ItemsTable.table.schema.createIfNotExists)

        val repository = new ItemRepository {
          private val items = ItemsTable.table

          def add(name: String): IO[Throwable, Long] =
            ZIO
              .fromDBIO((items returning items.map(_.id)) += Item(0L, name))
              .provideEnvironment(ZEnvironment(db))

          def getById(id: Long): IO[Throwable, Option[Item]] = {
            val query = items.filter(_.id === id).result

            ZIO.fromDBIO(query).map(_.headOption).provideEnvironment(ZEnvironment(db))
          }

          def upsert(name: String): IO[Throwable, Long] =
            ZIO
              .fromDBIO { implicit ec =>
                (for {
                  itemOpt <- items.filter(_.name === name).result.headOption
                  id <- itemOpt.fold[DBIOAction[Long, NoStream, Effect.Write]](
                    (items returning items.map(_.id)) += Item(0L, name)
                  )(item => (items.map(_.name) update name).map(_ => item.id))
                } yield id).transactionally
              }
              .provideEnvironment(Environment(db))
        }

        initialize.as(repository).provideEnvironment(Environment(db))
      }
    }
}

object Main extends zio.App {

  private val config = ConfigFactory.parseMap(
    Map(
      "url" -> "jdbc:h2:mem:test1;DB_CLOSE_DELAY=-1",
      "driver" -> "org.h2.Driver",
      "connectionPool" -> "disabled"
    ).asJava
  )

  private val env: ZLayer[Any, Throwable, ItemRepository] =
    (ZLayer.succeed(config) ++ ZLayer.succeed[JdbcProfile](
      slick.jdbc.H2Profile
    )) >>> DatabaseProvider.live >>> SlickItemRepository.live

  val myApp: ZIO[Console with Has[ItemRepository], Throwable, Unit] =
    for {
      repo <- ZIO.service[ItemRepository]
      aId1 <- repo.add("A")
      _ <- repo.add("B")
      a <- repo.getById(1L)
      b <- repo.getById(2L)
      aId2 <- repo.upsert("A")
      _ <- putStrLn(s"$aId1 == $aId2")
      _ <- putStrLn(s"A item: $a")
      _ <- putStrLn(s"B item: $b")
    } yield ()

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    myApp.provideCustom(env).exitCode
}
```

---

## ZIO Temporal

[ZIO Temporal](https://zio-temporal.vhonta.dev/) is a ZIO library for Temporal, a microservice workflow orchestration platform.

## Introduction

ZIO Temporal is a ZIO library based on the Temporal Java-SDK. ZIO Temporal brings first-class Scala & ZIO support with additional compile-time checks that Java SDK lacks.

[Temporal](https://temporal.io/) platform helps to eliminate complex errors or retry logic, avoid callbacks, and ensure that every workflow you start completes. Temporal delivers durable execution for your services and applications.

## Installation

To use the library, we need to add the following dependency:

```scala
libraryDependencies += "dev.vhonta" %% "zio-temporal-core" % "0.4.0"
```

## Examples

Here we have one ZIO app with two "modules". One is the worker that executes the workflow activity, and the other is the Client that sends the request to the Temporal platform. To run the sample, it's required to download [Temporal cli](https://docs.temporal.io/cli) (recommended for development).

Run the Temporal server on one shell:

```sh
temporal server start-dev --ip 0.0.0.0 --db-filename /tmp/temporal.db
```

Now we can run the sample application, which is based on Temporal Java SDK docs using [scala-cli](https://scala-cli.virtuslab.org):

```scala
//> using scala "3.3.0"

//> using lib "dev.zio::zio:2.0.16"
//> using lib "dev.vhonta::zio-temporal-core:0.4.0"
//> using lib "dev.zio::zio-logging:2.1.14"
//> using lib "dev.zio::zio-logging-slf4j2-bridge:2.1.14"

// This is our workflow interface
@workflowInterface
trait EchoWorkflow:

  @workflowMethod
  def echo(str: String): String

// Workflow implementation
class EchoWorkflowImpl extends EchoWorkflow:
  private val logger = ZWorkflow.makeLogger

  override def echo(str: String): String =
    logger.info(s"Worker: Received \"$str\"")
    s"ACK: $str"

// Main Application
object Main extends ZIOAppDefault:
  val taskQueue = "echo-queue"

  // Worker implementation
  val setupWorker = ZWorkerFactory.newWorker(taskQueue) @@
    ZWorker.addWorkflow[EchoWorkflowImpl].fromClass

  // Client implementation
  def invokeWorkflow(msg: String) = ZIO.serviceWithZIO[ZWorkflowClient]: client =>
    for
      uuid <- Random.nextUUID
      workflowID = s"echo-$uuid"
      echoWorkflow <- client
                        .newWorkflowStub[EchoWorkflow]
                        .withTaskQueue(taskQueue)
                        .withWorkflowId(workflowID)
                        .withWorkflowRunTimeout(2.seconds)
                        .withRetryOptions(ZRetryOptions.default.withMaximumAttempts(3))
                        .build
      _   <- ZIO.logInfo(s"Will submit message \"$msg\" with workflow ID $workflowID")
      res <- ZWorkflowStub.execute(echoWorkflow.echo(msg))
      _   <- ZIO.logInfo(s"Greeting received: $res")
    yield res

  // Logging configuration
  val logFilter: LogFilter[String] = LogFilter.logLevelByName(
    LogLevel.Info,
    "io.grpc.netty" -> LogLevel.Warning,
    "io.netty"      -> LogLevel.Warning,
    "io.temporal"   -> LogLevel.Error,
  )
  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleLogger(ConsoleLoggerConfig(LogFormat.colored, logFilter))

  // ZIO Main Program
  def run: ZIO[ZIOAppArgs & Scope, Any, Any] =
    val program =
      for
        args           <- getArgs
        msg             = if args.isEmpty then "testMsg" else args.mkString(" ")
        _              <- setupWorker
        _              <- ZWorkerFactory.setup
        _              <- ZWorkflowServiceStubs.setup()
        workflowResult <- invokeWorkflow(msg)
        _              <- ZIO.logInfo(s"The workflow result: $workflowResult")
      yield ExitCode.success

    program
      .provideSome[ZIOAppArgs & Scope](
        ZWorkflowServiceStubsOptions.make,
        ZWorkflowClientOptions.make,
        ZWorkerFactoryOptions.make,
        ZWorkflowClient.make,
        ZWorkflowServiceStubs.make,
        ZWorkerFactory.make,
        slf4j.bridge.Slf4jBridge.initialize,
      )
```

Generates the output:

```sh
❯ scli zio-temporal.scala
timestamp=2023-06-28T09:27:21.171306+02:00 level=INFO thread=zio-fiber-4 message="ZWorkerFactory started"
timestamp=2023-06-28T09:27:21.238575+02:00 level=INFO thread=zio-fiber-4 message="Will submit message "testMsg" with workflow ID echo-82026831-91df-4138-b862-710d87b3ebb4"
timestamp=2023-06-28T09:27:21.5684+02:00   level=INFO thread=zio-fiber-183 message="Worker: Received "testMsg""
timestamp=2023-06-28T09:27:21.645988+02:00 level=INFO thread=zio-fiber-4 message="Greeting received: ACK: testMsg"
timestamp=2023-06-28T09:27:21.6473+02:00   level=INFO thread=zio-fiber-4 message="The workflow result: ACK: testMsg"
timestamp=2023-06-28T09:27:21.705634+02:00 level=INFO thread=zio-fiber-4 message="ZWorkerFactory shutdownNow initiated..."
```

Results of the execution can also be seen in the [Temporal UI](http://localhost:8233) running locally or the [tctl](https://github.com/temporalio/tctl) tool:

```sh
❯ tctl workflow observe --workflow_id echo-81ef73da-d54d-492a-8f91-78e888dcebc8
Progress:
  1, 2023-04-18T20:34:00Z, WorkflowExecutionStarted
  2, 2023-04-18T20:34:00Z, WorkflowTaskScheduled
  3, 2023-04-18T20:34:00Z, WorkflowTaskStarted
  4, 2023-04-18T20:34:00Z, WorkflowTaskCompleted
  5, 2023-04-18T20:34:00Z, WorkflowExecutionCompleted

Result:
  Run Time: 1 seconds
  Status: COMPLETED
  Output: ["ACK: testMsg"]
```

## Want more?
You can find more complex and end-to-end example projects in [zio-temporal-samples](https://github.com/vitaliihonta/zio-temporal-samples) repo.

---

## ZIO Test Akka HTTP

[ZIO Test Akka HTTP](https://github.com/senia-psm/zio-test-akka-http) is an Akka-HTTP Route TestKit for zio-test.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "info.senia" %% "zio-test-akka-http" % "2.0.0-RC5"
```

## Example

An example of writing Akka HTTP Route test spec:

```scala

object MySpec extends AkkaZIOSpecDefault {
  def spec =
    suite("MySpec")(
      test("my test") {
        assertZIO(Get() ~> complete(HttpResponse()))(
          handled(
            response(equalTo(HttpResponse()))
          )
        )
      }
    )
}
```

---

## ZparkIO

[ZParkIO](https://github.com/leobenkel/ZparkIO) is a boilerplate framework to use _Spark_ and _ZIO_ together.

## Introduction

_ZparkIO_ enables us to:
- Wrap asynchronous and synchronous operations smoothly. So everything is wrapped in ZIO.
- Have ZIO features in our spark jobs, like forking and joining fibers, parallelizing tasks, retrying, and timing-out.
- Make our spark job much easier to debug

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "com.leobenkel" %% "zparkio" % "[SPARK_VERSION]_[VERSION]"
```

## Example

Using _ZparkIO_ we can write jobs like the following example:

```scala

def readParquetFile[A](parquetPath: String): ZIO[Clock with SparkModule, Throwable, DataFrame] =
  for {
    spark <- SparkModule()
    dataset <- Task(spark.read.parquet(parquetPath))
      .retry(
        Schedule.recurs(3) && Schedule.exponential(2.seconds)
      )
      .timeoutFail(ZparkioApplicationTimeoutException())(5.minutes)
  } yield dataset
```

## Resources

- [Accelerating Spark with ZIO](https://www.youtube.com/watch?v=bWgVGzb5-H8) by Leo Benkel (December 2020) — Apache Spark is a powerful tool for distributed analytics but isn’t built to offer rich primitives for local parallelism. Yet, using local parallelism, it is possible to dramatically improve the performance of ML projects—for example, by fetching data sources in parallel. Historically, Scala projects have used Scala's Future for local parallelism, but increasingly, many projects have begun adopting ZIO as a better Future. Although powerful, ZIO is not easy to set up with Spark. In this talk, Leo will introduce an open-source library that handles all boilerplate so you can easily implement Spark and ZIO in your ML projects, and will then present detailed benchmarks showing how much additional performance can be obtained through local parallelism. Learn how to use ZIO to make Spark better and faster!

---

## ZIO Compatible Libraries

List of ZIO compatible libraries:

- [d4s](https://github.com/PlayQ/d4s) — "Dynamo DB Database done Scala way". A library that allows accessing the DynamoDB in a purely functional way
- [Edomata](https://github.com/hnaderi/edomata) — Production ready event-driven automata for Scala, Scala.js and scala native
- [elastic4s](https://github.com/sksamuel/elastic4s) — Elasticsearch Scala Client - Reactive, Non Blocking, Type Safe, HTTP Client
- [Lepus](https://github.com/hnaderi/lepus) — Your principled, purely functional, non-blocking RabbitMQ client for scala, scala js and scala native built on top of fs2
- [neotypes](https://github.com/neotypes/neotypes) — A Scala lightweight, type-safe & asynchronous driver for neo4j
- [scanamo](https://github.com/scanamo/scanamo) — Simpler DynamoDB access for Scala
- [sttp](https://github.com/softwaremill/sttp) — The Scala HTTP client you always wanted!
- [doobie](https://github.com/tpolecat/doobie) — Functional JDBC layer for Scala.
- [fs2](https://github.com/typelevel/fs2) — Compositional, streaming I/O library for Scala
- [http4s](https://github.com/http4s/http4s) — A minimal, idiomatic Scala interface for HTTP
- [Automorph](https://automorph.org) - RPC in a few lines of Scala code

---

## ZIO Ecosystem

ZIO has a rich ecosystem of libraries and tools that enhance its capabilities and provide additional functionality. This ecosystem includes libraries for various purposes, such as web development, data processing, testing, and more.

But it doesn't end there! If you need a comprehensive list of libraries and tools, you can find them in one of the following sections:

- [Official libraries](officials/index.md), maintained by the ZIO team under the [ZIO Organization](https://github.com/zio).
- [Community libraries](community/index.md), maintained by members of the ZIO community.

Below are some of the highlights from the official libraries:

<Ecosystem  />

---

## ZIO Ecosystem Official Libraries

Official ZIO libraries are hosted in the [ZIO organization](https://github.com/zio/) on GitHub, and are generally maintained by core contributors to ZIO.

## Development Status

Each project in the ZIO organization namespace has a _Stage Badge_ which indicates the current status of that project:

* **Production Ready** — The project is stable and already used in production. We can expect reliability for the implemented use cases.
* **Development** — The project already has RC or milestone releases, but is still under active development. We should not expect full stability yet.
* **Experimental** — The project is not yet released, but an important part of the work is already done.
* **Research** — The project is at the design stage, with some sketches of work but nothing usable yet.
* **Concept** — The project is just an idea, development hasn't started yet.
* **Deprecated** — The project is not maintained anymore, and we don't recommend its usage.

## Highlights Projects

Here is the list of some of the highlighted projects that can be used in everyday ZIO development:

1. [ZIO HTTP](https://ziohttp.com): A type-safe, composable, and feature-rich HTTP library built on top of ZIO.
2. [ZIO Config](https://zio.dev/zio-config): A functional type-safe configuration library for ZIO applications.
3. [ZIO Logging](https://zio.dev/zio-logging): A logging library to integrate with different logging backends.
4. [ZIO JSON](https://zio.dev/zio-json): A JSON library to work with JSON data in ZIO applications.
5. [ZIO Schema](https://zio.dev/zio-schema): A library for generating/defining type-safe schemas for custom data types in ZIO applications.
6. [ZIO Quill](https://zio.dev/zio-quill): A library to interact with databases using type-safe queries.
7. [ZIO Kafka](https://zio.dev/zio-kafka): A Kafka client for ZIO and ZIO Streams
8. [ZIO Prelude](https://zio.dev/zio-prelude): A lightweight, distinctly Scala take on functional abstractions, with tight ZIO integration

## Official Libraries

 <DocCardList />

---

## Project Templates

List of project starters, bootstrap tools or, templates.

- [zio-scala2-quickstart.g8](https://github.com/ScalaConsultants/zio-scala2-quickstart.g8) — A Giter8 template for a basic Scala 2 application with a choice of libraries to use (Akka HTTP, ZIO HTTP, Quill, Slick)
- [zio-scala3-quickstart.g8](https://github.com/ScalaConsultants/zio-scala3-quickstart.g8) — A Giter8 template for a basic Scala 3 application with ZIO HTTP and Quill
- [zio-scala-3-project.g8](https://github.com/lachezar/zio-scala-3-project.g8) - A Giter8 template for Scala 3 + ZIO 2 web service applications using zio-http, zio-json, zio-quill, zio-kafka, zio-prelude and Onion architecure.

---

## ZIO Tools

- [ZIO IntelliJ](https://github.com/zio/zio-intellij) — A complementary, community-developed plugin for IntelliJ IDEA, brings enhancements when using ZIO in your projects

---

## Articles

:::caution
These articles reflect the state of ZIO at the time of their publication. The code samples might be outdated, considering ZIO was early in development at the time they were written. However, the concepts are still relevant.
:::

## ZIO Core
- [Wrapping impure code with ZIO](https://medium.com/@ghostdogpr/wrapping-impure-code-with-zio-9265c219e2e) by Pierre Ricadat (July 2019)
- [Thread shifting in cats-effect and ZIO](https://blog.softwaremill.com/thread-shifting-in-cats-effect-and-zio-9c184708067b) by Adam Warski (June 2019)
- [Performant Functional Programming to the Max with ZIO](https://cloudmark.github.io/A-Journey-To-Zio/) by Mark Galea (May 2019)
- [ZIO & Cats Effect: A Match Made in Heaven](http://degoes.net/articles/zio-cats-effect) by John De Goes (April 2019)
- [Beautiful, Simple, Testable Functional Effects for Scala](http://degoes.net/articles/zio-environment) (introducing ZIO Environment) by John De Goes (February 2019)
- [Thread Pool Best Practices with ZIO](http://degoes.net/articles/zio-threads) by John De Goes (January 2019)
- [Bifunctor IO: A Step Away from Dynamically-Typed Error Handling](http://degoes.net/articles/bifunctor-io) by John De Goes (May 2018)

## Patterns and Best Practices
- [Idiomatic dependency injection for ZIO applications in Scala](https://blog.pierre-ricadat.com/idiomatic-dependency-injection-for-zio-applications-in-scala) by Pierre Ricadat (Nov 2024)
- [The Tri-Z Architecture: a Pattern for Layering ZIO Applications in Scala](https://blog.pierre-ricadat.com/the-tri-z-architecture-a-pattern-for-layering-zio-applications-in-scala) by Pierre Ricadat (Aug 2024)
- [Tuning ZIO for high performance](https://blog.pierre-ricadat.com/tuning-zio-for-high-performance) by Pierre Ricadat (May 2024)
- [Processing ZIO effects through a pipeline](https://medium.com/@svroonland/processing-zio-effects-through-a-pipeline-c469e28dff62) (September 2020)
- [5 pitfalls to avoid when starting to work with ZIO](https://medium.com/wix-engineering/5-pitfalls-to-avoid-when-starting-to-work-with-zio-adefdc7d2d5c) by Natan Silnitsky (Jan 2020)

## ZIO STM
- [How to write a concurrent LRU cache with ZIO STM](https://scalac.io/how-to-write-a-completely-lock-free-concurrent-lru-cache-with-zio-stm/) by Jorge Vasquez (March 2020)
- [Exploring the STM functionality in ZIO](https://freskog.github.io/blog/2019/05/30/explore-zio-stm/) by Fredrik Skogberg (May 2019)

## Testing
- [Integration Testing](https://blended-zio.github.io/blended-zio/blog/integration-testing) by Andreas Gies (Feb 2021)
- [Effective testing with ZIO Test (RC18)](https://scala.monster/zio-test/) by Pavels Sisojevs (April 2020)
- [Testing background process with ZIO](https://www.rudder.io/blog/testing-background-process-zio/) by François Armand (March 2020)
- [Effective testing with ZIO Test (RC17)](https://scala.monster/zio-test-old/) by Pavels Sisojevs (January 2020)
- [Speeding up time with ZIO TestClock](https://timpigden.github.io/_pages/zio-streams/SpeedingUpTime.html) by Tim Pigden (October 2019)
- [Testing Incrementally with ZIO Environment](http://degoes.net/articles/testable-zio) by John De Goes (March 2019)

## ZIO Streams
- [ZIO Streams and JMS](https://blended-zio.github.io/blended-zio/blog/zio-streams-jms) by Andreas Gies (October 2020)
- [Building the Death Star with ZIO Stream](https://juliano-alves.com/2020/05/04/deathstar-zio-stream/) by Juliano Alves (May 2020)
- [Simulating IoT Events with ZIO Streams](https://timpigden.github.io/_pages/zio-streams/GeneratingChillEvents.html) by Tim Pigden (November 2019)

## ZIO Service Pattern
- [Example of ZLayers being used in combination](https://timpigden.github.io/_pages/zlayer/Examples.html) by Tim Pigden (March 2020)
- [From idea to product with ZLayer](https://scala.monster/welcome-zio/) by Pavels Sisojevs (March 2020)
- [What are the benefits of the ZIO modules with ZLayers](https://medium.com/@pascal.mengelt/what-are-the-benefits-of-the-zio-modules-with-zlayers-3bf6cc064a9b) by Pascal Mengelt (March 2020)
- [Decouple the Program from its Implementation with ZIO modules.](https://medium.com/@pascal.mengelt/decouple-the-program-from-its-implementation-with-zio-modules-d9b8713d502e) by Pascal Mengelt (December 2019)
- [Functional dependency injection in Scala using ZIO environments](https://blog.jdriven.com/2019/10/functional-dependency-injection-in-scala-using-zio-environments/) by Chiel van de Steeg (October 2019)

## ZIO Use Cases
- [Spring to ZIO 101 - ZIO CRUD](https://adrianfilip.com/2020/03/15/spring-to-zio-101/) by Adrian Filip (March 2020)
- [Implement your future with ZIO](https://scala.monster/welcome-zio-old/) by Pavels Sisojevs (December 2019)
- [How to write a command line application with ZIO?](https://scalac.io/write-command-line-application-with-zio/) by Piotr Gołębiewski (November 2019)
- [Hacker News API Part 5](http://justinhj.github.io/2019/04/07/hacker-news-api-5.html) by Justin Heyes-Jones (April 2019)
- [Building the Hangman Game using ScalaZ ZIO](https://abhsrivastava.github.io/2018/11/03/Hangman-Game-Using-ZIO/) by Abhishek Srivastava (November 2018)
- [Elevator Control System using ZIO](https://medium.com/@wiemzin/elevator-control-system-using-zio-c718ae423c58) by Wiem Zine El Abidine (September 2018)

## Integration with Other Libraries
- [Streaming microservices with ZIO and Kafka](https://scalac.io/streaming-microservices-with-zio-and-kafka/) by Aleksandar Skrbic (February 2021)
- [uzhttp + sttp for light-weight http and websockets](https://timpigden.github.io/_pages/zio-uzhttp-sttp/uzhttp-sttp.html) updated for 1.0.1 by Tim Pigden (August 2020)
- [Streaming all the way with ZIO, Doobie, Quill, http4s and fs2](https://juliano-alves.com/2020/06/15/streaming-all-the-way-zio-doobie-quill-http4s-fs2/) by Juliano Alves (June 2020)
- [Building a cool CLI with Decline for my ZIO App](https://medium.com/@pascal.mengelt/building-a-cool-cli-with-decline-for-my-zio-app-80e095b2899a) by Pascal Mengelt (May 2020)
- [ZIO with http4s, Auth, Codecs and zio-tests (RC18)](https://timpigden.github.io/_pages/zio-http4s/intro.html) by Tim Pigden (April 2020)
- [An Introduction to ZIO Kafka](https://www.ziverge.com/post/an-introduction-to-zio-kafka) by Ziverge (April 2020)
- [ZIO + Http4s: a simple API client](https://juliano-alves.com/2020/04/20/zio-http4s-a-simple-api-client/) by Juliano Alves (April 2020)
- [What can ZIO do for me? A Long Polling example with sttp.](https://medium.com/@pascal.mengelt/what-can-zio-do-for-me-32281e4e8b16) by Pascal Mengelt (November 2019)
- [Making ZIO, Akka and Slick play together nicely](https://scalac.io/making-zio-akka-slick-play-together-nicely-part-1-zio-and-slick/) by Jakub Czuchnowski (August 2019)
- [Combining ZIO and Akka to enable distributed FP in Scala](https://medium.com/@ghostdogpr/combining-zio-and-akka-to-enable-distributed-fp-in-scala-61ffb81e3283) by Pierre Ricadat (July 2019)
- [tAPIr’s Endpoint meets ZIO’s IO](https://blog.softwaremill.com/tapirs-endpoint-meets-zio-s-io-3278099c5e10) by Adam Warski (July 2019)
- [ZIO with http4s and doobie](https://medium.com/@wiemzin/zio-with-http4s-and-doobie-952fba51d089) by Wiem Zine El Abidine (June 2019)
- [Using 47 Degree's Fetch library with ZIO](http://justinhj.github.io/2019/05/05/using-47degs-fetch-with-zio.html) by Justin Heyes-Jones (May 2019)

## Contribution
- [Lessons Learned From Being a ZIO Contributor](https://www.softinio.com/post/lessons-learned-from-being-a-zio-contributor/) by [Salar Rahmanian](https://www.softinio.com) (September 2020)

## Benchmarking and Comparison
- [Benchmarking Functional Error Handling in Scala](https://www.iteratorshq.com/blog/benchmarking-functional-error-handling-in-scala/) by Marcin Rzeźnicki (April 2020)
- [Scalaz 8 IO vs Akka (typed) Actors vs Monix (part 1)](https://blog.softwaremill.com/scalaz-8-io-vs-akka-typed-actors-vs-monix-part-1-5672657169e1) + [part 2](https://blog.softwaremill.com/akka-vs-zio-vs-monix-part-2-communication-9ce7261aa08c) + [part 3](https://blog.softwaremill.com/supervision-error-handling-in-zio-akka-and-monix-part-3-series-summary-abe75f964c2a) by Adam Warski (June 2018)

---

## Summary(Resources)

ZIO has a huge ecosystem of libraries, tools, talks, tutorials, and more. In this section, we are going to introduce some of the most important ones.

:::info
If you find a new great library, talk, resource, slides, or project, related to ZIO, consider adding it to the list with your PR.
:::

---

## Projects using ZIO

- [Rudder](https://github.com/normation/rudder) — an example about how to manage error ADT in several sub-projects and specialized sub-domains, and how one can gradually contextualize error messages in domain layers. Uses queues, resources, interop with Java, and historical code. See [context and references](https://issues.rudder.io/issues/14870).
- [ZIO AI Platform Backend](https://github.com/Clover-Group/zio_front) — Clover Group AI Platform backend, which employs ZIO, Doobie, http4s and Kafka .
- [Polynote](https://github.com/polynote/polynote) — a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more.
- [Blended ZIO](https://blended-zio.github.io/blended-zio/) — A sample project migrating a largely untyped, actor based integration framework to ZIO.

---

## Sample Projects

- [GitHub Release Pager with ZIO 2.0](https://github.com/psisoyev/release-pager) by [Pavels Sisojevs](https://github.com/psisoyev)
- [Redis Streams with ZIO](https://github.com/kensuio-oss/redis-streams-zio) by [Leszek Gruchała](https://github.com/leszekgruchala)
- [ZIO Production Ready Microservice](https://github.com/saraiva132/zio-cats-backend) by [Rafael Figueiredo](https://github.com/saraiva132)
- [ZIO Modern Backend giter8 Template - Tapir, sttp, http4s](https://github.com/Anadyne/zio-full-backend.g8) by [Boris V.Kuznetsov](https://github.com/tampler)
- [ZIO CRUD sample - Code for the 'Spring to ZIO 101' blog post 03/2020](https://github.com/adrianfilip/zio-crud-sample) by [Adrian Filip](https://github.com/adrianfilip)
- [ZIO modules with different implementations - Code for the 'Decouple the Program from its Implementation...' blog post 12/2019](https://github.com/pme123/zio-comps-module) by [pme123](https://github.com/pme123)
- [TicTacToe command line game using Service Pattern](https://github.com/ioleo/zio-by-example) by [ioleo](https://github.com/ioleo)
- [Long Polling with ZIO - Code for the 'What can ZIO do for me?' blog post 11/2019](https://github.com/pme123/zio-http4s-long-polling) by [pme123](https://github.com/pme123)
- [Hello world with ZIO and http4s](https://gitlab.com/maplambda/zio-http4s) by [maplambda](https://gitlab.com/maplambda)
- [STM Partitioning - Code for the 'Exploring the STM functionality in ZIO' blog post](https://github.com/freskog/stm-partitioning)
- [ZIO Todo Backend](https://github.com/mschuwalow/zio-todo-backend) by [mschuwalow](https://github.com/mschuwalow)
- [Event Driven Messenger](https://github.com/edvmorango/event-driven-messenger) by [edvmorango](https://github.com/edvmorango)
- [Zorechka Bot](https://github.com/wix-incubator/zorechka-bot) by [wix-incubator](https://github.com/wix-incubator)
- [A ZIO + http4s + Circe + Quill + Tapir giter8 template](https://github.com/pandaforme/ultron.g8) by [pandaforme](https://github.com/pandaforme)
- [More ZIO/http4s: with http4s authentication, encoding/decoding + zio tests](https://github.com/TimPigden/zio-http4s-examples) by [Tim Pigden](https://github.com/TimPigden)
- [Full Scala Stack, a sample project that uses akka-http, slick, zio, scalajs, react, ScalablyTyped and semantic ui](https://github.com/rleibman/full-scala-stack) by Roberto Leibman
- [A minimal ZIO giter8 template](https://github.com/jchoffmann/zio-seed.g8) by [jchoffmann](https://github.com/jchoffmann)
- [Lagom CQRS/ES microservices with ZIO, Caliban, zio-grpc, elastic4s and Logstage](https://github.com/sigurdthor/book-shelf) by [Yaroslav Yaremenko](https://github.com/sigurdthor)
- [ZIO Advanced Template](https://github.com/Neurodyne/zio-top.g8) by [Boris V.Kuznetsov](https://github.com/tampler)
- [ZIO Quill, H2 Database with FlyWay migrations Explained](https://github.com/Neurodyne/h2db-quill-demo) by [Boris V.Kuznetsov](https://github.com/tampler)
- [ZIO Tapir Http4s Integration](https://github.com/Neurodyne/zio-tapir) by [Boris V.Kuznetsov](https://github.com/tampler)
- [Akka Microservice using DDD with ZIO Streams Tapir AkkaHTTP Integration](https://github.com/mvillafuertem/scalcite) by [Miguel Villafuerte](https://github.com/mvillafuertem)
- [Simple ZIO ScalaJS Skeleton](https://github.com/sb-dev/zio-scalajs) by [Samir Benzenine](https://github.com/sb-dev)
- [Full ZIO Stack: A sample IM that uses zio, zio-redis, zio-actors, zio-schema, zio-streams, zio-crypto, tapir, akka-http](https://github.com/bitlap/zim) by [bitlap](https://github.com/bitlap/zim) (Notes in Chinese)
- [JWT Auth based persisent (reactive MongoDB) RESTful API for managing notes with ZIO 2.0, Scala 3 and ZLayers](https://github.com/Ghurtchu/notes-rest-api) by [Anzori (Nika) Ghurtchumelia](https://github.com/Ghurtchu)
- [ZIO 2.0 Restful webservice example using zio, zio-http, zio-json, quill, H2, twirl, zio-streams, zio-cache, zio-logging](https://github.com/Fancellu/zio-restful-webservice) by [Dino Fancellu](https://github.com/Fancellu)
- [Simple web project using Scala3, ZIO2 (with tests), Caliban (with ZQuery), Http4s, Cats, Circe, Doobie](https://github.com/lewapek/DontLetExpireApp) by [Paweł Kołodziejczyk](https://github.com/lewapek)
- [All in one observability: Scala3 + ZIO + Prometheus metrics + OpenTelemetry tracing (Jaeger)](https://github.com/lewapek/observability-demo-apps) by [Paweł Kołodziejczyk](https://github.com/lewapek)
- [A simple demo application to show zio-cli in action](https://github.com/fancellu/zio-cli-demo) by [Dino Fancellu](https://github.com/Fancellu)
- [Kafka BigQuery Express](https://github.com/adevinta/kafka-bigquery-express/) A ZIO2 system to copy data from Kafka to BigQuery, safely and cost-effectively.

---

## Videos

## Functional Programming
- [FP to the Max](https://www.youtube.com/watch?v=sxudIMiOo68) by John De Goes (July 2018)
- [FP to the Min](https://www.youtube.com/watch?v=mrHphQT4RpU) by John De Goes (July 2020)

## Tour of ZIO
- [The ZIO of the Future](https://www.youtube.com/watch?v=u3pgyEiu9eU) by John De Goes (May 2021)
- [A Tour of ZIO](https://www.youtube.com/watch?v=TWdC7DhvD8M) by John De Goes (January 2020)

## ZIO Core
- [The Death of Tagless Final](https://www.youtube.com/watch?v=p98W4bUtbO8) by John De Goes (February 2019)
- [Upgrade Your Future](https://www.youtube.com/watch?v=USgfku1h7Hw) by John De Goes (September 2019)
- [One Monad to Rule Them All](https://www.youtube.com/watch?v=POUEz8XHMhE) by John De Goes (August 2019)
- [Functional Concurrency in Scala with ZIO](https://www.youtube.com/watch?v=m5nas4Hndqo) by Itamar Ravid (June 2019)
- [Thinking Functionally](https://www.youtube.com/watch?v=-KA3BSdqYug) by John De Goes (March 2019)
- [ZIO: Next-Generation Effects in Scala](https://www.youtube.com/watch?v=mkSHhsJXjdc&t=6s) by John De Goes (October 2018)
- [Functional Effects in ZIO](https://www.youtube.com/watch?v=4EeL8-chAR8) by  Wiem Zine El Abidine
- [ZIO WORLD - Blocking](https://www.youtube.com/watch?v=1g21c8VKeuU&t=320s) by Adam Fraser (March 2021) — Adam Fraser presented his work to make ZIO faster with blocking code, showing the tremendous (unbelievable) performance improvements possible with batch-mode blocking; part of ongoing work to make ZIO better at blocking workloads.

## ZIO Runtime System
- [ZIO WORLD - ZIO Runtime System](https://www.youtube.com/watch?v=OFFrw5aJzG4&t=313s) by John A. De Goes (March 2021) — John A. De Goes presented his work on the runtime system for ZIO 2.0, which involves a significant rewrite and simplification of the core execution model, resulting in what will be ZIO's best performance to date; benchmarks show nearly universally faster performance compared to CE3.

## Error Management
- [Error Management: Future vs ZIO](https://www.youtube.com/watch?v=mGxcaQs3JWI) by John De Goes and Kai (May 2019), [slide](https://www.slideshare.net/jdegoes/error-management-future-vs-zio)
- Systematic error management in application - We ported Rudder to ZIO: [video in French](https://www.youtube.com/watch?v=q0PlcgR5M1Q), [slides in English](https://speakerdeck.com/fanf42/systematic-error-management-we-ported-rudder-to-zio) by François Armand (Scala.io, October 2019)

## ZIO Service Pattern
- [ZLayers by example](https://www.youtube.com/watch?v=u5IrfkAo6nk) by Wiem Zine El Abidine (December 2020)
- [ZIO inception: Journey through layers and intersection types](https://www.youtube.com/watch?v=vNQFlq1SvaE) by Vladimir Pavkin (July 2020)
- [ZIO WORLD - ZLayer](https://www.youtube.com/watch?v=B3bAcU2-TGI) by Kit Langton (March 2021) — In this presentation, Kit Langton demonstrated a significant simplification of the service pattern that makes it familiar to OOP programmers, & showed how ZIO 2.0 will auto-wire ZLayers & guide users with rich & actionable error messages.

## ZIO Schedule
- [ZIO Schedule: Conquering Flakiness and Recurrence with Pure Functional Programming](https://www.youtube.com/watch?v=onQSHiafAY8&t=1s) by John De Goes (October 2018)

## ZIO Concurrency Primitives
- [Atomically: Delete Your Actors](https://www.youtube.com/watch?v=d6WWmia0BPM) by John De Goes and Wiem Zine El Abidine (April 2019)
- [ZIO Queue](https://www.youtube.com/watch?v=lBYkLc-j7Vo) by Wiem Zine El Abidine (January 2019)
- [ZIO Queue: A new Queue for a new Era](https://www.youtube.com/watch?v=8JLprl34xEw&t=2437s) by John De Goes (September 2018)
- [ZIO WORLD - Hub](https://www.youtube.com/watch?v=v7Ontn7jZt8) by Adam Fraser (March 2020) — In this presentation, Adam Fraser introduced a hyper-fast async concurrent hub and showed how ZIO Queue and ZIO Hub are radically faster than alternatives.

## ZIO STM
- [Declarative Concurrency with ZIO STM](https://www.youtube.com/watch?v=MEH7hQmGK5M) by Dejan Mijic (June 2020)

## ZIO Test and Debugging
- [0 to 100 with ZIO Test](https://www.youtube.com/watch?v=qDFfVinjDPQ) by Adam Fraser
- [ZIO WORLD - Execution Tracing](https://www.youtube.com/watch?v=1-z06KIde0k) by Rob Walsh (March 2021) — Rob Walsh then presented his work taking ZIO's execution tracing to the next level, which will improve the quality of tracing, as well as make it virtually "cost-free", which is effectively the same as more than doubling the perf of ZIO applications today.
- [Using Aspects To Transform Your Code With ZIO Environment](https://www.youtube.com/watch?v=gcqWdNwNEPg) by Adam Fraser (September 2020)
- [Zymposium — Smart Assertions](https://www.youtube.com/watch?v=lgCb4-4M-fw) by Adam Fraser and Kit Langton (August 2021) — This week, Adam and Kit will demo and explore the inner workings of zio-test's new Smart Assertions. This new test syntax allows you to write simple, straightforward, boolean expressions—all without losing the human-readable, debuggable error messages from other assertion DSLs. As we look at the implementation, we'll discuss the delicate fusion of principled combinator DSLs with hacky macros.

## Migration and Interoperability
- [Functional Legacy - How to Incorporate ZIO In Your Legacy Services](https://www.youtube.com/watch?v=pdgr9bbFQLE) by Natan Silnitsky (March 2020) — You want to introduce ZIO to your existing Scala codebase? Great Idea! It will make your code more efficient, readable, composable, and safe. For the past year, Natan Silnitsky has done this at Wix and has learned a lot about how to do it right. In this talk, Natan will show you how to successfully use ZIO in your legacy service using real-life code examples. You will learn key tips and takeaways including how and when to execute the ZIO runtime; how/when to introduce ZLayers into your codebase; how to make your existing APIs interop with ZIO; and how to have more flexibility on ZManaged resource shutdown. When you're done attending the presentation, you'll be able to introduce ZIO into your existing Scala code base with confidence!

## ZIO Streams
- [Modern Data Driven Applications with ZIO Streams](https://youtu.be/bbss7elSfxs) by Itamar Ravid (December 2019)
- [ZIO Stream: Rebirth](https://www.youtube.com/watch?v=mLJYODobz44&t=15s) by John De Goes and Itamar Ravid (November 2018)
- [The Joys of (Z)Streams](https://www.youtube.com/watch?v=XIIX2YSg7M0) by Itamar Ravid
- [A Tour of ZIO Streams](https://www.youtube.com/watch?v=OiuKbpMOKsc&t=291s) by Itamar Ravid (June 2020) — Watch this video for a brief tour of ZIO Streams from its author, Itamar Ravid. ZIO Streams is a Scala library for creating concurrent, asynchronous stream processing applications, based on the cutting edge functional effect system ZIO.
- [ZIO WORLD - ZIO Streams](https://www.youtube.com/watch?v=8b3t65tmMkE) by Itamar Ravid (March 2020) — In this presentation, Itamar Ravid introduces a radical new basis for ZIO 2.0 Streams, inspired by the work of Michael Snoyman, which unifies streams and sinks & offers next-level performance for streaming apps on the JVM.

## ZIO Libraries
- [What Happens In ZIO Stays in ZIO](https://www.youtube.com/watch?v=gZZazYy0tWM) by Willem Vermeer — ZIO has excellent interoperability with http4s, Cats, Akka libraries, and more, and there are a number of posts and videos that help developers use ZIO with these great libraries. Yet, the question remains: can you develop a web service using ZIO libraries alone? In this talk, we'll explore the answer to this question directly. We'll start by creating an SMTP server using ZIO NIO, and then we'll add a frontend based on ZIO gRPC and Scala.js. Along the way, we'll leverage ZIO Config to externalize the configuration, as well as the ZLayer mechanism for dependency injection. Maybe we'll throw ZIO JSON into the mix, and at the end, we'll take a step back to appreciate how much is really possible gluing those ZIO components together! Finally, we'll conclude by discussing what's still left to be done to make ZIO a full-fledged competitor in the area of full-stack web service development

### ZIO CLI
- [10 Minute Command-Line Apps With ZIO CLI](https://www.youtube.com/watch?v=UeR8YUN4Tws) by Aiswarya Prakasan (December 2020) — Command-line applications are pervasive, relied upon by developers,third-parties, and IT staff. While it’s easy to develop a basic command-line application, building one that features rich help, forgiving parsing, bulletproof validation, shell completions, and a clean and beautiful appearance is quite challenging. In this talk, Aiswarya Prakasan, a contributor to ZIO CLI, will lead you on a journey to building powerful command-line applications in Scala. Through the power of composition and strong types, you will discover how to build type-safe command-line applications in just a few simple lines of code, which give you rich features of command-line applications like git.

### ZIO Flow
- [ZIO WORLD - ZIO FLOW](https://www.youtube.com/watch?v=H4pMkTAsg48) by Aiswarya Prakasan (March 2020) - Aiswarya Prakasan introducing ZIO Flow the new open-source platform will help developers orchestrate microservices for mission-critical code. With a type-safe and compositional design, ZIO Flow will make it easier for developers to create and test robust workflows, which statefully interact with microservices, databases, and human agents, and which survive flakiness, software updates, and even data center failures.
- [Zymposium - ZIO Flow](https://www.youtube.com/watch?v=DDZ8HgWOpBk) (August 2021) — This Friday Adam and Kit will be learning about ZIO Flow, a project that aims to do for distributed programming what ZIO has done for concurrent programming, with special guest Ash. Ash will explain how ZIO Flow lets us execute persistent, distributed, fault-tolerant computations. Then we'll be live coding building a small application in ZIO Flow with her. 

### ZIO SQL
- [ZIO WORLD - ZIO SQL](https://www.youtube.com/watch?v=cIMA6iT9B-k) by Jakub Czuchnowski (March 2020) — Jakub Czuchnowski summarized the latest work in ZIO SQL, demonstrating type-inferred, type-safe queries with the full-range of SQL features, including database-specific functions, with working SELECT/DELETE/UPDATE for Postgres.

### ZIO Web
- [ZIO WORLD - ZIO WEB](https://www.youtube.com/watch?v=UBT-7h8JgU4) by Piotr Golebiewski (March 2020) — Piotr Golebiewski toured ZIO Web, reaching the milestone of "hello world", with working server and client generation for the HTTP protocol.

## Use Cases
- [Search Hacker News with ZIO with Scala, ZIO, Sttp and magic](https://www.youtube.com/watch?v=3P2Gi--dG9A&list=PL-G8WBFTPSVpCcFq6O7czfx9m9T21Cz24&index=11) — A practical look at building a ZIO program. + [Source Code](https://github.com/justinhj/magic-rate-limiter) by [Justin Heyes-Jones](https://twitter.com/justinhj) (April 2021)
- [ML Powered Apps with ZIO Scala](https://www.youtube.com/watch?v=nbZCfkGuNIE) by Aris Vlasakakis (August 2020)
- [Production-grade Microservices with ZIO](https://www.youtube.com/watch?v=oMJ1RMdR7wg) by Itamar Ravid (April 2021) — These days, there are all sorts of boxes to check when deploying production-grade microservices. Our applications must be (reasonably!) performant and correct, but they must also be observable, resilient, and easily extensible. In this talk, Itamar will share from his experience in running microservices based on ZIO in production: resilient, Kubernetes-friendly structuring; cloud-native observability with logs, metrics and telemetry, and modern ways of service communication.

## Others
- [Redis Streams with ZIO](https://youtu.be/jJnco6sMZQY) by [Leszek Gruchała](https://twitter.com/leszekgruchala) (October 2020)
- [The Rise Of Loom And The Evolution Of Reactive Programming](https://www.youtube.com/watch?v=SJeAb-XEIe8&t=2938s) by John A. De Goes (October 2020)
- [Izumi 1.0: Your Next Scala Stack](https://www.youtube.com/watch?v=o65sKWnFyk0) by Pavel Shirshov and Kai (December 2020) — Frameworks are bulky, quirky, and non-compositional, which has led to a rejection of Spring and similar frameworks in the Scala ecosystem. Yet, despite their drawbacks, frameworks have been used to boost team productivity in many large companies. In this presentation, Paul and Kai will introduce Izumi 1.0, a Scala microframework based on compositional functional programming. Designed to help you and your team achieve new levels of productivity, Izumi now includes full compile-time checks for your configurable applications and completely reworked Tagless Final hierarchy for Bifunctors and Trifunctors.
- [ZIO WORLD - Distage 1.0](https://www.youtube.com/watch?v=WJvno8yZuWU) (March 2020) — Distage 1.0, brings the power of tagless-final to trifunctor/bifunctor effect types, with a robust commercial-ready DI framework capable of working with TF & ZLayers w/compile-time validation.
- [Demystifying Functional Effect Systems, Or Build Your Own (Toy) ZIO](https://www.youtube.com/watch?v=Q4OCmKRPUf8) by Dmitry Karlinsky (December 2020) — Ever wondered how ZIO and other functional effect systems work under the hood? Sure feels like magic, sometimes! Dmitry Karlinsky is a big believer in learning by doing, and in this talk, the speaker will walk you through building a toy Scala effect system (called 'TIO'), from scratch, starting with a basic IO monad, and gradually adding capabilities such as Failure and Recovery, Asynchronous Effects, Stack Safety, and finally Concurrency with Fibers. Peek behind the curtain and demystify the technology that is becoming the backbone of modern Scala applications!

---

## ZIO Events

ZIO has a number of events that are organized by the community all around the world. These events are a great way to learn about ZIO, meet other ZIO users and contributors, and have fun.

If you know of an upcoming event related to ZIO, please add it to the list below by submitting a pull request.

## Upcoming Events

### Zymposiums, Online, Every Friday

Every Friday the [Ziverge](https://ziverge.com/) team hosts a Zymposium, an hour-long ZIO-focused live stream. You can find the recordings of past Zymposiums on the [Ziverge YouTube channel](https://www.youtube.com/playlist?list=PLvdARMfvom9C8ss18he1P5vOcogawm5uC).

### ZIO World 2023, Lisbon, Portugal, April 21

The [third edition](https://zioworld.com) of ZIO World brings together ZIO users and contributors worldwide to showcase the power, performance, and productivity of ZIO 2.0 and its broad-based maturing ecosystem. Discover how ZIO 2.0 and a stunning ecosystem of ZIO 2 libraries will give you a competitive edge, making your team more productive and happier than ever before.

### Functional Scala 2023, London, Uk, NOV 30 – DEC 1

[Functional Scala 2023](https://www.functionalscala.com/) is a two-day conference in London with excellent speakers from around the world who are passionate about Scala.

### ZIO Hackathons

ZIO Hackathons are designed to bring together many core contributors to ZIO from all around the world, as well as current ZIO users, future ZIO users and contributors, and contributors to other libraries for async and concurrent programming in the broader functional Scala ecosystem.

The goal of the hackathon is to work on ZIO and related libraries, improve the ecosystem, and have fun. We will have a number of ZIO core contributors present to help with questions and to work on ZIO. So by attending a ZIO Hackathon, you will have the opportunity to pair with other ZIO contributors and to learn from them.

- [ZIO Hackathon, Colorado, USA (in-person & online), 13 September 2023](https://www.eventbrite.com/e/zio-hackathon-colorado-usa-in-person-online-tickets-540880276467)
- [ZIO Hackathon, Inverness, Scotland (in-person and online), 27-28 July 2023](https://www.eventbrite.com/e/zio-hackathon-scotland-in-person-and-online-tickets-539525153257)

## Past Events

The followings are recordings of past ZIO events.

### Zymposiums

<iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLvdARMfvom9C8ss18he1P5vOcogawm5uC" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

### ZIO World 2021

<iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLvdARMfvom9COFgVauwWQ9PsCBOsDmHJm" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

### ZIO World 2022

<iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLvdARMfvom9CK_nXP41eioIXZLHvME5gB" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

### Functional Scala 2021

<iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLvdARMfvom9COGPlva4OnTkFEXzXhhqNH" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

### Functional Scala 2022

<iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLvdARMfvom9A0L97KNywK1lWEeHYDLX0P" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

---

## Frequently Answered Questions (FAQ)

In this page we are going to answer general questions related to the ZIO project.

## Where should we encode contextual values like `UserId`, `CorrelationId` in my ZIO application?

1. Should we put `CorrelationId` and `UserId` as well into a `FiberLocal`?
2. Should our effects be something like `val someEffect: ZIO[CorrerlationId & UserId, ErrorType, A]`?
3. Should we keep writing our effects with explicit params as `def someEffect(c: CorrelationId, u: UserId, params...): ZIO[Any, ErrorType, A]`?
4. Should we put these context parameters as implicits, like `def someEffect(params..)(implicit c: CorrelationId, u: UserId): ZIO[Any, ErrorType, A]`?

Before answering these question, make sure you have read the [ZIO Environment Use-cases](reference/contextual/index.md) section.

Now, let's go into this in a little more detail. We have some workflow, `someEffect` that conceptually requires both a `CorrelationId` and a `UserId` to be run. Let's consider any of these solutions in turn.

### Solution 1: Explicit Parameters

The simplest way to model this dependency is just as function parameters:

```scala
def someEffect(userId: UserId, correlationId: CorrelationId, ...): ZIO[Any, ErrorType, A] =
  ???
```

This is a fine starting point. Functions that take arguments are about as simple as you can get. However, there are two main issues with this.

First, it can just become a lot of boilerplate. The signature above looks potentially okay on its own, but if it is being called by some other function which in turn is being called by some other function it can become a lot of boilerplate very fast which is why people turn to all of these alternatives. I don't think there is a hard and fast rule for when it gets to be "too much" boilerplate but if you are unsure I think a good practice can be to use explicit parameters until you "feel the pain" and then you can refactor to one of the alternatives discussed below.

The second problem with this approach is that someEffect is a method and not a first class value, which can limit our ability to work with it in some cases. But typically the boilerplate is the overwhelming problem that leads people to move away from this approach.

### Solution 2: Implicit Parameters

The second alternative you highlight is to make these parameters implicit.

```scala
def someEffect(...)(implicit userId: UserId, correlationId: CorrelationId): ZIO[Any, ErrorType, A] =
  ???
```

This addresses the boilerplate problem with explicit parameters by allowing us to pass them implicitly. We would basically never recommend this solution.

To reason about implicit values in a principled way we want them to be "coherent" which means there is only one implicit value corresponding to any type within our entire program. For example, there is only one Associative instance for String if we are using functional abstractions, or there is only one `JsonEncoder` instance for Person.

If this requirement is not satisfied we get into anti-patterns like an `implicit ExecutionContext`, where changing our imports or moving a block of code can change which thread pool we run on.

By definition, contextual values like this never satisfy this coherence requirement because there are lots of different `CorrelationId` and `UserId` values in our program. We need to pass them around explicitly or implicitly precisely because there are different ones.

So while there are some cases where there might be different alternatives we want to consider this one I think we can rule out.

### Solution 3 and 4: Environment and FiberRefs

The final two alternatives are modeling these contextual values as part of the ZIO Environment or as `FiberRef` values.

If we model both of these requirements as part of the environment our method signature would look like this:

```scala
def someEffect(...): ZIO[UserId & CorrelationId, ErrorType, A] =
  ???
```

If we model them as `FiberRef` values we would define `FiberRef` values that described both the `UserId` and the `CorrelationId`:

```scala
val currentUserId: FiberRef[UserId] = ???
val currentCorrelationId: FiberRef[CorrelationId] = ???

def someEffect(...): ZIO[Any, ErrorType, A] =
  ???
```

Both of these approaches are similar in that they allow us to avoid the boilerplate associated with passing around the `UserId` and the `CorrelationId`, they allow us to treat `someEffect` as a value, and they allow us to locally modify the current value of the `UserId` and `CorrelationId`.

The main difference between these two approaches is that with the ZIO Environment we reflect the fact that `someEffect` needs a `CorrelationId` and a `UserId` in the type signature, whereas when we use a `FiberRef` this requirement is not reflected in the type signature.

Including these requirements in our type signature can be both an advantage and a disadvantage. The advantage is that we make explicit that `someEffect` requires this contextual information, and we cannot even run `someEffect` without providing it. The disadvantage is that we have to include these requirements in the type signatures of all of our workflows, which can "bubble up" through many method signatures and arguably exposes an implementation detail.

So the question I would ask here, going back to our original answer, is whether including these requirements in your type signature is helpful for you to reason about your program. Some related questions you might ask yourself are "Would it make sense to run the workflow if a requirement were not provided?" and "Is there a sensible default value of this requirement"?

Applying these to the CorrelationId and UserId we would be tempted to not include the `CorrelationId` in the environment. It may depend what we are doing with it but it seems like the `CorrelationId` is a low level implementation detail associated with logging that we do not need cluttering up our method signatures. There seems to be a very sensible default `CorrelationId` of None indicating that there is no `CorrelationId` associated with whatever we are doing and we can still run our program without having a `CorrelationId`, our logs will just not be as helpful as they otherwise would be which we can see and correct.

On the other hand for `UserId` we could easily see coming to the opposite conclusion. If `UserId` is supposed to tell us which user we are supposed to look up in a database or whether we are supposed to be able to look up certain information at all then we may not even be able to run `someEffect` without having a `UserId`. Of course we could just fail at runtime but failing at runtime is much more severe than just logging less precisely and normally we want to use the type system to convert runtime failures to compile time failures. So this seems like it might be a great case for using the ZIO environment.

## In ZIO ecosystem, there are lots of data types which they have `Z` prefix in their names. What this prefix stands for? Does it mean, that data type is effectual?

No, it doesn't denote that the data type is effectual. Instead, the `Z` prefix is used for two purposes:

1. **Polymorphic Version of Another Data Type** — The `Z` prefix indicates a more polymorphic version of another data type, not a data type that is effectual. So for example `IO` and `ZIO` are equally effectual but `ZIO` is more polymorphic because it has the additional type parameter `R`.

2. **Term Disambiguation** — There are some cases where the `Z` prefix is used to disambiguate a term that might otherwise be too common and create risk of name conflicts (e.g. `ZPool`).

This convention is true across all ZIO ecosystem. For example, in ZIO Prelude, the `ZValidation` is a more general version of `Validation` that is polymorphic in the log type. `ZSet` is a more polymorphic version of a _Set_ that is polymorphic in the measure type. `ZPure` is more polymorphic than its type aliases in several ways as represented by its different type parameters and also serves to disambiguate it as _Pure_ which is too general.

---

## ZIO Adopters

Following is a partial list of companies happily using ZIO in production to craft concurrent applications.

Want to see your company here? [Submit a PR](https://github.com/zio/zio/edit/series/2.x/docs/adopters.md)!

| Company                                                          | Logo                                                              |
| :--------------------------------------------------------------: | :---------------------------------------------------------------: |
| [AdGear / Samsung Ads](https://adgear.com/en/)                   |                     |
| [Adidas](https://www.adidas.com/)                                |                     |
| [adpulse.io](https://www.adpulse.io/)                            |                   |
| [adsquare](https://www.adsquare.com/)                            | ![Adsquare](/img/users/adsquare.svg)                              |
| [Ayolab](https://www.ayolab.com/)                                |             |
| [Asana](https://asana.com/)                                      |                     |
| [Aurinko](https://www.aurinko.io/)                               |                  |
| [auto.ru](https://auto.ru)                                       |                      |
| [AutoScout24](https://www.autoscout24.de)                        |              |
| [Avast](https://www.avast.com)                                   |                    |
| [Bank of America](https://www.bofa.com)                          |                     |
| [CalcBank](https://www.calcbank.com.br)                          |                  |
| [Call Handling](https://www.callhandling.co.uk/)                 |              |
| [Carvana](https://www.carvana.com)                               |                   |
| [Cellular](https://www.cellular.de)                              |                 |
| [Cloudfarms](https://cloudfarms.com)                             |               |
| [CodeComprehension](https://codecomprehension.com)               |        |
| [Codept](https://www.codept.de/)                                 |                   |
| [Colisweb](https://www.colisweb.com/en)                          |                  |
| [Collibra](https://www.collibra.com/)                            |                 |
| [Compellon](https://www.compellon.com/)                          |                |
| [Complicated Robot](https://www.complicatedrobot.com/)           |                                                                   |
| [Conduktor](https://www.conduktor.io)                            |                |
| [Coralogix](https://coralogix.com)                               |                |
| [CurrencyCloud](https://www.currencycloud.com/)                  |             |
| [D.E.Solution](https://de-solution.com/)                         |               |
| [DataChef](https://datachef.co)                                  |                  |
| [Demyst](https://demyst.com)                                     |                    |
| [Devsisters](https://devsisters.com/)                            |                |
| [DHL Parcel The Netherlands](https://www.werkenbijdhl.nl/it)     |             |
| [Doomoolmori](https://doomoolmori.com/)                          |              |
| [Dow Jones](https://www.dowjones.com)                            |                 |
| [DPG recruitment](https://www.dpgrecruitment.nl)                 |          |
| [Deutsche Telekom IoT GmbH](https://iot.telekom.com/en)          |               |
| [eBay](https://www.ebay.com)                                     |                     |
| [Eaglescience](https://www.eaglescience.nl)                      |             |
| [Electricité de France (EDF)](https://www.edf.fr/)               |                      |
| [EnelX](https://www.enelx.com)                                   |                   |
| [Evo](https://evo.company)                                       |                      |
| [Flipp](https://flipp.com/)                                      |                     |
| [Fugo.ai](https://www.fugo.ai)                                   |                     |
| [Glean](https://www.gleancompany.com)                            |             |
| [GrandParade](https://grandparade.co.uk)                         |              |
| [greyflower.media GmbH](https://greyflower.media)                |                                                                   |
| [Hunters.AI](https://hunters.ai)                                 |                  |
| [iHeartRadio](https://www.iheart.com/)                           |                   |
| [Investsuite](https://investsuite.com/)                          |              |
| [Kaizen Solutions](https://kaizen-solutions.net/)                |          |
| [Kamon APM](https://kamon.io/)                                   |         |
| [Kensu](https://kensu.io)                                        |                    |
| [LambdaWorks](https://www.lambdaworks.io/)                       |               |
| [LeadIQ](https://leadiq.com)                                     |                    |
| [LiveIntent Inc.](https://liveintent.com)                        |               |
| [MATECHS](https://matechs.com)                                   |                  |
| [Megogo](https://megogo.net)                                     |                   |
| [Mylivn](https://www.mylivn.com/)                                |                    |
| [NewMotion](https://newmotion.com)                               |                |
| [Nexxchange](https://www.nexxchange.com)                         |                |
| [NsLookup](https://www.nslookup.io)                              |                 |
| [Ocado Technology](https://ocadotechnology.com)                  |               |
| [Olyro GmbH](https://olyro.de)                                   |                     |
| [Optrak](https://optrak.com)                                     |                   |
| [Performance Immo](https://www.performance-immo.com/)            |          |
| [Playtika](https://www.playtika.com)                             |                 |
| [PPC Samurai](https://ppcsamurai.com/)                           |                |
| [Randstad Groep Nederland](https://www.werkenbijrandstad.nl)     |        |
| [PIM Solutions](https://pimsolutions.ru/)                        |              |
| [Risk Ident](https://riskident.com/)                             |                |
| [Rocker](https://rocker.com/)                                    |                   |
| [Rudder](https://www.rudder.io/)                                 |                   |
| [Sanjagh](https://sanjagh.pro/)                                  |                  |
| [Scalac](https://scalac.io/)                                     |                   |
| [SecurityScorecard](https://www.securityscorecard.io/)           |        |
| [SentinelOne](https://www.sentinelone.com/)                      |              |
| [Signicat](https://www.signicat.com/)                            |                  |
| [SoftwareMill](https://softwaremill.com/)                        |             |
| [StreamWeaver](https://www.streamweaver.com/)                    |             |
| [Stuart](https://stuart.com/)                                    |                    |
| [Teads](https://teads.com)                                       |                     |
| [TomTom](https://tomtom.com)                                     |                   |
| [Tinkoff](https://tinkoff.ru)                                    |                  |
| [Tranzzo](https://tranzzo.com)                                   |                  |
| [Tweddle Group](https://tweddle.com)                             |                   |
| [Unit](https://unit.co)                                          |                      |
| [Univalence](https://univalence.io)                              |               |
| [Vakantiediscounter](https://www.vakantiediscounter.nl)          |       |
| [Verbund AG](https://www.verbund.com)                            |                  |
| [Wehkamp](https://www.wehkamp.nl)                                |                   |
| [Wolt](https://www.wolt.com/)                                    |                      |
| [Audela](https://audela.ca)                                      |                   |
| [Valamis Group](https://valamis.com)                             |                   |
| [VirtusLab](https://virtuslab.com/)                              |                 |
| [Vish](https://getvish.com)                                      |                      |
| [Zalando](https://zalando.com/)                                  |                  |
| [Zooz](https://zooz.com/)                                        |                      |

---

## Compile Time Errors for Handling Combinators

ZIO provides a variety of combinators to handle errors such as `orElse`, `catchAll`, `catchSome`, `option`, `either`, and `retry`. However, these combinators only make sense for effects that can fail (i.e. where the error type is not `Nothing`). To help you identify code that doesn't make sense, error handling combinators require implicit evidence `CanFail[E]`, which is automatically available for all types except `Nothing`. The table below includes a list of combinators that only make sense for effects that can fail along with value preserving rewrites.

## ZIO

Code | Rewrite 
--- | ---
`uio <> zio` | `uio`
`uio.catchAll(f)` | `uio`
`uio.catchSome(pf)` | `uio`
`uio.either` | `uio`*
`uio.eventually` | `uio`
`uio.flatMapError(f)` | `uio`
`uio.fold(f, g)` | `uio.map(g)`
`uio.foldZIO(f, g)` | `uio.flatMap(g)`
`uio.mapBoth(f, g)` |  `uio.map(g)`
`uio.mapError(f)` | `uio`
`uio.option` | `uio`*
`uio.orDie` | `uio`
`uio.orDieWith(f)` | `uio`
`uio.orElse(zio)` | `uio`
`uio.orElseEither(zio)` | `uio`*
`uio.orElseFail(e)` | `uio`
`uio.asElseSucceed(a)` | `uio`
`uio.refineOrDie(pf)` | `uio`
`uio.refineOrDieWith(pf)(f)` | `uio`
`uio.refineToOrDie` | `uio`
`uio.retry(s)` | `uio`
`uio.retryOrElse(s, f)` | `uio`
`uio.retryOrElseEither(s, f)` | `uio`*
`uio.tapBoth(f, g)` | `uio.tap(g)`
`uio.tapError(f)` | `uio`
`ZIO.partitionZIO(in)(f)` | `ZIO.foreach(in)(f)`*
`ZIO.partitionZIOPar(in)(f)` | `ZIO.foreachPar(in)(f)`*
`ZIO.validateZIO(in)(f)` | `ZIO.foreach(in)(f)`*
`ZIO.validateFirstZIO(in)(f)` | `ZIO.foreach(in)(f)`*

## ZStream

Code | Rewrite 
--- | ---
`ustream.catchAll(f)` | `ustream`
`ustream.either` | `ustream`*
`ustream.mapBoth(f, g)` | `ustream.map(g)`
`ustream.mapError(f)` | `ustream`
`ustream.orElse(zstream)` | `ustream`

## ZStreamChunk

Code | Rewrite 
--- | ---
`ustream.either` | `ustream`
`ustream.orElse(zstream)` | `ustream`

## (*) Notes:

- `either`, `option`, `orElseEither`, and `retryOrElseEither` wrap their results in `Some` or `Right` so after rewriting, code calling these methods can be simplified to accept an `A` rather than an `Option[A]` or `Either[E, A]`. 

- `partitionZIO`, `partitionZIOPar`, `validateZIO` and `validateFirstZIO` have error accumulating semantics on either error channel or success channel. After rewrite the error type can be simplified to `E` rather than `List[E]` or the success type `List[B]` instead of `(List[E], List[B])`.

---

## Code of Conduct

We are committed to providing a friendly, safe and welcoming
environment for all, regardless of level of experience, gender, gender
identity and expression, sexual orientation, disability, personal
appearance, body size, race, ethnicity, age, religion, nationality, or
other such characteristics.

The ZIO project follows the [Scala Code of Conduct](https://www.scala-lang.org/conduct/), with
an additional clause regarding moderation that is detailed below. All participants, contributors and 
members are expected to follow the Scala Code of Conduct when discussing the project on the available 
communication channels. If you are being harassed, please contact us immediately so that we can support you.

## Moderation and Steering Committee
 
The ZIO project is moderated by the Steering Committee team members:

- Itamar Ravid [@iravid](https://github.com/iravid)
- John De Goes [@jdegoes](https://github.com/jdegoes)
- Kai [@neko-kai](https://github.com/neko-kai)
- Paul Shirshov [@pshirshov](https://github.com/pshirshov)
- Pierre Ricadat [@ghostdogpr](https://github.com/ghostdogpr)
- Wiem Zine El Abidine [@wi101](https://github.com/wi101)

The ZIO project requires that drastic moderation actions detailed in the code of 
conduct - for example, removing a user from the Discord Server - be agreed upon
a group of over 2/3rds of the steering committee.

For any questions, concerns, or moderation requests please contact any member of
the project, or the people listed above.

## BDFL

In addition to the above, the ZIO project's BDFL (benevolent dictator for life) is
John De Goes (@jdegoes), owing to his original founding of the project and continued
investments in it. While John adheres to the same code of conduct as everyone else,
he is entitled to override moderation decisions made by the steering committee.

We do not take the BDFL position lightly, especially with regards to moderation. John
has consistently shown he is level-headed and able to handle conflict responsibly. Feel
free to reach out to any member of the steering committee, including John himself,
with any concern you might have.

---

## Coding Guidelines

These are coding guidelines strictly for ZIO contributors for ZIO projects and 
not general conventions to be applied by the Scala community at large.

Additionally, bear in mind that, although we try to enforce these rules to the 
best of our ability, both via automated rules (scalafmt) and strict reviewing 
processes, it is both possible to find existing code that does not comply to 
these rules. If that is the case, we would be extremely grateful if you could 
make a contribution, by providing a fix to said issue.

Last, but not least, these rules are continuously evolving and as such, 
refer to them once in a while when in doubt.

### Defining classes and traits

1. Value classes must be final and extend `AnyVal`. 
This is done to avoid allocating runtime objects; 

2. Method extension classes must be final and extend `AnyVal`;

3. Avoid overloading standard interfaces. When creating services avoid using the same names as well known standard interfaces.
Example: Instead of having a service `Random` with methods `nextLong(n)` and `nextInt(n)` consider choosing something like 
`nextLongBounded(n)` and `nextIntBounded(n)`.

4. Sealed traits that are ADTs (Algebraic data types) should extend `Product` and `Serializable`. 
This is done to help the compiler infer types;

5. Regular traits and sealed trait that do not form ADTs should extend `Serializable` but not `Product`;

6. Traits should always extend `Serializable`. (e.g. `ZIO`).

### Final and private modifiers 

1. All methods on classes / traits are declared `final`, by default;

2. No methods on objects declared `final`, because they are `final` by default;

3. No methods on final classes declared `final`, because they are `final` by default;

4. All classes inside objects should be defined `final`, because otherwise they could still be extended;

5. In general, classes that are not case classes have their constructors & constructor parameters private. 
   Typically, it is not good practice to expose constructors and constructor parameters but exceptions apply (i.e. `Assertion` and `TestAnnotation`);

6. All `vals` declared `final`, even in objects or `final classes`, if they are constant expressions and without type annotations;

7. Package-private `vals` and methods should be declared `final`.

### Refactoring

1. If a class has all its members `final`, the class should be declared `final` and `final` member annotations should be removed except constant expressions;

2. All type annotations should use the least powerful type alias. This means, that, let us say, a `ZIO` effect that has 
   no dependencies but throws an arbitrary error, should be defined as `IO`.
3. Use `def` in place of `val` for an abstract data member to avoid `NullPointerException` risk.

### Understanding naming of parameters or values

ZIO code often uses the following naming conventions, and you might be asked to change method parameters to follow these conventions. This guide can help you understand where the names come from. 
Naming expectations can be helpful in understanding the role of certain parameters without even glancing at its type signature when reading code or class/method signatures.

1. Partial functions have a shortened name `pf`;

2. In ZIO implicit parameters are often used as compiler evidences;
   These evidences help you, as a developer, prove something to the compiler (at compile time), and they have the ability to add constraints to a method;
   They are typically called `ev` if there is only one. Or `ev1`, `ev2`... if more than one;
   
3. Promises are called `p` (unless in its own class methods, in that case it is called `that`, like point 8 defines);

4. Functions are called `fn`, `fn1`, unless they bear specific meaning: `use`, `release`;

5. ZIO effects are called `f`, unless they bear specific meaning like partially providing environment: `r0`;

6. Iterable are called `in`;

7. When a parameter type equals own (in a method of a trait) call it `that`;

8. Be mindful of using by-name parameters. Mind the `Function[0]` extra allocation and loss of clean syntax when invoking the method.
   Loss of syntax means that instead of being able to do something like `f.flatMap(ZIO.success)` you require to explicitly do `f.flatMap(ZIO.success(_))`;
   
9. Fold or fold variants initial values are called `zero`.

### Understanding naming of methods

ZIO goes to great lengths to define method names that are intuitive to the library user. Naming is hard!!! 
This section will attempt to provide some guidelines and examples to document, guide and explain naming of methods in ZIO.

1. All operators that return effects should be lazy in all parameters that are not lambdas. Strict parameters are a source of bugs when users inadvertently call side effecting code
   in arguments to these parameters. Preventing these bugs is the responsibility of the effect system and lazy parameters allow the runtime to properly manage these side effects.
   There are two exceptions. First, strict parameters should be used when by name parameters would cause duplicate method signatures due to type erasure. Second, operators on high
   performance concurrent data structures such as `Ref`, `Queue`, and `Hub` should be strict in their parameters.

2. Methods that have the form of `List#zip` are called `zip`, and have an alias called `<*>`. The parallel version, if applicable, has the name `zipPar`, with an alias called `<&>`;

3. Avoid the use of `effect` in constructors as this pushes responsibility on users to identify what code is or is not side effecting whereas the effect system should handle this
   correctly regardless. Instead prefer more specific names such as `succeed`, `attempt`, or `suspend`.

4. The dual of zip, which is trying either a left or right side, producing an Either of the result, should be called `orElseEither`, with alias `<+>`. 
   The simplified variant where both left and right have the same type should be called `orElse`, with alias `<>`;
    
5. Constructors for a data type `X` that are based on another data type `Y` should be placed in the companion object `X` and named `fromY`. 
   For example, `ZIO.fromOption`, `ZStream.fromEffect`;
   
6. Parallel versions of methods should be named the same, but with a `Par` suffix.

7. `foreach` should be used for operators that effectually iterate over a collection. For example, `ZIO.foreach`.

8. Variants of operators that accept arguments or return results in the context of an effect type should be suffixed by the name of the effect type, for example `mapZIO` or
   `mapSTM`. The `M` suffix should not be used as it is not idiomatic Scala and does not specify what effect type is involved.

9. Use the `Discard` suffix for variants of methods that discard their results. For example, `foreachDiscard`. The `_` suffix should not be used as it is not idiomatic Scala and
   does not describe what it does.

10. Methods that are necessarily side effecting should be prefixed with `unsafe`, for example `unsafeRun`. This does not apply to methods on internal data types that are inherently imperative in nature, for example `MutableConcurrentQueue`.
   
### Type annotations

ZIO goes to great lengths to take advantage of the scala compiler in varied ways. Type variance is one of them. 
The following rules are good to have in mind when adding new `types`, `traits` or `classes` that have either covariant or contravariant types.

1. Generalized ADTs should always have type annotation. (i.e. `final case class Fail[+E](value: E) extends Cause[E]`);
   
2. Type alias should always have type annotation. Much like in Generalized ADTs defining type aliases should carry the type annotations 
   (i.e. `type IO[+E, +A] = ZIO[Any, E, A]`).

When defining new methods, keep in mind the following rules:

1. Accept the most general type possible. For example, if a method accepts a collection, prefer `Iterable[A]` to `List[A]`.
2. Return the most specific type possible, e.g., prefer `UIO[Unit]` to `UIO[Any]`.

### Method alphabetization

In general the following rules should be applied regarding method alphabetization. 
To fix forward references of values we recommend the programmer to make them lazy (`lazy val`).
Operators are any methods that only have non-letter characters (i.e. `<*>` , `<>`, `*>`).

1. Public abstract defs / vals listed first, and alphabetized, with operators appearing before names.

2. Public concrete defs / vals listed second, and alphabetized, with operators appearing before names.

3. Private implementation details listed third, and alphabetized, with operators appearing before names.

### Scala documentation

It is strongly recommended to use scala doc links when referring to other members. 
This both makes it easier for users to navigate the documentation and enforces that the references are accurate.
A good example of this are `ZIO` type aliases that are extremely pervasive in the codebase: `Task`, `RIO`, `URIO` and `UIO`.
To make it easy for developers to see the implementation scala doc links are used, for example:

```
  /**
   * @see See [[zio.ZIO.absolve]]
   */
  def absolve[R, A](v: RIO[R, Either[Throwable, A]]): RIO[R, A] =
    ZIO.absolve(v)
```

---

## Contributing to The ZIO Documentation

The ZIO documentation is provided by a worldwide community, just like the project itself. So if you are reading this page, you can help us to improve the documentation.

Please read the [Contributor Guideline](contributor-guidelines.md) before contributing to documentation.

## Toolchain

1. The documentation is written in [Markdown](https://en.wikipedia.org/wiki/Markdown) format.
2. During the build process of the microsite, we use [mdoc](https://scalameta.org/mdoc/) to type-check code snippets in Markdown.
3. Our site generator is [Docusaurus](https://docusaurus.io/).

## Editing with GitHub Editor

We encourage contributors to use GitHub's editor for making minor changes to existing documents.

1. On each page, there is a button called _Edit this page_, by clicking this button, we will be redirected to the GitHub editor.

![Edit this page](/img/assets/edit-this-page.png)

2. After editing the page we can check whether our changes have been formatted correctly or not by using the _Preview_ tab.

![GitHub Editor](/img/assets/github-editor.png)

3. We can scroll to the bottom of the page and write a title and description of the work, and then propose the changes by clicking on _Propose changes_.

![Propose changes](/img/assets/propose-changes.png)

4. Our browser will be redirected to a new page titled _Comparing changes_ after clicking the _Propose changes_ button. We can compare our changes proposal and then create a pull request by clicking the _Create pull request_ button.

![Open a pull request](/img/assets/comparing-changes.png)

5. On the new page, we can edit the title and description of our pull request on the new page and finally click _Create pull request_.

![Open a pull request](/img/assets/open-a-pull-request.png)

6. A pull request has been created. Eventually, our work will be reviewed by the rest of the team.

## Editing Documentation Locally

ZIO contributors are encouraged to use this approach for introducing new documentation pages, or when we have lots of improvements on code snippets since we can compile check all changes locally before committing and sending a pull request to the project:

1. First, we need to fork and clone the ZIO project on our machine. Follow the [Get The Project](contributor-guidelines#get-the-project) instructions to fork the repository and clone your fork.

2. The documentation source code can be found in the `docs` directory and they are all in Markdown format. Now we can begin improving the existing documentation or adding new documentation.

3. To generate documentation site from type-checked markdowns we can use the following command:

```bash
sbt docs/mdoc
```

If one of our snippet codes fails to compile, this command doesn't succeed and will guide us on which line of the documentation caused this error.

It is recommended to run this command with sbt shell with the `--watch` option. This will start a file watcher and livereload on changes. It's useful when we want to see the intermediate results while we are writing documentation:

```bash
sbt
sbt:docs> docs/mdoc --watch
```

4. Finally, by the following command we can serve the microsite locally:

```bash
cd website
npm install
npm run start --watch
```

It will be served on [localhost](http://127.0.0.1:3000/) address.

5. When we are finished with the documentation, we can commit those changes and [create a pull request](contributor-guidelines.md#create-a-pull-request).

## AI-friendly Markdown Variants

The website publishes `/llms.txt` and `/llms-full.txt` at the site root (per the [llmstxt.org](https://llmstxt.org/) standard) so LLMs and AI agents can discover and ingest the documentation without scraping HTML. It also serves a Markdown variant next to every documentation page — for example, the page rendered at `/reference/schedule/` is also available as plain Markdown at `/reference/schedule.md`.

Two pieces cooperate to produce this:

1. The [`docusaurus-plugin-llms`](https://github.com/rachfop/docusaurus-plugin-llms) plugin generates `llms.txt`, `llms-full.txt`, and a `.md` variant for every documentation source file. Its configuration lives in `website/docusaurus.config.js`.
2. A small post-build script, `website/scripts/llms-md-mirror.js`, runs after `docusaurus build` (from the `build` entry in `website/package.json`) and mirrors the generated Markdown to match the canonical page URL.

The mirror step exists because `docusaurus-plugin-llms` writes each `.md` at the *source file path*, while Docusaurus renders HTML at a different path in two common cases:

- `reference/schedule/index.md` is rendered at `/reference/schedule/`, so the natural Markdown URL is `/reference/schedule.md`, not `/reference/schedule/index.md`.
- `reference/core/zio/zio.md` is rendered at `/reference/core/zio/` (Docusaurus's folder-named-doc convention), so the natural Markdown URL is `/reference/core/zio.md`.

The script scans the build output and, for every `<dir>/index.md` or `<dir>/<dir>.md` it finds, creates a sibling `<dir>.md` so appending `.md` to any rendered URL resolves. It never overwrites an existing file, so its output composes cleanly with the llms plugin and with any future upstream fix. Running after `docusaurus build` (rather than as another Docusaurus plugin) avoids racing against the llms plugin — Docusaurus runs plugins' `postBuild` hooks concurrently, so a mirror plugin could walk the build directory before the llms plugin had finished writing its `.md` files. This keeps the solution in one place rather than requiring explicit `slug:` frontmatter on every index and folder-named-doc file across the site and the ~30 ecosystem subprojects synced in from npm.

## Giving Feedback

Sometimes we see some problem in the documentation, or we have some idea to make better documentation, but we haven't time or knowledge to do that personally. We can discuss those ideas with the community. There are two ways to do this:

1. Using Discord (https://discord.gg/2ccFBr4) is a great way to share our thoughts with others, discuss them, and brainstorm big ideas.
2. Opening a new issue (https://github.com/zio/zio/issues/new) is appropriate when we have actionable ideas, such as reorganizing a specific page of a documentation, or a problem with the current documentation.

---

## Contributing to the ZIO Ecosystem Projects

The ZIO ecosystem is provided by a worldwide community, just like the project itself. So if you are reading this page, you can help us to improve the ecosystem.

Please read the [Contributor Guideline](contributor-guidelines.md) before contributing to the ecosystem.

## Branching Naming Convention

Until now, the [ZIO core library](https://github.com/zio/zio) has undergone two major releases: 1.x and 2.x. Therefore, the ZIO ecosystem projects may maintain two branches: `series/1.x` and `series/2.x`, with the default branch typically set to `series/2.x`. The `series/1.x` branch accomodates releases compatible with ZIO 1.x, while the `series/2.x` branch is designated for releases compatible with ZIO 2.x.

When considering a contribution to a ZIO ecosystem project, it's essential to determine the targeted ZIO version beforehand. Once identified, we can proceed to checkout the corresponding branch, make changes, and submit a pull request against that specific branch.

## Documentation

The documentation for official libraries is also part of the ZIO documentation. However, the source code for their documentation is not housed within the [ZIO repository](https://github.com/zio/zio). Instead, they are located in the `docs` directory of each library repository.

As an example, the documentation for the ZIO Prelude library, accessible at [https://zio.dev/zio-prelude](https://zio.dev/zio-prelude), can be found within its respective respository at [https://github.com/zio/zio-prelude/tree/series/2.x/docs](https://github.com/zio/zio-prelude/tree/series/2.x/docs).

Each official ZIO project maintains its dedicated documentation project, which is published with every release to the npm registry under the [@zio.dev](https://www.npmjs.com/package/@zio.dev/) organization. Subsequently, these documentation projects are seamlessly integrated into the [ZIO Website](https://zio.dev) during the CI process of [ZIO core](https://github.com/zio/zio) repository.

## Documentation Structure

The base minimum structure of the documentation source code under the `docs` directory of each library repository is as follows:

```shell
~/zio-xyz > tree docs
docs
├── index.md
├── package.json
└── sidebars.js
```

Now, let's delve into each of them.

### The `package.json` File

Each library's documentation source code is a NPM project that will be published to the NPM registry. So the `package.json` file is required. The file is used to specify the name, description, and license of the documentation project. Example:

```json
{
  "name": "@zio.dev/zio-prelude",
  "description": "ZIO Prelude Documentation",
  "license": "Apache-2.0"
}
```

That is, the name of the documentation project must be `@zio.dev/<library-name>`, and the description and license must be the same as the library. All official libraries will be published under the [`@zio.dev`](https://www.npmjs.com/package/@zio.dev/) scope on the npm registry. For example, the ZIO Prelude's documentation package will be published at [https://www.npmjs.com/package/@zio.dev/zio-prelude](https://www.npmjs.com/package/@zio.dev/zio-prelude). We will discuss more about this on the CI section.

### The `index.md` File

The `index.md` file serves as the main page of the documentation, providing a description of the library and its features. The structure of this file is as follows:

```markdown
---
id: index
title: "Introduction to ZIO XYZ"
sidebar_title: "ZIO XYZ"
---

ZIO XYZ is a ZIO Scala library that ...

@‎PROJECT_BADGES@

## Introduction

Key features of ZIO XYZ are:
  - Feature 1
  - Feature 2
  - ...

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

‎`‎`‎`‎scala
libraryDependencies += "dev.zio" %% "zio-xyz" % "@‎VERSION@"
`‎`‎`‎

## Example

Let's see an example of how to use this library:

....
```

Inside the `index.md` file, the `sidebar_title` field is used to set the name of the library in the sidebar of the documentation website on [this page](ecosystem/officials/index.md). Therefore, make sure to set it at the top of the `docs/index.md` file.

Also, we can use `@‎PROJECT_BADGES@` placeholder to add badges to the documentation. The `@‎PROJECT_BADGES@` placeholder will be replaced with the badges of the library. Also, the `@‎VERSION@` placeholder will be replaced with the latest version of the library.

The `index.md` file is the primary source for generating the `README.md` file of the library. Further details on this process will be discussed in the [ZIO SBT Website Plugin](#zio-sbt-website-plugin) section.

### The `sidebars.js` File

Inside the `sidebars.js` file, we can specify the documentation's structure, including the order of pages, the arrangement of sections within each page, and the introduction of sections and subsections. The structure of this file is as follows:

```js
const sidebars = {
  sidebar: [
    {
      type: "category",
      label: "ZIO XYZ", // Project Name
      collapsed: false,
      link: { type: "doc", id: "index" },
      items: [ ]
    }
  ]
};

module.exports = sidebars;
```

To add a new page to the documentation, in addition to the `index.md` page, we can include a new item in the `items` array. For instance, to add a new page with the id `getting-started`, we can do the following:

```js
const sidebars = {
  sidebar: [
    {
      type: "category",
      label: "ZIO XYZ", // Project Name
      collapsed: false,
      link: { type: "doc", id: "index" },
      items: [ ]
    }
  ]
};

module.exports = sidebars;
```

To introduce a new section, simply append a new object to the `items` array. Set the `type` field to `category`, the `label` field to the desired section name, the `link` field to the main page's ID corresponding to the section, and finally, populate the `items` field with a list of pages to be included in that section:

```js
const sidebars = {
  sidebar: [
    {
      type: "category",
      label: "ZIO XYZ", // Project Name
      collapsed: false,
      link: { type: "doc", id: "index" },
      items: [
        `installation`,
        `getting-started`,
        `basics`,
        {
          type: "category",
          label: "Examples",
          collapsed: false,
          link: { type: "doc", id: "examples/index" },
          items: [
            `examples/example-1`,
            `examples/example-2`
          ]
        }
      ]
    }
  ]
};

module.exports = sidebars;
```

In the above example, we have introduced a new section called `Examples` which refers to the `examples/index.md` page. Additionally, we have added two pages to this section.

To learn more about the `sidebars.js` file, please refer to the [Docusaurus documentation](https://docusaurus.io/docs/sidebar).

## ZIO SBT Website Plugin

After preparing the documentation source code inside the `docs` directory of the library repository, we are ready to compile the documentation. To do this, we need to install the `zio-sbt-website` plugin.

With this plugin, we gain the capability to enhance our documentation in several ways:

1. **Ensure Type-Checked Code Snippets:** Leverage the power of the [`mdoc`](https://scalameta.org/mdoc/) tool to include type-checked code snippets seamlessly within our documentation.
2. **Effortless README.md Generation:** Streamline our documentation process by generating the `README.md` file automatically from the content inside `docs/index.md` file.
3. **Simplified Documentation Deployment:** Install and preview our documentation website effortlessly.

### Installing the Plugin

To install the plugin, we need to add the following line to the `project/plugins.sbt` file of the library repository:

```scala
// replace <version> with the latest version currently 0.4.0-alpha.22
addSbtPlugin("dev.zio" % "zio-sbt-website" % "<version>") 
```

To enable the plugin, we suggest adding new submodule to the `build.sbt` file called `docs` and then configuring the plugin for that submodule:

```scala
lazy val root = (project in file("."))
  .settings(publish / skip := true)
  .aggregate(core, docs)

lazy val core = (project in file("zio-xyz"))

lazy val docs = (project in file("zio-xyz-docs"))
  .enablePlugins(WebsitePlugin)
  .settings(projectName := "ZIO XYZ")
  .settings(mainModuleName := (core / moduleName).value)
  .settings(projectStage := ProjectStage.ProductionReady)
```

Let's explain the above settings:

- `projectName`: This setting is used to set the name of the documentation project.
- `mainModuleName`: This setting is used to specify the name of the main module of the library. It is employed to create the `@‎PROJECT_BADGES@` placeholder in the `docs/index.md` file.
- `projectStage`: This setting is used to determine the stage of the documentation project. Possible values are `ProductionReady`, `Development`, `Experimental`, `Research`, `Concept`, and `Deprecated`. It is utilized to create the `@‎PROJECT_STAGE@` placeholder in the `docs/index.md` file. To learn more about these stages, please refer to [this page](https://github.com/zio#project-status).

### Configuring Readme Settings

After installing the plugin, we can generate the `README.md` file by running the `sbt docs/generateReadme` command. This command will generate the `README.md` file from the `docs/index.md` file and place it in the root directory of the repository.

:::note
As a contributor to a well-established official library, you don't need to update the `README.md` file every time you make changes to the `docs/index.md` file. The CI of the ZIO repository will take care of this for you. After merging any pull request that has changes to the `docs/index.md` file, the CI of the ZIO repository will automatically create a new pull request to update the `README.md` file. We will discuss this later.
:::

There are some sbt settings used to generate the `README.md` file, and they have default values. However, if you want to customize them, you can do so. These settings are as follows:

- `readmeBanner`
- `readmeDocumentation`
- `readmeContribution`
- `readmeCodeOfConduct`
- `readmeSupport`
- `readmeMaintainers`
- `readmeCredits`
- `readmeAcknowledgement`
- `readmeLicense`

For example, to change the credits section, we can add the following setting to the `build.sbt` file:

```scala
lazy val docs = (project in file("zio-xyz-docs"))
  .enablePlugins(WebsitePlugin)
  .settings(projectName := "ZIO XYZ")
  .settings(mainModuleName := (core / moduleName).value)
  .settings(projectStage := ProjectStage.ProductionReady)
  .settings(
    readmeCredits :=
      """
        |The creation of this library is deeply influenced by the exploration and  
        | implementation conducted at ABC corporation.""".stripMargin,
  )
```

### Live Preview of Documentation Website

Before creating pull requests for the documentation of an official library, it is advisable to check if everything is fine, especially when there are numerous changes to the documentation. To do so, we can locally build the documentation by following these steps:

1. Install the local website using the `sbt docs/installWebsite` command.
2. Run the `sbt docs/previewWebsite` command to serve the website locally.
3. Open the http://localhost:3000/ address in the browser.

By editing the documentation source code of the library, we can observe the changes on the local website.

:::note
If the `sbt docs/previewWebsite` command fails, please clean the `target` directory for the documentation module and try again from step 1.
:::

## Publishing Documentation

To contribute to the documentation of an official library, one must make changes to the documentation source code of that library and then create a pull request to the respective repository. Upon merging that pull request, the goal is to have the documentation for that library updated on the zio.dev website with the release of the next version of the library.

To achieve this, the only thing that needs to be done from the library repository perspective is to publish the documentation to the npm registry. There are two options for this:

1. Write a manual CI job responsible for publishing the documentation package after each release of the library.
2. Use the ZIO SBT CI plugin to generate the CI jobs automatically, including the job for publishing the documentation package.

## Writing Manual CI for Publishing Documentation

### Releasing Documentation Package

By adding the following CI job to the `.github/workflows/<workflow-name>.yaml` file, we can utilize the `sbt docs/publishToNpm` command provided by the ZIO SBT Website plugin to publish the documentation package to the npm registry:

```yaml
permissions:
  id-token: write    # Required for OIDC trusted publishing
  contents: read

release-docs:
  name: Release Docs
  runs-on: ubuntu-latest
  continue-on-error: false
  needs:
  - release
  if: ${{ ((github.event_name == 'release') && (github.event.action == 'published')) }}
  steps:
  - name: Git Checkout
    uses: actions/checkout@v4.1.1
    with:
      fetch-depth: '0'
  - name: Setup NodeJs
    uses: actions/setup-node@v4
    with:
      node-version: 20.x
      registry-url: https://registry.npmjs.org
  - name: Publish Docs to NPM Registry
    run: sbt docs/publishToNpm
```

This CI job uses **npm Trusted Publishing with OpenID Connect (OIDC)** for secure authentication. No npm token secret is required. Instead:

1. The workflow requests an OIDC token from GitHub (via the `id-token: write` permission)
2. The `setup-node` action configures npm to use this OIDC token for authentication
3. The npm package on npmjs.com must be configured to accept OIDC tokens from your repository

**Setup Instructions for npm Trusted Publisher:**

1. Go to your npm package settings page: `https://www.npmjs.com/package/@zio.dev/<package-name>/settings/access`
2. Add a **Trusted Publisher** with:
   - **Provider**: GitHub Actions
   - **Organization**: `zio`
   - **Repository**: `<your-repo-name>`
   - **Workflow filename**: `ci.yml` (or your actual workflow filename - must be exact, not a wildcard)
   - **Environment name**: (leave blank unless you use a specific environment)

For more information, see the [npm Trusted Publishers documentation](https://docs.npmjs.com/trusted-publishers/).

### Notifying the Main Repository

After publishing the documentation package, we need to notify the main repository about the new release of the documentation package. To do so, we can add the following CI job to the `.github/workflows/<workflow-name>.yaml` file:

```yaml
notify-docs-release:
  name: Notify Docs Release
  runs-on: ubuntu-latest
  continue-on-error: false
  needs:
  - release-docs
  if: ${{ (github.event_name == 'release') && (github.event.action == 'published') }}
  steps:
  - name: Git Checkout
    uses: actions/checkout@v4.1.1
    with:
      fetch-depth: '0'
  - name: notify the main repo about the new release of docs package
    run: |
      PACKAGE_NAME=$(cat docs/package.json | grep '"name"' | awk -F'"' '{print $4}')
      PACKAGE_VERSION=$(npm view $PACKAGE_NAME version)
      curl -L \
        -X POST \
        -H "Accept: application/vnd.github+json" \
        -H "Authorization: token ${{ secrets.PAT_TOKEN }}"\
          https://api.github.com/repos/zio/zio/dispatches \
          -d '{
                "event_type":"update-docs",
                "client_payload":{
                  "package_name":"'"${PACKAGE_NAME}"'",
                  "package_version": "'"${PACKAGE_VERSION}"'"
                }
              }'
```

This CI job will create a new dispatch event to the ZIO repository, which will then generate a new pull request to update the documentation package version in the `website/package.json` file of the ZIO repository. After merging that pull request, the new version of the documentation will be deployed to the zio.dev website.

### Checking Website Build

Before releasing the documentation package, it is advisable to check if everything is fine. To do so, we can add the following CI steps to `build` phase of the CI:

```diff
  build:
    runs-on: ubuntu-22.04
    timeout-minutes: 60
    steps:
      - name: Checkout current branch
        uses: actions/checkout@v4.1.1
      - name: Setup Java
        uses: actions/setup-java@v4.2.1
        with:
          distribution: temurin
          java-version: 17
          check-latest: true
+     - name: Setup NodeJs
+       uses: actions/setup-node@v4
+       with:
+         node-version: 16.x
+         registry-url: https://registry.npmjs.org
      - name: Cache scala dependencies
        uses: coursier/cache-action@v6
      - name: Check Building Packages
        run: ./sbt +publishLocal
+     - name: Check Website Build Process
+       run: sbt docs/clean; sbt docs/buildWebsite
```

### Automate README Generation

Everytime we change the content of the `docs/index.md` file, we need to update the `README.md` file. To automate this process, we can add the following CI job to the `.github/workflows/<workflow-name>.yaml` file:

```yaml
update-readme:
  name: Update README
  runs-on: ubuntu-latest
  continue-on-error: false
  if: ${{ github.event_name == 'push' }}
  steps:
  - name: Git Checkout
    uses: actions/checkout@v4.1.1
    with:
      fetch-depth: '0'
  - name: Setup Scala
    uses: actions/setup-java@v4.2.1
    with:
      distribution: temurin
      java-version: 17
      check-latest: true
  - name: Cache Dependencies
    uses: coursier/cache-action@v6
  - name: Generate Readme
    run: sbt docs/generateReadme
  - name: Commit Changes
    run: |
      git config --local user.email "zio-assistant[bot]@users.noreply.github.com"
      git config --local user.name "ZIO Assistant"
      git add README.md
      git commit -m "Update README.md" || echo "No changes to commit"
  - name: Generate Token
    id: generate-token
    uses: zio/generate-github-app-token@v1.0.0
    with:
      app_id: ${{ secrets.APP_ID }}
      app_private_key: ${{ secrets.APP_PRIVATE_KEY }}
  - name: Create Pull Request
    id: cpr
    uses: peter-evans/create-pull-request@v5.0.2
    with:
      body: |-
        Autogenerated changes after running the `sbt docs/generateReadme` command of the [zio-sbt-website](https://zio.dev/zio-sbt) plugin.

        I will automatically update the README.md file whenever there is new change for README.md, e.g.
          - After each release, I will update the version in the installation section.
          - After any changes to the "docs/index.md" file, I will update the README.md file accordingly.
      branch: zio-sbt-website/update-readme
      commit-message: Update README.md
      token: ${{ steps.generate-token.outputs.token }}
      delete-branch: true
      title: Update README.md
  - name: Approve PR
    if: ${{ steps.cpr.outputs.pull-request-number }}
    run: gh pr review "$PR_URL" --approve
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      PR_URL: ${{ steps.cpr.outputs.pull-request-url }}
  - name: Enable Auto-Merge
    if: ${{ steps.cpr.outputs.pull-request-number }}
    run: gh pr merge --auto --squash "$PR_URL" || gh pr merge --squash "$PR_URL"
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      PR_URL: ${{ steps.cpr.outputs.pull-request-url }}
```

This CI job will create a new pull request to update the `README.md` file after each push event. The pull request will be merged automatically if the CI job is successful. It utilizes the [ZIO Assistant](https://github.com/apps/zio-assistant), which requires a token to be generated. The token will be generated using the [`zio/generate-github-app-token`](https://github.com/zio/generate-github-app-token) action.

## Utilizing Autogenerated CI through the ZIO SBT CI Plugin (Recommended)

The ZIO SBT CI plugin can generate the CI jobs automatically, including the job for publishing the documentation package to the NPM registry using npm Trusted Publishing with OIDC.

To do this, first, we need to install the plugin in the `project/plugins.sbt` file of the library repository:

```scala
// replace <version> with the latest version currently 0.4.0-alpha.22
addSbtPlugin("dev.zio" % "zio-sbt-ci" % "<version>") 
```

By adding this line, the plugin will be enabled, and we can use the `sbt ciGenerateGithubWorkflow` command to generate the CI jobs. The generated CI jobs will be placed in the `.github/workflows/ci.yml`.

**Important:** After generating the workflow, you must configure your npm package as a **Trusted Publisher** on npmjs.com:

1. Go to: `https://www.npmjs.com/package/@zio.dev/<package-name>/settings/access`
2. Add a **Trusted Publisher** with:
   - **Provider**: GitHub Actions
   - **Organization**: `zio`
   - **Repository**: `<your-repo-name>`
   - **Workflow filename**: `ci.yml` (exact match required, not a wildcard)
   - **Environment name**: (leave blank)

:::note
After each release of an official library, the documentation for that library will be published to the npm registry. In the CI of the ZIO repository, the latest versions of documentations that are published to the NPM registry will be downloaded and integrated into the zio.dev site.

The generated CI uses npm Trusted Publishing with OIDC for secure, token-free authentication. No NPM_TOKEN secret is needed.
:::

To learn more about the ZIO SBT CI plugin, please refer to [this page](https://zio.dev/zio-sbt).

## ZIO SBT Ecosystem Plugin (Configuring Project's SBT Settings)

Over time, within the ZIO Ecosystem projects, we've identified a common thread: the need to configure specific settings when initiating a new project. These settings often share a commonality across projects, albeit with some variations.

This involves setting up crucial elements such as build information, test configurations, documentation settings, macro settings, activating industry-standard plugins, ensuring cross-platform compatibility, and configuring supported Scala versions, among other essential parameters.

The ZIO SBT Ecosystem was created to address this need. It attempts to configure common settings among ZIO ecosystem projects. Besides providing various sbt tasks/settings, it also configures and installs essential SBT plugins.

Let's take a closer look at this plugin and see how to use it.

### Installing the Plugin

To install the plugin, we need to add the following line to the `project/plugins.sbt` file of the library repository:

```scala
// replace <version> with the latest version currently 0.4.0-alpha.22
addSbtPlugin("dev.zio" % "zio-sbt-ecosystem" % "<version>")
```

Then add the following line to the `build.sbt` file of the library repository:

```scala
enablePlugins(ZioSbtEcosystemPlugin)
```

Now, let's explore the various settings available for configuring our project modules:

### Global Settings

After enabling the `zio-sbt-ecosystem` plugin, it will automatically apply some global settings, such as `licenses` to `Apache2`, `organization` to `dev.zio`, homepage to `https://zio.dev/<project-name>`, and also configuring the `scmInfo` setting.

### Project Settings

Additionally, there are some settings that will be activated in the scope of each project, such as introducing new commands and tasks, as well as the help manual:

1. Commands:
   1. `fix` - Fixes source files using scalafix.
   2. `fmt` - Formats source files using scalafmt.
   3. `quiet` - Disable/enable welcome banner.
   4. `lint` - Verifies that all source files are properly formatted and have had all scalafix rules applied.
   5. `prepare` - Prepares sources by applying scalafmt and running scalafix:
   6. `publishAll` - Signs and publishes all artifacts to Maven Central.
2. Tasks:
   1. `build` - Compiles all sources including tests.
   2 `enableStrictCompile` - Enables strict compile option.
   3 `disableStrictCompile` - Disables strict compile option.

### Build Settings

The following settings will be activated at the build scope:

1. `scala3` - Version of Scala 3 to use.
2. `scala212` - Version of Scala 2.12 to use.
3. `scalaVersion` - The default version of scala used for build.
4. `crossScalaVersions` - The versions of Scala used when cross-building which by default is set to `Seq(scala212.value, scala213.value, scala3.value)`
5. `zioVersion` - The version of ZIO to use.
6. `javaPlatform` - The java platform to use which by default is set to `8`

### Helper Functions

The ZIO SBT Ecosystem Plugin also provides some helper functions that can be used in the `build.sbt` file to configure the project:

1. `stdSettings`
2. `enableZIO` 
3. `crossProjectSettings`
4. `scalafixSettings`
5. `scalaReflectTestSettings`
6. `buildInfoSettings`  
7. `macroDefinitionSettings`
8. `jsSettings` and `scalajs`
9. `nativeSettings`

## Configuring Scala Steward

[Scala Steward](https://github.com/scala-steward-org/scala-steward) is a bot that helps you keep library dependencies and sbt plugins up-to-date. It is a GitHub App that automatically updates dependencies in Scala projects with pull requests.

To configure Scala Steward on a repository inside the ZIO organization, we have two options:

1. Using the public Scala Steward instance (recommended)
2. Using the ZIO Scala Steward instance

Most of the ZIO ecosystem projects use the public instance of Scala Steward, and we recommend using it. To enable Scala Steward on a repository, you can add the project to [this file](https://github.com/VirtusLab/scala-steward-repos/blob/main/repos-github.md) and then create a pull request.

However, some projects may require the ZIO Scala Steward instance hosted on the ZIO organization. In this case, please contact the ZIO team through the [ZIO's Contributor Channel on Discord](https://discord.com/channels/629491597070827530/629491597070827534). After enabling the ZIO Scala Steward bot on the repository, add the following CI job to the `.github/workflows/scala-steward.yml` file:

```yaml
name: Scala Steward

# This workflow will launch everyday at 00:00
on:
  schedule:
    - cron: '0 0 * * *'
  workflow_dispatch: {}

jobs:
  scala-steward:
    timeout-minutes: 45
    runs-on: ubuntu-latest
    name: Scala Steward
    steps:
      - name: Scala Steward
        uses: scala-steward-org/scala-steward-action@v2.61.0
        with:
          github-app-id: ${{ secrets.SCALA_STEWARD_GITHUB_APP_ID }}
          github-app-installation-id: ${{ secrets.SCALA_STEWARD_GITHUB_APP_INSTALLATION_ID }}
          github-app-key: ${{ secrets.SCALA_STEWARD_GITHUB_APP_PRIVATE_KEY }}
          github-app-auth-only: true
```

## Project Checklist

As a ZIO ecosystem contributor, you can help us improve by checking if the project follows our guidelines. Some are must-follow, others are optional. Follow as many as you can. If the project doesn't meet any, kindly make a pull request to fix it. Here are the guidelines:

### General Checklist

1. [ ] The project has a `LICENSE` file (required).
2. [ ] The CI workflow is generated using the `zio-sbt-ci` plugin (optional).
3. [ ] The project uses the `zio-sbt-ecosystem` plugin to configure SBT settings (optional).

### Documentation Checklist

1. [ ] The documentation project uses the `zio-sbt-website` plugin (required).
2. [ ] The project has well-written documentation in the `docs` directory (required).
3. [ ] The project has a `docs/index.md` file that serves as the main page of the documentation (required).
4. [ ] The project has a `docs/sidebars.js` file that defines the structure of the documentation (required).
5. [ ] The project has a `docs/package.json` file that defines the name, description, and license of the documentation project (required).
6. [ ] The documentation project uses type-safe code snippets using the `mdoc` tool (optional).

### CI Checklist

Check if all the following items are done on `pull_request` and `push` (push to the main branch) and `release` events:

1. [ ] Lint: Check if all source files are properly formatted (required).
2. [ ] Test: Check if all tests are passing (required).
3. [ ] Build: The CI checks if the documentation is building successfully (required).
   1. [ ] Check if all source files (including the test sources) are compiled successfully.
   2. [ ] Check if packaging the project is successful.
   3. [ ] Check if packaging the documentation project is successful.

For releasing the project, check if all the following items are done on `release` event:

1. [ ] If the lint, test, and build CI jobs are successful, release the project and its documentation package to the Maven Central and NPM registries (required).
2. [ ] After the release job, notify the main repository about the new release of the project (optional).

On `push` events to the main branch, check if all the following items are done:

1. [ ] The CI generates a pull request to update the `README.md` file and creates a pull request to the main repository (optional).

---

## Contributor Guidelines

Thank you for your interest in contributing to ZIO, which is a small, zero-dependency library for doing type-safe, composable concurrent and asynchronous programming!

We welcome contributions from all people! You will learn about functional programming, and you will add your own unique touch to the ZIO project. We are happy to help you to get started and to hear your suggestions and answer your questions.

_You too can contribute to ZIO, we believe in you!_

# Contributing

## Getting Started

To begin contributing, please follow these steps:

### Get The Project

If you do not already have one, sign up for a free [GitHub Account](https://github.com/join?source=header-home).

After you [log into](https://github.com/login) GitHub using your account, go to the [ZIO Project Page](https://github.com/zio/zio), and click on [Fork](https://github.com/zio/zio/fork) to fork the ZIO repository into your own account.

You will make _all_ contributions from your own account. No one contributes _directly_ to the main repository. Contributors only ever merge code from other people's forks into the main repository.

Once you have forked the repository, you can now clone your forked repository to your own machine, so you have a complete copy of the project and can begin safely making your modifications (even without an Internet connection).

To clone your forked repository, first make sure you have installed [Git](https://git-scm.com/downloads), the version control system used by GitHub. Then open a Terminal and type the following commands:

```bash
mkdir zio
cd zio
git clone git@github.com:your-user-name/zio.git .
```

Then you need to add a remote upstream to the original ZIO project repository and fetch tags that will be used for proper project versioning by the build tool:

```bash
git remote add upstream git@github.com:zio/zio.git
git fetch --tags upstream
```

If these steps were successful, then congratulations, you now have a complete copy of the ZIO project!

The next step is to build the project on your machine, to ensure you know how to compile the project and run tests.

### Build the Project

The official way to build the project is with [sbt](https://www.scala-sbt.org/download/). An sbt build file is included in the project, so if you choose to build the project this way, you won't have to do any additional configuration or setup (others choose to build the project using IntelliJ IDEA, Gradle, Maven, Mill, or Fury).

To launch this sbt from your Terminal window, simply type:

```bash
sbt -J-Xmx8g
```

Where `-J-Xmx8g` is an optional settings to increase maximum heap size of JVM that could be useful to avoid slowdown on default settings.

Sbt will launch, read the project build file, and download dependencies as required.

You can now compile the production source code with the following sbt command:

```bash
compile
```

You can compile the test source code with the following sbt command:

```bash
Test/compile
```

[Learn more](https://www.scala-sbt.org) about sbt to understand how you can list projects, switch projects, and otherwise manage an sbt project.

The main project in ZIO is `coreJVM` (the core project on the JVM; there is also `coreJS` for the core project on Scala.js), which you can focus on using sbt by issuing the following command:

```bash
project coreJVM
```

### Find an Issue

You may have your own idea about what contributions to make to ZIO, which is great! If you want to make sure the ZIO contributors are open to your idea, you can [open an issue](https://github.com/zio/zio/issues/new) first on the ZIO project site.

Otherwise, if you have no ideas about what to contribute, you can find a large supply of feature requests and bugs on the project's [issue tracker](https://github.com/zio/zio/issues).

Issues are tagged with various labels, such as `good first issue`, which help you find issues that are a fit for you.

If some issue is confusing or you think you might need help, then just post a comment on the issue asking for help. Typically, the author of the issue will provide as much help as you need, and if the issue is critical, leading ZIO contributors will probably step in to mentor you and give you a hand, making sure you understand the issue thoroughly.

Once you've decided on an issue and understand what is necessary to complete the issue, then it's a good idea to post a comment on the issue saying that you intend to work on it. Otherwise, someone else might work on it too!

### Fix an Issue

Once you have an issue, the next step is to fix the bug or implement the feature. Since ZIO is an open source project, there are no deadlines. Take your time!

The only thing you have to worry about is if you take too long, especially for a critical issue, eventually someone else will come along and work on the issue.

If you shoot for 2-3 weeks for most issues, this should give you plenty of time without having to worry about having your issue stolen.

If you get stuck, please consider [opening a pull request](https://github.com/zio/zio/compare) for your incomplete work, and asking for help (just prefix the pull request by _WIP_). In addition, you can comment on the original issue, pointing people to your own fork. Both of these are great ways to get outside help from people more familiar with the project.

### Prepare Your Code

If you've gotten this far, congratulations! You've implemented a new feature or fixed a bug. Now you're in the last mile, and the next step is submitting your code for review, so that other contributors can spot issues and help improve the quality of the code.

To do this, you need to commit your changes locally. A good way to find out what you did locally is to use the `git status` command:

```bash
git status
```

If you see new files, you will have to tell `git` to add them to the repository using `git add`:

```bash
git add core/src/shared/zio/zio/NewFile.scala
```

Then you can commit all your changes at once with the following command:

```bash
git commit -am "Fixed #94211 - Optimized race for lists of effects"
```

At this point, you have saved your work locally, to your machine, but you still need to push your changes to your fork of the repository. To do that, use the `git push` command:

```bash
git push
```

Now while you were working on this great improvement, it's quite likely that other ZIO contributors were making their own improvements. You need to pull all those improvements into your own code base to resolve any conflicts and make sure the changes all work well together.

To do that, use the `git pull` (for the 2.x branch) command:

```bash
git pull git@github.com:zio/zio.git series/2.x
```

You may get a warning from Git that some files conflicted. Don't worry! That just means you and another contributor edited the same parts of the same files.

Using a text editor, open up the conflicted files, and try to merge them together, preserving your changes and the other changes (both are important!).

Once you are done, you can commit again:

```bash
git commit -am "merged upstream changes"
```

At this point, you should re-run all tests to make sure everything is passing:

```bash
# If you are already in a SBT session you can type only 'test'

sbt test
```

If all the tests are passing, then you can format your code to be shipped:

```bash
# If you are already in a SBT session you can type only 'fmt'

sbt fmt
```

For simplicity, there is a command that does everything. Formats the code, compiles it and runs tests:

```bash
# If you are already in a SBT session you can type only 'build'

sbt build
```

If your changes altered an API, then you may need to rebuild the microsite to make sure none of the (compiled) documentation breaks:

```bash
# If you are already in a SBT session you can type only 'docs/docusaurusCreateSite'

sbt docs/docusaurusCreateSite
```

(If you get an error about _Jekyll_, that means all the code examples work and you can ignore the rest.)

Finally, if you are up-to-date with master, all your tests are passing, you have properly formatted your code, and the microsite builds properly, then it's time to submit your work for review!

### Create a Pull Request

To create a pull request, first push all your changes to your fork of the project repository:

```bash
git push
```

Next, [open a new pull request](https://github.com/zio/zio/compare) on GitHub, and select _Compare Across Forks_. On the right hand side, choose your own fork of the ZIO repository, in which you've been making your contribution.

Provide a description for the pull request, which details the issue it is fixing, and has other information that may be helpful to developers reviewing the pull request.

Finally, click _Create Pull Request_!

### Get Your Pull Request Merged

Once you have a pull request open, it's still your job to get it merged! To get it merged, you need at least one core ZIO contributor to approve the code.

If you know someone who would be qualified to review your code, you can request that person, either in the comments of the pull request, or on the right-hand side, if you have appropriate permissions.

Code reviews can sometimes take a few days, because open source projects are largely done outside of work, in people's leisure time. Be patient, but don't wait forever. If you haven't gotten a review within a few days, then consider gently reminding people that you need a review.

Once you receive a review, you will probably have to go back and make minor changes that improve your contribution and make it follow existing conventions in the code base. This is normal, even for experienced contributors, and the rigorous reviews help ensure ZIO's code base stays high quality.

If iterating on a feature resulted in several commits, consider squashing them before pushing the final version of the code.
One of the ways to squash e.g. 5 last commits: 
```bash
git reset --soft HEAD~5
git commit -am "add feature X"
git push --force
```

After you make changes, you may need to remind reviewers to check out the code again. If they give a final approval, it means your code is ready for merge! Usually this will happen at the same time, though for controversial changes, a contributor may wait for someone more senior to merge.

If you don't get a merge in a day after your review is successful, then please gently remind folks that your code is ready to be merged.

Sit back, relax, and enjoy being a ZIO contributor!

# ZIO Contributor License Agreement

Thank you for your interest in contributing to the ZIO open source project.

This contributor agreement ("Agreement") describes the terms and conditions under which you may Submit a Contribution to Us. By Submitting a Contribution to Us, you accept the terms and conditions in the Agreement. If you do not accept the terms and conditions in the Agreement, you must not Submit any Contribution to Us.

This is a legally binding document, so please read it carefully before accepting the terms and conditions. If you accept this Agreement, the then-current version of this Agreement shall apply each time you Submit a Contribution. The Agreement may cover more than one software project managed by Us.

## 1. Definitions

"We" or "Us" means Ziverge, Inc., and its duly appointed and authorized representatives.

"You" means the individual or entity who Submits a Contribution to Us.

"Contribution" means any work of authorship that is Submitted by You to Us in which You own or assert ownership of the Copyright. You may not Submit a Contribution if you do not own the Copyright in the entire work of authorship.

"Copyright" means all rights protecting works of authorship owned or controlled by You, including copyright, moral and neighboring rights, as appropriate, for the full term of their existence including any extensions by You.

"Material" means the work of authorship which is made available by Us to third parties. When this Agreement covers more than one software project, the Material means the work of authorship to which the Contribution was Submitted. After You Submit the Contribution, it may be included in the Material.

"Submit" means any form of electronic, verbal, or written communication sent to Us or our representatives, including but not limited to electronic mailing lists, electronic mail, source code control systems, pull requests, and issue tracking systems that are managed by, or on behalf of, Us for the purpose of discussing and improving the Material, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."

"Submission Date" means the date on which You Submit a Contribution to Us.

"Effective Date" means the earliest date You execute this Agreement by Submitting a Contribution to Us.

## 2. Grant of Rights

### 2.1 Copyright License

2.1.1. You retain ownership of the Copyright in Your Contribution and have the same rights to use or license the Contribution which You would have had without entering into the Agreement.

2.1.2. To the maximum extent permitted by the relevant law, You grant to Us a perpetual, worldwide, non-exclusive, transferable, royalty-free, irrevocable license under the Copyright covering the Contribution, with the right to sublicense such rights through multiple tiers of sublicensees, to reproduce, modify, display, perform and distribute the Contribution as part of the Material; provided that this license is conditioned upon compliance with Section 2.3.

### 2.2 Patent License

For patent claims including, without limitation, method, process, and apparatus claims which You own, control or have the right to grant, now or in the future, You grant to Us a perpetual, worldwide, non-exclusive, transferable, royalty-free, irrevocable patent license, with the right to sublicense these rights to multiple tiers of sublicensees, to make, have made, use, sell, offer for sale, import and otherwise transfer the Contribution and the Contribution in combination with the Material (and portions of such combination). This license is granted only to the extent that the exercise of the licensed rights infringes such patent claims; and provided that this license is conditioned upon compliance with Section 2.3.

### 2.3 Outbound License

Based on the grant of rights in Sections 2.1 and 2.2, if We include Your Contribution in a Material, We may license the Contribution under any license, including copyleft, permissive, commercial, or proprietary licenses. As a condition on the exercise of this right, We agree to also license the Contribution under the terms of the license or licenses which We are using for the Material on the Submission Date.

### 2.4 Moral Rights

If moral rights apply to the Contribution, to the maximum extent permitted by law, You waive and agree not to assert such moral rights against Us or our successors in interest, or any of our licensees, either direct or indirect.

### 2.5 Our Rights

You acknowledge that We are not obligated to use Your Contribution as part of the Material and may decide to include any Contribution We consider appropriate.

### 2.6 Reservation of Rights

Any rights not expressly licensed under this section are expressly reserved by You.

## 3. Agreement

You confirm that:

a. You have the legal authority to enter into this Agreement.

b. You own the Copyright and patent claims covering the Contribution which are required to grant the rights under Section 2.

c. The grant of rights under Section 2 does not violate any grant of rights which You have made to third parties, including Your employer. If You are an employee, You have had Your employer approve this Agreement or sign the Entity version of this document. If You are less than eighteen years old, please have Your parents or guardian sign the Agreement.

d. You have followed the instructions in, if You do not own the Copyright in the entire work of authorship Submitted.

## 4. Disclaimer

EXCEPT FOR THE EXPRESS WARRANTIES IN SECTION 3, THE CONTRIBUTION IS PROVIDED "AS IS". MORE PARTICULARLY, ALL EXPRESS OR IMPLIED WARRANTIES INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE EXPRESSLY DISCLAIMED BY YOU TO US. TO THE EXTENT THAT ANY SUCH WARRANTIES CANNOT BE DISCLAIMED, SUCH WARRANTY IS LIMITED IN DURATION TO THE MINIMUM PERIOD PERMITTED BY LAW.

## 5. Consequential Damage Waiver

TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL YOU BE LIABLE FOR ANY LOSS OF PROFITS, LOSS OF ANTICIPATED SAVINGS, LOSS OF DATA, INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL AND EXEMPLARY DAMAGES ARISING OUT OF THIS AGREEMENT REGARDLESS OF THE LEGAL OR EQUITABLE THEORY (CONTRACT, TORT OR OTHERWISE) UPON WHICH THE CLAIM IS BASED.

## 6. Miscellaneous

6.1. This Agreement will be governed by and construed in accordance with the laws of the state of Maryland, in the United States of America, excluding its conflicts of law provisions. Under certain circumstances, the governing law in this section might be superseded by the United Nations Convention on Contracts for the International Sale of Goods ("UN Convention") and the parties intend to avoid the application of the UN Convention to this Agreement and, thus, exclude the application of the UN Convention in its entirety to this Agreement.

6.2. This Agreement sets out the entire agreement between You and Us for Your Contributions to Us and overrides all other agreements or understandings.

6.3. If You or We assign the rights or obligations received through this Agreement to a third party, as a condition of the assignment, that third party must agree in writing to abide by all the rights and obligations in the Agreement.

6.4. The failure of either party to require performance by the other party of any provision of this Agreement in one situation shall not affect the right of a party to require such performance at any time in the future. A waiver of performance under a provision in one situation shall not be considered a waiver of the performance of the provision in the future or a waiver of the provision in its entirety.

6.5. If any provision of this Agreement is found void and unenforceable, such provision will be replaced to the extent possible with a provision that comes closest to the meaning of the original provision and which is enforceable. The terms and conditions set forth in this Agreement shall apply notwithstanding any failure of essential purpose of this Agreement or any limited remedy to the maximum extent possible under law.

---

## Introduction to izumi-reflect

# izumi-reflect

> @quote: Looks a bit similar to TypeTag

`izumi-reflect` is a fast, lightweight, portable and efficient alternative for `TypeTag` from `scala-reflect`.

`izumi-reflect` is a lightweight model of Scala type system and provides a simulator of the important parts of the Scala typechecker.

## Why `izumi-reflect`

1. `izumi-reflect` compiles faster, runs a lot faster than `scala-reflect` and is fully immutable and [thread-safe](https://github.com/scala/bug/issues/10766),
2. `izumi-reflect` supports Scala 2.11, 2.12, 2.13 and **Scala 3**,
3. `izumi-reflect` supports Scala.js and Scala Native,
4. `izumi-reflect` works well with [GraalVM Native Image](https://www.graalvm.org/reference-manual/native-image/),
5. `izumi-reflect` allows you to obtain tags for unapplied type constructors (`F[_]`) and combine them at runtime.

## Credits

`izumi-reflect` has been created by [Septimal Mind](https://7mind.io) to power [Izumi Project](https://github.com/7mind/izumi),
as a replacement for `TypeTag` in reaction to a lack of confirmed information about the future of `scala-reflect`/`TypeTag` in Scala 3 ([Motivation](https://blog.7mind.io/lightweight-reflection.html)), and donated to ZIO.

  
  
  

## Limitations

`izumi-reflect` model of the Scala type system is not 100% precise, but "good enough" for the vast majority of the usecases.

Known limitations are:

1. Recursive type bounds (F-bounded types) are not preserved and may produce false positives,
2. Existential types, both written with wildcards and `forSome` may produce unexpected results, the support is limited,
3. Path-Dependent Types are based on variable names and may cause unexpected results when variables with different names have the same type or vice-versa (vs. Scala compiler)
4. This-Types such as `X.this.type` are ignored and identical to `X`
5. `izumi-reflect` is less powerful than `scala-reflect`: it does not preserve fields and methods when it's not necessary for equality and subtype checks, it does not preserve code trees, internal compiler data structures, etc.
6. There are some optimizations in place which reduce correctness, namely: subtype check for `scala.Matchable` will always return true, no distinction is made between `scala.Any` and `scala.AnyRef`.
7. Lower bounds are not preserved in abstract higher-kinded type members which may produce false comparisons.
8. Type and value members are not preserved in concrete types which may produce false comparisons with refined/structural types. (https://github.com/zio/izumi-reflect/issues/481)

## Debugging

Set [`-Dizumi.reflect.debug.macro.rtti=true`](https://javadoc.io/doc/dev.zio/izumi-reflect_2.13/latest/izumi/reflect/DebugProperties$.html#izumi.reflect.debug.macro.rtti:String(%22izumi.reflect.debug.macro.rtti%22)) to enable debug output during compilation when tags are constructed and at runtime when they are compared.

```shell
sbt -Dizumi.reflect.debug.macro.rtti=true
```

To see debug output when compiling in Intellij, add the above flag to `VM options` in [Preferences -> Build, Execution, Deployment -> Compiler -> Scala Compiler -> Scala Compile Server](jetbrains://idea/settings?name=Build%2C+Execution%2C+Deployment--Compiler--Scala+Compiler--Scala+Compile+Server)

You may also set it in `.jvmopts` file during development. (`.jvmopts` properties will not apply to Intellij compile server, only to sbt)

Set `-Dizumi.reflect.debug.macro.rtti.assertions=true` to enable additional assertions.

Other useful system properties are:

- [`izumi.reflect.rtti.optimized.equals`](https://javadoc.io/doc/dev.zio/izumi-reflect_2.13/latest/izumi/reflect/DebugProperties$.html#izumi.reflect.rtti.optimized.equals:String(%22izumi.reflect.rtti.optimized.equals%22))
- [`izumi.reflect.rtti.cache.compile`](https://javadoc.io/doc/dev.zio/izumi-reflect_2.13/latest/izumi/reflect/DebugProperties$.html#izumi.reflect.rtti.cache.compile:String(%22izumi.reflect.rtti.cache.compile%22))

# Talks

* [Kit Langton — Scala 3 Macro Fun (Open Source Hackery)](https://www.youtube.com/watch?v=wsLhjqCKZuU)
* [Pavel Shirshov — Izumi Reflect: Scala Type System Model](https://www.youtube.com/watch?v=ShRfzVz58OY)

# See also

## [`gzoller/scala-reflection`](https://github.com/gzoller/scala-reflection)

* Scala 3 only
* No support for subtype checks
* Type lambdas are not supported
* _Preserves field information_

## [`airframe-surface`](https://wvlet.org/airframe/docs/airframe-surface)

* Scala 2 and Scala 3
* No support for subtype checks
* _Preserves field information_

## And even more

1. https://github.com/gaeljw/typetrees - a very basic type tag substitute for Scala 3
2. https://stackoverflow.com/questions/75752812/is-there-a-simple-scala-3-example-of-how-to-use-quoted-type-as-replacement-for - discussion on StackOverflow
3. https://contributors.scala-lang.org/t/scala-3-and-reflection/3627 - original discussion on Scala Contributors forum

[Stage]: https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg
[Stage-Page]: https://github.com/zio/zio/wiki/Project-Stages

---

## ZIO AMQP(Zio-amqp)

ZIO AMQP is a ZIO-based wrapper around the RabbitMQ client. It provides a streaming interface to AMQP queues and helps to prevent you from shooting yourself in the foot with thread-safety issues. 

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-amqp/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-amqp_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-amqp_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-amqp_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-amqp_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-amqp-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-amqp-docs_2.13) [![ZIO AMQP](https://img.shields.io/github/stars/zio/zio-amqp?style=social)](https://github.com/zio/zio-amqp)

## Installation

Add the following lines to your `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-amqp" % "1.0.0-alpha.3"
```

The latest version is built against ZIO 2.0.0-RC6.

## Consuming

The example below creates a connection to an AMQP server and then creates a channel. Both are created as Managed resources, which means they are closed automatically after using even in the face of errors.

The example then creates a stream of the messages consumed from a queue named `"queueName"`. Each received message is acknowledged back to the AMQP server.

```scala

val channel: ZIO[Scope, Throwable, Channel] = for {
  connection <- Amqp.connect(URI.create("amqp://my_amqp_server_uri"))
  channel    <- Amqp.createChannel(connection)
} yield channel

val effect: ZIO[Any, Throwable, Unit] =
  ZIO.scoped {
    channel.flatMap { channel =>
      channel
        .consume(queue = QueueName("queueName"), consumerTag = ConsumerTag("test"))
        .mapZIO { record =>
          val deliveryTag = record.getEnvelope.getDeliveryTag
          printLine(s"Received ${deliveryTag}: ${new String(record.getBody)}") *>
            channel.ack(DeliveryTag(deliveryTag))
        }
        .take(5)
        .runDrain
    }
  }
```

See the [ZIO documentation](https://zio.dev/docs/overview/overview_running_effects#defaultruntime) for more information on how to run this effect or integrate with an existing application.

---

## Artifacts

# Published artifacts

## Core module

```scala
"dev.zio" %% "zio-aws-core" % "7.30.16.1"
```

## HTTP client modules:

```scala
"dev.zio" %% "zio-aws-akka-http" % "7.30.16.1"
"dev.zio" %% "zio-aws-http4s" % "7.30.16.1"
"dev.zio" %% "zio-aws-netty" % "7.30.16.1"
"dev.zio" %% "zio-aws-crt-http" % "7.30.16.1"
```

## List of all the generated libraries:

```scala
"dev.zio" %% "zio-aws-accessanalyzer" % "7.30.16.1"
"dev.zio" %% "zio-aws-account" % "7.30.16.1"
"dev.zio" %% "zio-aws-acm" % "7.30.16.1"
"dev.zio" %% "zio-aws-acmpca" % "7.30.16.1"
"dev.zio" %% "zio-aws-amp" % "7.30.16.1"
"dev.zio" %% "zio-aws-amplify" % "7.30.16.1"
"dev.zio" %% "zio-aws-amplifybackend" % "7.30.16.1"
"dev.zio" %% "zio-aws-amplifyuibuilder" % "7.30.16.1"
"dev.zio" %% "zio-aws-apigateway" % "7.30.16.1"
"dev.zio" %% "zio-aws-apigatewaymanagementapi" % "7.30.16.1"
"dev.zio" %% "zio-aws-apigatewayv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-appconfig" % "7.30.16.1"
"dev.zio" %% "zio-aws-appconfigdata" % "7.30.16.1"
"dev.zio" %% "zio-aws-appfabric" % "7.30.16.1"
"dev.zio" %% "zio-aws-appflow" % "7.30.16.1"
"dev.zio" %% "zio-aws-appintegrations" % "7.30.16.1"
"dev.zio" %% "zio-aws-applicationautoscaling" % "7.30.16.1"
"dev.zio" %% "zio-aws-applicationcostprofiler" % "7.30.16.1"
"dev.zio" %% "zio-aws-applicationdiscovery" % "7.30.16.1"
"dev.zio" %% "zio-aws-applicationinsights" % "7.30.16.1"
"dev.zio" %% "zio-aws-applicationsignals" % "7.30.16.1"
"dev.zio" %% "zio-aws-appmesh" % "7.30.16.1"
"dev.zio" %% "zio-aws-apprunner" % "7.30.16.1"
"dev.zio" %% "zio-aws-appstream" % "7.30.16.1"
"dev.zio" %% "zio-aws-appsync" % "7.30.16.1"
"dev.zio" %% "zio-aws-apptest" % "7.30.16.1"
"dev.zio" %% "zio-aws-arczonalshift" % "7.30.16.1"
"dev.zio" %% "zio-aws-artifact" % "7.30.16.1"
"dev.zio" %% "zio-aws-athena" % "7.30.16.1"
"dev.zio" %% "zio-aws-auditmanager" % "7.30.16.1"
"dev.zio" %% "zio-aws-autoscaling" % "7.30.16.1"
"dev.zio" %% "zio-aws-autoscalingplans" % "7.30.16.1"
"dev.zio" %% "zio-aws-b2bi" % "7.30.16.1"
"dev.zio" %% "zio-aws-backup" % "7.30.16.1"
"dev.zio" %% "zio-aws-backupgateway" % "7.30.16.1"
"dev.zio" %% "zio-aws-backupsearch" % "7.30.16.1"
"dev.zio" %% "zio-aws-batch" % "7.30.16.1"
"dev.zio" %% "zio-aws-bcmdataexports" % "7.30.16.1"
"dev.zio" %% "zio-aws-bcmpricingcalculator" % "7.30.16.1"
"dev.zio" %% "zio-aws-bedrock" % "7.30.16.1"
"dev.zio" %% "zio-aws-bedrockagent" % "7.30.16.1"
"dev.zio" %% "zio-aws-bedrockagentruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-bedrockdataautomation" % "7.30.16.1"
"dev.zio" %% "zio-aws-bedrockdataautomationruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-bedrockruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-billing" % "7.30.16.1"
"dev.zio" %% "zio-aws-billingconductor" % "7.30.16.1"
"dev.zio" %% "zio-aws-braket" % "7.30.16.1"
"dev.zio" %% "zio-aws-budgets" % "7.30.16.1"
"dev.zio" %% "zio-aws-chatbot" % "7.30.16.1"
"dev.zio" %% "zio-aws-chime" % "7.30.16.1"
"dev.zio" %% "zio-aws-chimesdkidentity" % "7.30.16.1"
"dev.zio" %% "zio-aws-chimesdkmediapipelines" % "7.30.16.1"
"dev.zio" %% "zio-aws-chimesdkmeetings" % "7.30.16.1"
"dev.zio" %% "zio-aws-chimesdkmessaging" % "7.30.16.1"
"dev.zio" %% "zio-aws-chimesdkvoice" % "7.30.16.1"
"dev.zio" %% "zio-aws-cleanrooms" % "7.30.16.1"
"dev.zio" %% "zio-aws-cleanroomsml" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloud9" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudcontrol" % "7.30.16.1"
"dev.zio" %% "zio-aws-clouddirectory" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudformation" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudfront" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudfrontkeyvaluestore" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudhsm" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudhsmv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudsearch" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudsearchdomain" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudtrail" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudtraildata" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudwatch" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudwatchevents" % "7.30.16.1"
"dev.zio" %% "zio-aws-cloudwatchlogs" % "7.30.16.1"
"dev.zio" %% "zio-aws-codeartifact" % "7.30.16.1"
"dev.zio" %% "zio-aws-codebuild" % "7.30.16.1"
"dev.zio" %% "zio-aws-codecatalyst" % "7.30.16.1"
"dev.zio" %% "zio-aws-codecommit" % "7.30.16.1"
"dev.zio" %% "zio-aws-codeconnections" % "7.30.16.1"
"dev.zio" %% "zio-aws-codedeploy" % "7.30.16.1"
"dev.zio" %% "zio-aws-codeguruprofiler" % "7.30.16.1"
"dev.zio" %% "zio-aws-codegurureviewer" % "7.30.16.1"
"dev.zio" %% "zio-aws-codegurusecurity" % "7.30.16.1"
"dev.zio" %% "zio-aws-codepipeline" % "7.30.16.1"
"dev.zio" %% "zio-aws-codestarconnections" % "7.30.16.1"
"dev.zio" %% "zio-aws-codestarnotifications" % "7.30.16.1"
"dev.zio" %% "zio-aws-cognitoidentity" % "7.30.16.1"
"dev.zio" %% "zio-aws-cognitoidentityprovider" % "7.30.16.1"
"dev.zio" %% "zio-aws-cognitosync" % "7.30.16.1"
"dev.zio" %% "zio-aws-comprehend" % "7.30.16.1"
"dev.zio" %% "zio-aws-comprehendmedical" % "7.30.16.1"
"dev.zio" %% "zio-aws-computeoptimizer" % "7.30.16.1"
"dev.zio" %% "zio-aws-config" % "7.30.16.1"
"dev.zio" %% "zio-aws-connect" % "7.30.16.1"
"dev.zio" %% "zio-aws-connectcampaigns" % "7.30.16.1"
"dev.zio" %% "zio-aws-connectcampaignsv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-connectcases" % "7.30.16.1"
"dev.zio" %% "zio-aws-connectcontactlens" % "7.30.16.1"
"dev.zio" %% "zio-aws-connectparticipant" % "7.30.16.1"
"dev.zio" %% "zio-aws-controlcatalog" % "7.30.16.1"
"dev.zio" %% "zio-aws-controltower" % "7.30.16.1"
"dev.zio" %% "zio-aws-costandusagereport" % "7.30.16.1"
"dev.zio" %% "zio-aws-costexplorer" % "7.30.16.1"
"dev.zio" %% "zio-aws-costoptimizationhub" % "7.30.16.1"
"dev.zio" %% "zio-aws-customerprofiles" % "7.30.16.1"
"dev.zio" %% "zio-aws-databasemigration" % "7.30.16.1"
"dev.zio" %% "zio-aws-databrew" % "7.30.16.1"
"dev.zio" %% "zio-aws-dataexchange" % "7.30.16.1"
"dev.zio" %% "zio-aws-datapipeline" % "7.30.16.1"
"dev.zio" %% "zio-aws-datasync" % "7.30.16.1"
"dev.zio" %% "zio-aws-datazone" % "7.30.16.1"
"dev.zio" %% "zio-aws-dax" % "7.30.16.1"
"dev.zio" %% "zio-aws-deadline" % "7.30.16.1"
"dev.zio" %% "zio-aws-detective" % "7.30.16.1"
"dev.zio" %% "zio-aws-devicefarm" % "7.30.16.1"
"dev.zio" %% "zio-aws-devopsguru" % "7.30.16.1"
"dev.zio" %% "zio-aws-directconnect" % "7.30.16.1"
"dev.zio" %% "zio-aws-directory" % "7.30.16.1"
"dev.zio" %% "zio-aws-directoryservicedata" % "7.30.16.1"
"dev.zio" %% "zio-aws-dlm" % "7.30.16.1"
"dev.zio" %% "zio-aws-docdb" % "7.30.16.1"
"dev.zio" %% "zio-aws-docdbelastic" % "7.30.16.1"
"dev.zio" %% "zio-aws-drs" % "7.30.16.1"
"dev.zio" %% "zio-aws-dsql" % "7.30.16.1"
"dev.zio" %% "zio-aws-dynamodb" % "7.30.16.1"
"dev.zio" %% "zio-aws-dynamodbstreams" % "7.30.16.1"
"dev.zio" %% "zio-aws-ebs" % "7.30.16.1"
"dev.zio" %% "zio-aws-ec2" % "7.30.16.1"
"dev.zio" %% "zio-aws-ec2instanceconnect" % "7.30.16.1"
"dev.zio" %% "zio-aws-ecr" % "7.30.16.1"
"dev.zio" %% "zio-aws-ecrpublic" % "7.30.16.1"
"dev.zio" %% "zio-aws-ecs" % "7.30.16.1"
"dev.zio" %% "zio-aws-efs" % "7.30.16.1"
"dev.zio" %% "zio-aws-eks" % "7.30.16.1"
"dev.zio" %% "zio-aws-eksauth" % "7.30.16.1"
"dev.zio" %% "zio-aws-elasticache" % "7.30.16.1"
"dev.zio" %% "zio-aws-elasticbeanstalk" % "7.30.16.1"
"dev.zio" %% "zio-aws-elasticinference" % "7.30.16.1"
"dev.zio" %% "zio-aws-elasticloadbalancing" % "7.30.16.1"
"dev.zio" %% "zio-aws-elasticloadbalancingv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-elasticsearch" % "7.30.16.1"
"dev.zio" %% "zio-aws-elastictranscoder" % "7.30.16.1"
"dev.zio" %% "zio-aws-emr" % "7.30.16.1"
"dev.zio" %% "zio-aws-emrcontainers" % "7.30.16.1"
"dev.zio" %% "zio-aws-emrserverless" % "7.30.16.1"
"dev.zio" %% "zio-aws-entityresolution" % "7.30.16.1"
"dev.zio" %% "zio-aws-eventbridge" % "7.30.16.1"
"dev.zio" %% "zio-aws-evidently" % "7.30.16.1"
"dev.zio" %% "zio-aws-finspace" % "7.30.16.1"
"dev.zio" %% "zio-aws-finspacedata" % "7.30.16.1"
"dev.zio" %% "zio-aws-firehose" % "7.30.16.1"
"dev.zio" %% "zio-aws-fis" % "7.30.16.1"
"dev.zio" %% "zio-aws-fms" % "7.30.16.1"
"dev.zio" %% "zio-aws-forecast" % "7.30.16.1"
"dev.zio" %% "zio-aws-forecastquery" % "7.30.16.1"
"dev.zio" %% "zio-aws-frauddetector" % "7.30.16.1"
"dev.zio" %% "zio-aws-freetier" % "7.30.16.1"
"dev.zio" %% "zio-aws-fsx" % "7.30.16.1"
"dev.zio" %% "zio-aws-gamelift" % "7.30.16.1"
"dev.zio" %% "zio-aws-geomaps" % "7.30.16.1"
"dev.zio" %% "zio-aws-geoplaces" % "7.30.16.1"
"dev.zio" %% "zio-aws-georoutes" % "7.30.16.1"
"dev.zio" %% "zio-aws-glacier" % "7.30.16.1"
"dev.zio" %% "zio-aws-globalaccelerator" % "7.30.16.1"
"dev.zio" %% "zio-aws-glue" % "7.30.16.1"
"dev.zio" %% "zio-aws-grafana" % "7.30.16.1"
"dev.zio" %% "zio-aws-greengrass" % "7.30.16.1"
"dev.zio" %% "zio-aws-greengrassv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-groundstation" % "7.30.16.1"
"dev.zio" %% "zio-aws-guardduty" % "7.30.16.1"
"dev.zio" %% "zio-aws-health" % "7.30.16.1"
"dev.zio" %% "zio-aws-healthlake" % "7.30.16.1"
"dev.zio" %% "zio-aws-iam" % "7.30.16.1"
"dev.zio" %% "zio-aws-identitystore" % "7.30.16.1"
"dev.zio" %% "zio-aws-imagebuilder" % "7.30.16.1"
"dev.zio" %% "zio-aws-inspector" % "7.30.16.1"
"dev.zio" %% "zio-aws-inspector2" % "7.30.16.1"
"dev.zio" %% "zio-aws-inspectorscan" % "7.30.16.1"
"dev.zio" %% "zio-aws-internetmonitor" % "7.30.16.1"
"dev.zio" %% "zio-aws-invoicing" % "7.30.16.1"
"dev.zio" %% "zio-aws-iot" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotanalytics" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotdataplane" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotdeviceadvisor" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotevents" % "7.30.16.1"
"dev.zio" %% "zio-aws-ioteventsdata" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotfleethub" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotfleetwise" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotjobsdataplane" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotsecuretunneling" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotsitewise" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotthingsgraph" % "7.30.16.1"
"dev.zio" %% "zio-aws-iottwinmaker" % "7.30.16.1"
"dev.zio" %% "zio-aws-iotwireless" % "7.30.16.1"
"dev.zio" %% "zio-aws-ivs" % "7.30.16.1"
"dev.zio" %% "zio-aws-ivschat" % "7.30.16.1"
"dev.zio" %% "zio-aws-ivsrealtime" % "7.30.16.1"
"dev.zio" %% "zio-aws-kafka" % "7.30.16.1"
"dev.zio" %% "zio-aws-kafkaconnect" % "7.30.16.1"
"dev.zio" %% "zio-aws-kendra" % "7.30.16.1"
"dev.zio" %% "zio-aws-kendraranking" % "7.30.16.1"
"dev.zio" %% "zio-aws-keyspaces" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesis" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisanalytics" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisanalyticsv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisvideo" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisvideoarchivedmedia" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisvideomedia" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisvideosignaling" % "7.30.16.1"
"dev.zio" %% "zio-aws-kinesisvideowebrtcstorage" % "7.30.16.1"
"dev.zio" %% "zio-aws-kms" % "7.30.16.1"
"dev.zio" %% "zio-aws-lakeformation" % "7.30.16.1"
"dev.zio" %% "zio-aws-lambda" % "7.30.16.1"
"dev.zio" %% "zio-aws-launchwizard" % "7.30.16.1"
"dev.zio" %% "zio-aws-lexmodelbuilding" % "7.30.16.1"
"dev.zio" %% "zio-aws-lexmodelsv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-lexruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-lexruntimev2" % "7.30.16.1"
"dev.zio" %% "zio-aws-licensemanager" % "7.30.16.1"
"dev.zio" %% "zio-aws-licensemanagerlinuxsubscriptions" % "7.30.16.1"
"dev.zio" %% "zio-aws-licensemanagerusersubscriptions" % "7.30.16.1"
"dev.zio" %% "zio-aws-lightsail" % "7.30.16.1"
"dev.zio" %% "zio-aws-location" % "7.30.16.1"
"dev.zio" %% "zio-aws-lookoutequipment" % "7.30.16.1"
"dev.zio" %% "zio-aws-lookoutmetrics" % "7.30.16.1"
"dev.zio" %% "zio-aws-lookoutvision" % "7.30.16.1"
"dev.zio" %% "zio-aws-m2" % "7.30.16.1"
"dev.zio" %% "zio-aws-machinelearning" % "7.30.16.1"
"dev.zio" %% "zio-aws-macie2" % "7.30.16.1"
"dev.zio" %% "zio-aws-mailmanager" % "7.30.16.1"
"dev.zio" %% "zio-aws-managedblockchain" % "7.30.16.1"
"dev.zio" %% "zio-aws-managedblockchainquery" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplaceagreement" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplacecatalog" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplacecommerceanalytics" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplacedeployment" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplaceentitlement" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplacemetering" % "7.30.16.1"
"dev.zio" %% "zio-aws-marketplacereporting" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediaconnect" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediaconvert" % "7.30.16.1"
"dev.zio" %% "zio-aws-medialive" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediapackage" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediapackagev2" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediapackagevod" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediastore" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediastoredata" % "7.30.16.1"
"dev.zio" %% "zio-aws-mediatailor" % "7.30.16.1"
"dev.zio" %% "zio-aws-medicalimaging" % "7.30.16.1"
"dev.zio" %% "zio-aws-memorydb" % "7.30.16.1"
"dev.zio" %% "zio-aws-mgn" % "7.30.16.1"
"dev.zio" %% "zio-aws-migrationhub" % "7.30.16.1"
"dev.zio" %% "zio-aws-migrationhubconfig" % "7.30.16.1"
"dev.zio" %% "zio-aws-migrationhuborchestrator" % "7.30.16.1"
"dev.zio" %% "zio-aws-migrationhubrefactorspaces" % "7.30.16.1"
"dev.zio" %% "zio-aws-migrationhubstrategy" % "7.30.16.1"
"dev.zio" %% "zio-aws-mq" % "7.30.16.1"
"dev.zio" %% "zio-aws-mturk" % "7.30.16.1"
"dev.zio" %% "zio-aws-mwaa" % "7.30.16.1"
"dev.zio" %% "zio-aws-neptune" % "7.30.16.1"
"dev.zio" %% "zio-aws-neptunedata" % "7.30.16.1"
"dev.zio" %% "zio-aws-neptunegraph" % "7.30.16.1"
"dev.zio" %% "zio-aws-networkfirewall" % "7.30.16.1"
"dev.zio" %% "zio-aws-networkflowmonitor" % "7.30.16.1"
"dev.zio" %% "zio-aws-networkmanager" % "7.30.16.1"
"dev.zio" %% "zio-aws-networkmonitor" % "7.30.16.1"
"dev.zio" %% "zio-aws-notifications" % "7.30.16.1"
"dev.zio" %% "zio-aws-notificationscontacts" % "7.30.16.1"
"dev.zio" %% "zio-aws-oam" % "7.30.16.1"
"dev.zio" %% "zio-aws-observabilityadmin" % "7.30.16.1"
"dev.zio" %% "zio-aws-omics" % "7.30.16.1"
"dev.zio" %% "zio-aws-opensearch" % "7.30.16.1"
"dev.zio" %% "zio-aws-opensearchserverless" % "7.30.16.1"
"dev.zio" %% "zio-aws-opsworks" % "7.30.16.1"
"dev.zio" %% "zio-aws-opsworkscm" % "7.30.16.1"
"dev.zio" %% "zio-aws-organizations" % "7.30.16.1"
"dev.zio" %% "zio-aws-osis" % "7.30.16.1"
"dev.zio" %% "zio-aws-outposts" % "7.30.16.1"
"dev.zio" %% "zio-aws-panorama" % "7.30.16.1"
"dev.zio" %% "zio-aws-partnercentralselling" % "7.30.16.1"
"dev.zio" %% "zio-aws-paymentcryptography" % "7.30.16.1"
"dev.zio" %% "zio-aws-paymentcryptographydata" % "7.30.16.1"
"dev.zio" %% "zio-aws-pcaconnectorad" % "7.30.16.1"
"dev.zio" %% "zio-aws-pcaconnectorscep" % "7.30.16.1"
"dev.zio" %% "zio-aws-pcs" % "7.30.16.1"
"dev.zio" %% "zio-aws-personalize" % "7.30.16.1"
"dev.zio" %% "zio-aws-personalizeevents" % "7.30.16.1"
"dev.zio" %% "zio-aws-personalizeruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-pi" % "7.30.16.1"
"dev.zio" %% "zio-aws-pinpoint" % "7.30.16.1"
"dev.zio" %% "zio-aws-pinpointemail" % "7.30.16.1"
"dev.zio" %% "zio-aws-pinpointsmsvoice" % "7.30.16.1"
"dev.zio" %% "zio-aws-pinpointsmsvoicev2" % "7.30.16.1"
"dev.zio" %% "zio-aws-pipes" % "7.30.16.1"
"dev.zio" %% "zio-aws-polly" % "7.30.16.1"
"dev.zio" %% "zio-aws-pricing" % "7.30.16.1"
"dev.zio" %% "zio-aws-privatenetworks" % "7.30.16.1"
"dev.zio" %% "zio-aws-proton" % "7.30.16.1"
"dev.zio" %% "zio-aws-qapps" % "7.30.16.1"
"dev.zio" %% "zio-aws-qbusiness" % "7.30.16.1"
"dev.zio" %% "zio-aws-qconnect" % "7.30.16.1"
"dev.zio" %% "zio-aws-qldb" % "7.30.16.1"
"dev.zio" %% "zio-aws-qldbsession" % "7.30.16.1"
"dev.zio" %% "zio-aws-quicksight" % "7.30.16.1"
"dev.zio" %% "zio-aws-ram" % "7.30.16.1"
"dev.zio" %% "zio-aws-rbin" % "7.30.16.1"
"dev.zio" %% "zio-aws-rds" % "7.30.16.1"
"dev.zio" %% "zio-aws-rdsdata" % "7.30.16.1"
"dev.zio" %% "zio-aws-redshift" % "7.30.16.1"
"dev.zio" %% "zio-aws-redshiftdata" % "7.30.16.1"
"dev.zio" %% "zio-aws-redshiftserverless" % "7.30.16.1"
"dev.zio" %% "zio-aws-rekognition" % "7.30.16.1"
"dev.zio" %% "zio-aws-repostspace" % "7.30.16.1"
"dev.zio" %% "zio-aws-resiliencehub" % "7.30.16.1"
"dev.zio" %% "zio-aws-resourceexplorer2" % "7.30.16.1"
"dev.zio" %% "zio-aws-resourcegroups" % "7.30.16.1"
"dev.zio" %% "zio-aws-resourcegroupstaggingapi" % "7.30.16.1"
"dev.zio" %% "zio-aws-robomaker" % "7.30.16.1"
"dev.zio" %% "zio-aws-rolesanywhere" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53domains" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53profiles" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53recoverycluster" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53recoverycontrolconfig" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53recoveryreadiness" % "7.30.16.1"
"dev.zio" %% "zio-aws-route53resolver" % "7.30.16.1"
"dev.zio" %% "zio-aws-rum" % "7.30.16.1"
"dev.zio" %% "zio-aws-s3" % "7.30.16.1"
"dev.zio" %% "zio-aws-s3control" % "7.30.16.1"
"dev.zio" %% "zio-aws-s3outposts" % "7.30.16.1"
"dev.zio" %% "zio-aws-s3tables" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemaker" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemakera2iruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemakeredge" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemakerfeaturestoreruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemakergeospatial" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemakermetrics" % "7.30.16.1"
"dev.zio" %% "zio-aws-sagemakerruntime" % "7.30.16.1"
"dev.zio" %% "zio-aws-savingsplans" % "7.30.16.1"
"dev.zio" %% "zio-aws-scheduler" % "7.30.16.1"
"dev.zio" %% "zio-aws-schemas" % "7.30.16.1"
"dev.zio" %% "zio-aws-secretsmanager" % "7.30.16.1"
"dev.zio" %% "zio-aws-securityhub" % "7.30.16.1"
"dev.zio" %% "zio-aws-securityir" % "7.30.16.1"
"dev.zio" %% "zio-aws-securitylake" % "7.30.16.1"
"dev.zio" %% "zio-aws-serverlessapplicationrepository" % "7.30.16.1"
"dev.zio" %% "zio-aws-servicecatalog" % "7.30.16.1"
"dev.zio" %% "zio-aws-servicecatalogappregistry" % "7.30.16.1"
"dev.zio" %% "zio-aws-servicediscovery" % "7.30.16.1"
"dev.zio" %% "zio-aws-servicequotas" % "7.30.16.1"
"dev.zio" %% "zio-aws-ses" % "7.30.16.1"
"dev.zio" %% "zio-aws-sesv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-sfn" % "7.30.16.1"
"dev.zio" %% "zio-aws-shield" % "7.30.16.1"
"dev.zio" %% "zio-aws-signer" % "7.30.16.1"
"dev.zio" %% "zio-aws-simspaceweaver" % "7.30.16.1"
"dev.zio" %% "zio-aws-sms" % "7.30.16.1"
"dev.zio" %% "zio-aws-snowball" % "7.30.16.1"
"dev.zio" %% "zio-aws-snowdevicemanagement" % "7.30.16.1"
"dev.zio" %% "zio-aws-sns" % "7.30.16.1"
"dev.zio" %% "zio-aws-socialmessaging" % "7.30.16.1"
"dev.zio" %% "zio-aws-sqs" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssm" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssmcontacts" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssmincidents" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssmquicksetup" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssmsap" % "7.30.16.1"
"dev.zio" %% "zio-aws-sso" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssoadmin" % "7.30.16.1"
"dev.zio" %% "zio-aws-ssooidc" % "7.30.16.1"
"dev.zio" %% "zio-aws-storagegateway" % "7.30.16.1"
"dev.zio" %% "zio-aws-sts" % "7.30.16.1"
"dev.zio" %% "zio-aws-supplychain" % "7.30.16.1"
"dev.zio" %% "zio-aws-support" % "7.30.16.1"
"dev.zio" %% "zio-aws-supportapp" % "7.30.16.1"
"dev.zio" %% "zio-aws-swf" % "7.30.16.1"
"dev.zio" %% "zio-aws-synthetics" % "7.30.16.1"
"dev.zio" %% "zio-aws-taxsettings" % "7.30.16.1"
"dev.zio" %% "zio-aws-textract" % "7.30.16.1"
"dev.zio" %% "zio-aws-timestreaminfluxdb" % "7.30.16.1"
"dev.zio" %% "zio-aws-timestreamquery" % "7.30.16.1"
"dev.zio" %% "zio-aws-timestreamwrite" % "7.30.16.1"
"dev.zio" %% "zio-aws-tnb" % "7.30.16.1"
"dev.zio" %% "zio-aws-transcribe" % "7.30.16.1"
"dev.zio" %% "zio-aws-transcribestreaming" % "7.30.16.1"
"dev.zio" %% "zio-aws-transfer" % "7.30.16.1"
"dev.zio" %% "zio-aws-translate" % "7.30.16.1"
"dev.zio" %% "zio-aws-trustedadvisor" % "7.30.16.1"
"dev.zio" %% "zio-aws-verifiedpermissions" % "7.30.16.1"
"dev.zio" %% "zio-aws-voiceid" % "7.30.16.1"
"dev.zio" %% "zio-aws-vpclattice" % "7.30.16.1"
"dev.zio" %% "zio-aws-waf" % "7.30.16.1"
"dev.zio" %% "zio-aws-wafregional" % "7.30.16.1"
"dev.zio" %% "zio-aws-wafv2" % "7.30.16.1"
"dev.zio" %% "zio-aws-wellarchitected" % "7.30.16.1"
"dev.zio" %% "zio-aws-wisdom" % "7.30.16.1"
"dev.zio" %% "zio-aws-workdocs" % "7.30.16.1"
"dev.zio" %% "zio-aws-workmail" % "7.30.16.1"
"dev.zio" %% "zio-aws-workmailmessageflow" % "7.30.16.1"
"dev.zio" %% "zio-aws-workspaces" % "7.30.16.1"
"dev.zio" %% "zio-aws-workspacesthinclient" % "7.30.16.1"
"dev.zio" %% "zio-aws-workspacesweb" % "7.30.16.1"
"dev.zio" %% "zio-aws-xray" % "7.30.16.1"
```

---

## Aspects


It is possible to define _aspects_ of type `AwsCallAspect[R]` that can modify the behavior of the AWS client modules. This can be used for example 
to add logging or metrics to the AWS clients and it's also the recommended way to handle retries or apply rate limiting and other similar patterns.

The library contains two default aspects, implementing logging and metrics on top of ZIO 2's core capabilities:

```scala
val callLogging: AwsCallAspect[Any]
def callDuration(prefix: String, boundaries: MetricKeyType.Histogram.Boundaries): AwsCallAspect[Any]
```

To define an aspect, create an instance of the `AwsCallAspect` trait:

```scala
val callLogging: AwsCallAspect[Any] =
    new AwsCallAspect[Any] {
      override final def apply[R, E >: AwsError, A <: Described[_]](
          f: ZIO[R, E, A]
      )(implicit trace: Trace): ZIO[R, E, A] = {
        f.either.timed
          .flatMap {
            case (duration, Right(r)) =>
              ZIO.succeed(r)
            case (duration, Left(error)) =>
              Console
                .printLine(
                  s"AWS call FAILED in $duration with $error"
                )
                .ignore *> ZIO.fail(error)
          }
      }
    }
// callLogging: AwsCallAspect[Any] = repl.MdocSession$MdocApp$$anon$1@57c4462f
```

This aspect can attached to a _client layer_ with the `@@@` operator. Multiple aspects can be composed with `>>>`.

To see a full example, check [example #2](https://github.com/vigoo/zio-aws/blob/master/examples/example2/src/main/scala/Main.scala).

---

## Changelog

# Version history

Note: this is a manually maintained list of important changes. Because of having auto-release from CI, this
list may not reflect _all_ changes immediately. Dependencies are continuously updated and are not documented here.

#### 7.x.x.x
- ZIO 2.0.18 introduced a `ZLayerAspect` which uses the same operator (`@@`) that ZIO AWS was using on layers, but it is not powerful enough to express this library's aspects. The new operator conflicts with the existing extension method, so starting from this version it has been renamed from `@@` to `@@@` to avoid the conflict. 

#### 6.x.x.x
- Using the new ZIO config providers, compatible with ZIO Config 4.x.x

#### 5.x.x.x
- Migrated to `ZIO 2.0.0` 
- Moved to the ZIO organisation
- Using the new ZIO service pattern
- Changed getter naming to match `zio-k8s`
- Using `zio-prelude` _newtype wrappers_ for primitives
- Using `ZIO` aspects as a base of `AwsCallAspect`
- Exposed lower level pagination API calls beside the streaming ones
- Using the `Optional` type instead of `Option` to reduce boilerplate

See the [migration guide](overview_migration_guide) for more information.

#### 4.17.104.1
- Migrated to `zio-config v2.0.0`

#### 3.15.35.5
- Support for defining dual HTTP/1.1 and HTTP/2 `HttpClient` layers
- Convert model case classes to `.ReadOnly` trait with `.asReadOnly`

#### 3.15.19.10
- Fix for libraries that use the `Integer` alias for ints (such as `zio-aws-sqs`)

#### 3.15.19.8
- Updated [fs2](https://fs2.io) release that fixes the http4s backend

#### 3.15.19.7
- Fix for event streaming wrappers such as `subscribeToShard` in `zio-aws-kinesis`

#### 3.15.16.0

- Introduced automatic release from CI, each new AWS SDK release triggers a new `zio-aws` build now
- zio-config support 

#### 2.14.7.0

- Updated to AWS SDK 2.14.7
- Fix an [issue](https://github.com/vigoo/zio-aws/issues/23) with http4s streaming uploads
- `Iterable` in place of `List` in the request models
- The akka-http client now gets the _actor system_ from the environment
- Code generator rewritten as an sbt plugin

#### 2.14.3.0
API breaking changes to make the streaming interface more ergonomic:
- Input/output byte streams are now flat (`ZStream[Any, AwsError, Byte]` instead of `ZStream[Any, AwsError, Chunk[Byte]`)
- Streaming operations return a `ZStream` that performs the request on first pull instead of a `ZIO[..., ZStream[...]]`
- Streaming for paginated operations that does not have a paginator in the Java SDK
- No `xxxStream` variants, streaming is the default and only interface for paginatable operations
- Updated to AWS SDK 2.14.3
- Fixed handling of some error cases
- Scala 2.12 version is now available

#### 1.13.69.1
Initial release republished with fixed metadata in POMs

#### 1.13.69.0
Initial release based on AWS Java SDK 2.13.69

---

## Configuration


## Common configuration

Each _service module_ depends on the `AwsConfig` layer. This layer is responsible for setting up the 
AWS Java SDK's async client, by setting the [underlying HTTP engine](http.md) and all the common
settings. You can use the following layers to provide `AwsConfig`:

#### Default
`AwsConfig.default` requires a `HttpClient` as dependency, but does not customize any other setting of the client

#### Fully customized
`AwsConfig.customized(customization)` gives the freedom to customize the creation of the AWS async client directly by modifying it's `Builder`

#### Configured
`AwsConfig.configured()` is the *recommended* way to construct an `AwsConfig`. Beside requiring a `HttpClient` it also has `ZConfig[CommonAwsConfig]` as dependency.
The `CommonAwsConfig` value can be either provided from code for example by `ZLayer.succeed(CommonAwsConfig(...))` or it can
be read from any of the supported config sources by [zio-config](https://zio.dev/zio-config/).

Note that **AWS level retries are disabled** by the configuration layer and it is not exposed in the `CommonAwsConfig` data structure either. The reason for this is that the recommended way to handle retries is to use [aspects on the service layers](aspects.md).
 
See the following table about the possible configuration values. Please note that the underlying HTTP engine also has its own
specific configuration which is described [on the page about the HTTP engines](http.md). 

## Configuration Details

|FieldName|Format                     |Description|Sources|
|---      |---                        |---        |---    |
|         |[all-of](fielddescriptions)|           |       |

### Field Descriptions

|FieldName       |Format                           |Description                                        |Sources|
|---             |---                              |---                                                |---    |
|region          |primitive                        |a text property, AWS region to connect to          |       |
|                |[any-one-of](fielddescriptions-1)|                                                   |       |
|endpointOverride|primitive                        |a text property, Overrides the AWS service endpoint|       |
|[client](client)|[all-of](client)                 |Common settings for AWS service clients            |       |

### Field Descriptions

|FieldName|Format   |Description                                  |Sources|
|---      |---      |---                                          |---    |
|type     |map      |a text property, AWS credentials provider    |       |
|         |primitive|a constant property, AWS credentials provider|       |

### client

|FieldName                   |Format              |Description                                                                                     |Sources|
|---                         |---                 |---                                                                                             |---    |
|[extraHeaders](extraheaders)|[list](extraheaders)|Extra headers to be sent with each request                                                      |       |
|apiCallTimeout              |primitive           |a duration property, Amount of time to allow the client to complete the execution of an API call|       |
|apiCallAttemptTimeout       |primitive           |a duration property, Amount of time to wait for the HTTP request to complete before giving up   |       |
|defaultProfileName          |primitive           |a text property, Default profile name                                                           |       |

### extraHeaders

|FieldName|Format   |Description                  |Sources|
|---      |---      |---                          |---    |
|name     |primitive|a text property, Header name |       |
|value    |list     |a text property, Header value|       |

## Service layer
Each AWS service's generated client has it own layer that depends on `AwsConfig`. It is possible to reuse the same `AwsConfig` layer
for multiple AWS service clients, sharing a common configuration. Usually the service client does not require any additional configuration,
in this case the `live` layer can be used, for example:

```scala
program.provide(
    awsConfig,
    Ec2.live,
    ElasticBeanstalk.live
)
```

---

## Examples(Zio-aws)


The following example uses the ElasticBeanstalk and EC2 APIs to print some info. 

```scala
object Main extends ZIOAppDefault {
  val program: ZIO[Ec2 & ElasticBeanstalk, AwsError, Unit] =
    for {
      appsResult <- ElasticBeanstalk.describeApplications(
        DescribeApplicationsRequest(applicationNames = Some(List(ApplicationName("my-service"))))
      )
      app <- appsResult.getApplications.map(_.headOption)
      _ <- app match {
        case Some(appDescription) =>
          for {
            applicationName <- appDescription.getApplicationName
            _ <- Console
              .printLine(
                s"Got application description for $applicationName"
              )
              .ignore

            envStream = ElasticBeanstalk.describeEnvironments(
              DescribeEnvironmentsRequest(applicationName =
                Some(applicationName)
              )
            )

            _ <- envStream.run(ZSink.foreach { env =>
              env.getEnvironmentName.flatMap { environmentName =>
                (for {
                  environmentId <- env.getEnvironmentId
                  _ <- Console
                    .printLine(
                      s"Getting the EB resources of $environmentName"
                    )
                    .ignore

                  resourcesResult <-
                    ElasticBeanstalk.describeEnvironmentResources(
                      DescribeEnvironmentResourcesRequest(environmentId =
                        Some(environmentId)
                      )
                    )
                  resources <- resourcesResult.getEnvironmentResources
                  _ <- Console
                    .printLine(
                      s"Getting the EC2 instances in $environmentName"
                    )
                    .ignore
                  instances <- resources.getInstances
                  instanceIds <- ZIO.foreach(instances)(_.getId)
                  _ <- Console
                    .printLine(
                      s"Instance IDs are ${instanceIds.mkString(", ")}"
                    )
                    .ignore

                  reservationsStream = Ec2.describeInstances(
                    DescribeInstancesRequest(instanceIds = Some(instanceIds.map(id => zio.aws.ec2.model.primitives.InstanceId(ResourceId.unwrap(id)))))
                  )
                  _ <- reservationsStream.run(ZSink.foreach { reservation =>
                    reservation.getInstances
                      .flatMap { instances =>
                        ZIO.foreach(instances) { instance =>
                          for {
                            id <- instance.getInstanceId
                            typ <- instance.getInstanceType
                            launchTime <- instance.getLaunchTime
                            _ <- Console.printLine(s"  instance $id:").ignore
                            _ <- Console.printLine(s"    type: $typ").ignore
                            _ <- Console
                              .printLine(
                                s"    launched at: $launchTime"
                              )
                              .ignore
                          } yield ()
                        }
                      }
                  })
                } yield ()).catchAll { error =>
                  Console
                    .printLineError(
                      s"Failed to get info for $environmentName: $error"
                    )
                    .ignore
                }
              }
            })
          } yield ()
        case None =>
          ZIO.unit
      }
    } yield ()

  override def run: URIO[ZIOAppArgs with zio.Scope, ExitCode] = {
    val httpClient = NettyHttpClient.default
    val awsConfig = httpClient >>> AwsConfig.default
    val aws = awsConfig >>> (Ec2.live ++ ElasticBeanstalk.live)

    program
      .provideLayer(aws)
      .either
      .flatMap {
        case Left(error) =>
          Console.printLineError(s"AWS error: $error").ignore.as(ExitCode.failure)
        case Right(_) =>
          ZIO.unit.as(ExitCode.success)
      }
  }
}
```

---

## Summary(Zio-aws)

# Getting started

Low-level AWS wrapper for [ZIO](https://zio.dev) for _all_ AWS services using the AWS Java SDK v2.

The library's goal is to have access to all AWS functionality for cases when only a simple, direct access is needed from a ZIO application, and to be used as a building block for higher level wrappers around specific services.

Check the [list of available artifacts](artifacts.md) to get started. 

The [wrapper page](wrappers.md) shows in details how the library wraps the underlying _Java SDK_. On the [configuration page](configuration.md) you
can learn more about how set the common properties of the AWS clients in addition to setting up one of the [HTTP implementations](http.md).

### Features
- Common [configuration](configuration.md) layer
- ZIO layer per AWS service
- [Wrapper](wrappers.md) for all operations on all services
- [Http service implementations](http.md) for functional Scala http libraries, injected through ZIO's module system
- ZStream wrapper around paginated and streaming operations
- Service-specific extra configuration
- More idiomatic Scala request and response types wrapping the Java classes
- [Aspects](aspects.md) to take care of additional concerns like logging, metrics, circuit breaking, etc.

### Design
The library consists of a core module and one generated library for _each_ AWS service, based on the official JSON
schema from the AWS Java SDK's repository. By only providing a wrapper on top of the Java SDK the code
generator does not have to know all the implementation details and features of the schema. 

### Higher level AWS libraries
The following libraries are built on top of `zio-aws` providing higher level interfaces for specific AWS services:

- [ZIO DynamoDB](https://github.com/zio/zio-dynamodb)
- [ZIO Kinesis](https://github.com/svroonland/zio-kinesis)
- [ZIO SQS](https://github.com/zio/zio-sqs)

### Additional resources

- There is a [blog post](https://vigoo.github.io/posts/2020-09-23-zioaws-code-generation.html) explaining how the code generator is implemented.
- [This post](https://vigoo.github.io/posts/2020-11-01-zioaws-zioquery.html) shows an example of using `zio-aws` together with [ZIO Query](https://zio.github.io/zio-query/) 
- [Talk about generating libraries](https://www.youtube.com/watch?v=HCPTmytex3U) from Functional Scala 2021

---

## HTTP

# HTTP implementations

By default the AWS Java SDK uses _netty_ under the hood to make the HTTP client calls. `zio-aws` defines the http client
as a _layer_ (`HttpClient`) that has to be provided to the _AWS configuration layer_.

Currently the following implementations can be used:
- `zio-aws-netty` contains the default netty implementation packed as a layer
- `zio-aws-akka-http` is based on Matthias Lüneberg's [aws-spi-akka-http library](https://github.com/matsluni/aws-spi-akka-http)
- `zio-aws-http4s` is an implementation on top of _http4s_
- `zio-aws-crt-http` is an implementation on top of the _AWS Common Runtime_ (CRT) HTTP client

## Netty
The default HTTP implementation in the AWS Java SDK is _Netty_. To use it with the default settings, use the `netty.default`
layer to provide the `HttpClient` for the `AwsConfig` layer. It is also possible to customize the `NettyNioAsyncHttpClient`
directly by manipulation it's `Builder`, by using the `netty.customized(customization)` layer.

The recommended way for configuration is to use the zio-config support:

```scala
def configured(
      tlsKeyManagersProvider: Option[TlsKeyManagersProvider] = None,
      tlsTrustManagersProvider: Option[TlsTrustManagersProvider] = None
  ): ZLayer[ZConfig[NettyClientConfig], Throwable, HttpClient]
```

Everything except the TLS key and trust managers are described by the zio-config provided `NettyClientConfig` data structure.
See the following table for all the options:

## Configuration Details

|FieldName|Format                     |Description|Sources|
|---      |---                        |---        |---    |
|         |[all-of](fielddescriptions)|           |       |

### Field Descriptions

|FieldName     |Format                            |Description                                 |Sources|
|---           |---                               |---                                         |---    |
|              |[any-one-of](fielddescriptions-11)|                                            |       |
|              |[any-one-of](fielddescriptions-10)|                                            |       |
|              |[any-one-of](fielddescriptions-9) |                                            |       |
|              |[any-one-of](fielddescriptions-8) |                                            |       |
|              |[any-one-of](fielddescriptions-7) |                                            |       |
|              |[any-one-of](fielddescriptions-6) |                                            |       |
|              |[any-one-of](fielddescriptions-5) |                                            |       |
|              |[any-one-of](fielddescriptions-4) |                                            |       |
|              |[any-one-of](fielddescriptions-3) |                                            |       |
|              |[any-one-of](fielddescriptions-2) |                                            |       |
|              |[any-one-of](fielddescriptions-1) |                                            |       |
|sslProvider   |primitive                         |a text property, The SSL provider to be used|       |
|[proxy](proxy)|[all-of](proxy)                   |Proxy configuration                         |       |
|[http2](http2)|[all-of](http2)                   |HTTP/2 specific options                     |       |

### Field Descriptions

|FieldName     |Format   |Description                                                       |Sources|
|---           |---      |---                                                               |---    |
|maxConcurrency|primitive|an integer property, Maximum number of allowed concurrent requests|       |
|              |primitive|a constant property, Maximum number of allowed concurrent requests|       |

### Field Descriptions

|FieldName                   |Format   |Description                                                        |Sources|
|---                         |---      |---                                                                |---    |
|maxPendingConnectionAcquires|primitive|an integer property, The maximum number of pending acquires allowed|       |
|                            |primitive|a constant property, The maximum number of pending acquires allowed|       |

### Field Descriptions

|FieldName  |Format   |Description                                                           |Sources|
|---        |---      |---                                                                   |---    |
|readTimeout|primitive|a duration property, The amount of time to wait for a read on a socket|       |
|           |primitive|a constant property, The amount of time to wait for a read on a socket|       |

### Field Descriptions

|FieldName   |Format   |Description                                                            |Sources|
|---         |---      |---                                                                    |---    |
|writeTimeout|primitive|a duration property, The amount of time to wait for a write on a socket|       |
|            |primitive|a constant property, The amount of time to wait for a write on a socket|       |

### Field Descriptions

|FieldName        |Format   |Description                                                                                              |Sources|
|---              |---      |---                                                                                                      |---    |
|connectionTimeout|primitive|a duration property, The amount of time to wait when initially establishing a connection before giving up|       |
|                 |primitive|a constant property, The amount of time to wait when initially establishing a connection before giving up|       |

### Field Descriptions

|FieldName                   |Format   |Description                                                                                               |Sources|
|---                         |---      |---                                                                                                       |---    |
|connectionAcquisitionTimeout|primitive|a duration property, The amount of time to wait when acquiring a connection from the pool before giving up|       |
|                            |primitive|a constant property, The amount of time to wait when acquiring a connection from the pool before giving up|       |

### Field Descriptions

|FieldName           |Format   |Description                                                                                                                      |Sources|
|---                 |---      |---                                                                                                                              |---    |
|connectionTimeToLive|primitive|a duration property, The maximum amount of time that a connection should be allowed to remain open, regardless of usage frequency|       |
|                    |primitive|a constant property, The maximum amount of time that a connection should be allowed to remain open, regardless of usage frequency|       |

### Field Descriptions

|FieldName            |Format   |Description                                                                                              |Sources|
|---                  |---      |---                                                                                                      |---    |
|connectionMaxIdleTime|primitive|a duration property, Maximum amount of time that a connection should be allowed to remain open while idle|       |
|                     |primitive|a constant property, Maximum amount of time that a connection should be allowed to remain open while idle|       |

### Field Descriptions

|FieldName              |Format   |Description                                                                    |Sources|
|---                    |---      |---                                                                            |---    |
|useIdleConnectionReaper|primitive|a boolean property, If true, the idle connections in the pool should be closed |       |
|                       |primitive|a constant property, If true, the idle connections in the pool should be closed|       |

### Field Descriptions

|FieldName|Format   |Description                                    |Sources|
|---      |---      |---                                            |---    |
|protocol |primitive|a text property, HTTP/1.1 or HTTP/2 or Dual    |       |
|         |primitive|a constant property, HTTP/1.1 or HTTP/2 or Dual|       |

### Field Descriptions

|FieldName                       |Format                  |Description                                      |Sources|
|---                             |---                     |---                                              |---    |
|[channelOptions](channeloptions)|[all-of](channeloptions)|Custom Netty channel options                     |       |
|                                |primitive               |a constant property, Custom Netty channel options|       |

### channelOptions

|FieldName                     |Format   |Description                                                             |Sources|
|---                           |---      |---                                                                     |---    |
|SO_BROADCAST                  |primitive|a boolean property, Allow transmission of broadcast datagrams           |       |
|SO_KEEPALIVE                  |primitive|a boolean property, Keep connection alive                               |       |
|SO_SNDBUF                     |primitive|an integer property, The size of the socket send buffer                 |       |
|SO_RCVBUF                     |primitive|an integer property, The size of the socket receive buffer              |       |
|SO_REUSEADDR                  |primitive|a boolean property, Re-use address                                      |       |
|SO_LINGER                     |primitive|an integer property, Linger on close if data is present                 |       |
|IP_TOS                        |primitive|an integer property, The ToS octet in the IP header                     |       |
|IP_MULTICAST_IF               |primitive|a text property, The network interface's name for IP multicast datagrams|       |
|IP_MULTICAST_TTL              |primitive|an integer property, The time-to-live for IP multicast datagrams        |       |
|IP_MULTICAST_LOOP             |primitive|a boolean property, Loopback for IP multicast datagrams                 |       |
|TCP_NODELAY                   |primitive|a boolean property, Disable the Nagle algorithm                         |       |
|CONNECT_TIMEOUT_MILLIS        |primitive|a duration property, Connect timeout                                    |       |
|WRITE_SPIN_COUNT              |primitive|an integer property, Write spin count                                   |       |
|ALLOW_HALF_CLOSURE            |primitive|a boolean property, Allow half closure                                  |       |
|AUTO_READ                     |primitive|a boolean property, Auto read                                           |       |
|AUTO_CLOSE                    |primitive|a boolean property, Auto close                                          |       |
|SINGLE_EVENTEXECUTOR_PER_GROUP|primitive|a boolean property, Single event executor per group                     |       |

### proxy

|FieldName|Format                            |Description                           |Sources|
|---      |---                               |---                                   |---    |
|         |[any-one-of](fielddescriptions-13)|                                      |       |
|host     |primitive                         |a text property, Hostname of the proxy|       |
|port     |primitive                         |an integer property, Port of the proxy|       |
|         |[any-one-of](fielddescriptions-12)|                                      |       |

### Field Descriptions

|FieldName|Format   |Description                          |Sources|
|---      |---      |---                                  |---    |
|scheme   |primitive|a text property, The proxy scheme    |       |
|         |primitive|a constant property, The proxy scheme|       |

### Field Descriptions

|FieldName    |Format   |Description                                          |Sources|
|---          |---      |---                                                  |---    |
|nonProxyHosts|list     |a text property, Hosts that should not be proxied    |       |
|             |primitive|a constant property, Hosts that should not be proxied|       |

### http2

|FieldName        |Format                            |Description                                                         |Sources|
|---              |---                               |---                                                                 |---    |
|maxStreams       |primitive                         |an integer property, Max number of concurrent streams per connection|       |
|initialWindowSize|primitive                         |an integer property, Initial window size of a stream                |       |
|                 |[any-one-of](fielddescriptions-12)|                                                                    |       |

### Field Descriptions

|FieldName            |Format   |Description                                                                                       |Sources|
|---                  |---      |---                                                                                               |---    |
|healthCheckPingPeriod|primitive|a duration property, The period that the Netty client will send PING frames to the remote endpoint|       |
|                     |primitive|a constant property, The period that the Netty client will send PING frames to the remote endpoint|       |

## Akka HTTP
The _Akka HTTP implementation_ can be chosen by using the `akkahttp.client()` layer for providing `HttpClient` to `AwsConfig`.
This implementation uses the [standard akka-http settings](https://doc.akka.io/docs/akka-http/current/configuration.html) from the application's _Lightbend config_,
it is not described with zio-config descriptors.

## http4s
Another alternative is the _http4s client_. To use the default settings, provide the `http4s.default` layer to `AwsConfig`. Customization by manipulating the builder
is also possible by `http4s.customized(customization)`. And similarly to the _Netty_ client, configuration is also possible via zio-config:

## Configuration Details

|FieldName|Format                     |Description|Sources|
|---      |---                        |---        |---    |
|         |[all-of](fielddescriptions)|           |       |

### Field Descriptions

|FieldName|Format                            |Description|Sources|
|---      |---                               |---        |---    |
|         |[any-one-of](fielddescriptions-15)|           |       |
|         |[any-one-of](fielddescriptions-14)|           |       |
|         |[any-one-of](fielddescriptions-13)|           |       |
|         |[any-one-of](fielddescriptions-12)|           |       |
|         |[any-one-of](fielddescriptions-11)|           |       |
|         |[any-one-of](fielddescriptions-10)|           |       |
|         |[any-one-of](fielddescriptions-9) |           |       |
|         |[any-one-of](fielddescriptions-8) |           |       |
|         |[any-one-of](fielddescriptions-7) |           |       |
|         |[any-one-of](fielddescriptions-6) |           |       |
|         |[any-one-of](fielddescriptions-5) |           |       |
|         |[any-one-of](fielddescriptions-4) |           |       |
|         |[any-one-of](fielddescriptions-3) |           |       |
|         |[any-one-of](fielddescriptions-2) |           |       |
|         |[any-one-of](fielddescriptions-1) |           |       |

### Field Descriptions

|FieldName            |Format   |Description                                                               |Sources|
|---                  |---      |---                                                                       |---    |
|responseHeaderTimeout|primitive|a duration property, Timeout for receiving the header part of the response|       |
|                     |primitive|a constant property, Timeout for receiving the header part of the response|       |

### Field Descriptions

|FieldName  |Format   |Description                                                    |Sources|
|---        |---      |---                                                            |---    |
|idleTimeout|primitive|a duration property, Timeout for client connection staying idle|       |
|           |primitive|a constant property, Timeout for client connection staying idle|       |

### Field Descriptions

|FieldName     |Format   |Description                                       |Sources|
|---           |---      |---                                               |---    |
|requestTimeout|primitive|a duration property, Timeout for the whole request|       |
|              |primitive|a constant property, Timeout for the whole request|       |

### Field Descriptions

|FieldName     |Format   |Description                                              |Sources|
|---           |---      |---                                                      |---    |
|connectTimeout|primitive|a duration property, Timeout for connecting to the server|       |
|              |primitive|a constant property, Timeout for connecting to the server|       |

### Field Descriptions

|FieldName|Format   |Description                                              |Sources|
|---      |---      |---                                                      |---    |
|userAgent|primitive|a text property, User-Agent header sent by the client    |       |
|         |primitive|a constant property, User-Agent header sent by the client|       |

### Field Descriptions

|FieldName          |Format   |Description                                                |Sources|
|---                |---      |---                                                        |---    |
|maxTotalConnections|primitive|an integer property, Maximum number of parallel connections|       |
|                   |primitive|a constant property, Maximum number of parallel connections|       |

### Field Descriptions

|FieldName        |Format   |Description                                             |Sources|
|---              |---      |---                                                     |---    |
|maxWaitQueueLimit|primitive|an integer property, Maximum number of requests in queue|       |
|                 |primitive|a constant property, Maximum number of requests in queue|       |

### Field Descriptions

|FieldName                  |Format   |Description                              |Sources|
|---                        |---      |---                                      |---    |
|checkEndpointIdentification|primitive|a boolean property, Check https identity |       |
|                           |primitive|a constant property, Check https identity|       |

### Field Descriptions

|FieldName          |Format   |Description                                                    |Sources|
|---                |---      |---                                                            |---    |
|maxResponseLineSize|primitive|an integer property, Maximum line length of headers in response|       |
|                   |primitive|a constant property, Maximum line length of headers in response|       |

### Field Descriptions

|FieldName      |Format   |Description                                                      |Sources|
|---            |---      |---                                                              |---    |
|maxHeaderLength|primitive|an integer property, Maximum total length of the response headers|       |
|               |primitive|a constant property, Maximum total length of the response headers|       |

### Field Descriptions

|FieldName   |Format   |Description                            |Sources|
|---         |---      |---                                    |---    |
|maxChunkSize|primitive|an integer property, Maximum chunk size|       |
|            |primitive|a constant property, Maximum chunk size|       |

### Field Descriptions

|FieldName         |Format   |Description                                          |Sources|
|---               |---      |---                                                  |---    |
|chunkBufferMaxSize|primitive|an integer property, Maximum size of the chunk buffer|       |
|                  |primitive|a constant property, Maximum size of the chunk buffer|       |

### Field Descriptions

|FieldName |Format   |Description                                        |Sources|
|---       |---      |---                                                |---    |
|parserMode|primitive|a text property, Parser mode, strict or lenient    |       |
|          |primitive|a constant property, Parser mode, strict or lenient|       |

### Field Descriptions

|FieldName |Format   |Description                     |Sources|
|---       |---      |---                             |---    |
|bufferSize|primitive|an integer property, Buffer size|       |
|          |primitive|a constant property, Buffer size|       |

### Field Descriptions

|FieldName                       |Format                  |Description                                      |Sources|
|---                             |---                     |---                                              |---    |
|[channelOptions](channeloptions)|[all-of](channeloptions)|Collection of socket options                     |       |
|                                |primitive               |a constant property, Collection of socket options|       |

### channelOptions

|FieldName        |Format   |Description                                                             |Sources|
|---              |---      |---                                                                     |---    |
|SO_BROADCAST     |primitive|a boolean property, Allow transmission of broadcast datagrams           |       |
|SO_KEEPALIVE     |primitive|a boolean property, Keep connection alive                               |       |
|SO_SNDBUF        |primitive|an integer property, The size of the socket send buffer                 |       |
|SO_RCVBUF        |primitive|an integer property, The size of the socket receive buffer              |       |
|SO_REUSEADDR     |primitive|a boolean property, Re-use address                                      |       |
|SO_LINGER        |primitive|an integer property, Linger on close if data is present                 |       |
|IP_TOS           |primitive|an integer property, The ToS octet in the IP header                     |       |
|IP_MULTICAST_IF  |primitive|a text property, The network interface's name for IP multicast datagrams|       |
|IP_MULTICAST_TTL |primitive|an integer property, The time-to-live for IP multicast datagrams        |       |
|IP_MULTICAST_LOOP|primitive|a boolean property, Loopback for IP multicast datagrams                 |       |
|TCP_NODELAY      |primitive|a boolean property, Disable the Nagle algorithm                         |       |

## AWS CRT HTTP
The new _AWS Common Runtime_ (CRT) HTTP client can be used by providing the `crt.default` layer for providing `HttpClient` to `AwsConfig`.

## Configuration Details

|FieldName|Format                     |Description|Sources|
|---      |---                        |---        |---    |
|         |[all-of](fielddescriptions)|           |       |

### Field Descriptions

|FieldName                           |Format                    |Description                                                                                                                                                                                                                                                                                                                                  |Sources|
|---                                 |---                       |---                                                                                                                                                                                                                                                                                                                                          |---    |
|maxConcurrency                      |primitive                 |an integer property, The Maximum number of allowed concurrent requests. For HTTP/1.1 this is the same as max connections.                                                                                                                                                                                                                    |       |
|readBufferSizeInBytes               |primitive                 |an integer property, Configures the number of unread bytes that can be buffered in the client before we stop reading from the underlying TCP socket and wait for the Subscriber to read more data.                                                                                                                                           |       |
|[proxy](proxy)                      |[all-of](proxy)           |Sets the http proxy configuration to use for this client.                                                                                                                                                                                                                                                                                    |       |
|[connectionHealth](connectionhealth)|[all-of](connectionhealth)|Configure the health checks for all connections established by this client.                                                                                                                                                                                                                                                                  |       |
|connectionMaxIdleTime               |primitive                 |a duration property, Configure the maximum amount of time that a connection should be allowed to remain open while idle.                                                                                                                                                                                                                     |       |
|connectionTimeout                   |primitive                 |a duration property, The amount of time to wait when initially establishing a connection before giving up and timing out.                                                                                                                                                                                                                    |       |
|[tcpKeepAlive](tcpkeepalive)        |[all-of](tcpkeepalive)    |Configure whether to enable tcpKeepAlive and relevant configuration for all connections established by this client.                                                                                                                                                                                                                          |       |
|postQuantumTlsEnabled               |primitive                 |a boolean property, Configure whether to enable a hybrid post-quantum key exchange option for the Transport Layer Security (TLS) network encryption protocol when communicating with services that support Post Quantum TLS. If Post Quantum cipher suites are not supported on the platform, the SDK will use the default TLS cipher suites.|       |

### proxy

|FieldName|Format   |Description                                |Sources|
|---      |---      |---                                        |---    |
|scheme   |primitive|a text property, The scheme of the proxy   |       |
|host     |primitive|a text property, The host of the proxy     |       |
|port     |primitive|an integer property, The port of the proxy |       |
|username |primitive|a text property, The username for the proxy|       |
|password |primitive|a text property, The password for the proxy|       |

### connectionHealth

|FieldName               |Format   |Description                                                                                                               |Sources|
|---                     |---      |---                                                                                                                       |---    |
|minimumThroughputInBps  |primitive|an integer property, Sets a throughput threshold for connections. Throughput below this value will be considered unhealthy|       |
|minimumThroughputTimeout|primitive|a duration property, Sets how long a connection is allowed to be unhealthy before getting shut down                       |       |

### tcpKeepAlive

|FieldName        |Format   |Description                                                                                                            |Sources|
|---              |---      |---                                                                                                                    |---    |
|keepAliveInterval|primitive|a duration property, The number of seconds between TCP keepalive packets being sent to the peer                        |       |
|keepAliveTimeout |primitive|a duration property, The number of seconds to wait for a keepalive response before considering the connection timed out|       |

---

## Overview

[ZIO AWS](https://zio.dev/zio-aws) is AWS wrapper for [ZIO](https://zio.dev) for _all_ AWS services using the AWS Java SDK v2.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-aws/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-aws-core_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-aws-core_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-aws-core_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-aws-core_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-aws-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-aws-docs_2.13) [![ZIO AWS](https://img.shields.io/github/stars/zio/zio-aws?style=social)](https://github.com/zio/zio-aws)

## Introduction

The goal is to have access to all AWS functionality for cases when only a simple, direct access is needed from a ZIO application, or to be used as a building block for higher level wrappers around specific services.

Key features of ZIO AWS:

- Common configuration layer
- ZIO module layer per AWS service
- Wrapper for all operations on all services
- HTTP service implementations for functional Scala HTTP libraries, injected through ZIO’s module system
- ZStream wrapper around paginated operations
- Service-specific extra configuration
- More idiomatic Scala request and response types wrapping the Java classes

## Installation

There are tones of artifacts [published](artifacts.md) for each AWS service. We can pick whichever services we need.

## Example

The following example uses the _ElasticBeanstalk_ and _EC2_ APIs:

> **Note**
> 
> See the [artifacts](artifacts.md) page for the latest release.

```scala
libraryDependencies += "dev.zio" %% "zio-aws-core"             % "<version>"
libraryDependencies += "dev.zio" %% "zio-aws-ec2"              % "<version>"
libraryDependencies += "dev.zio" %% "zio-aws-elasticbeanstalk" % "<version>"
libraryDependencies += "dev.zio" %% "zio-aws-netty"            % "<version>"
```

And here is the example code:

```scala

object ZIOAWSExample extends ZIOAppDefault {
  val program: ZIO[Ec2 & ElasticBeanstalk, AwsError, Unit] =
    for {
      appsResult <- ElasticBeanstalk.describeApplications(
        DescribeApplicationsRequest(applicationNames = Some(List(ApplicationName("my-service"))))
      )
      app <- appsResult.getApplications.map(_.headOption)
      _ <- app match {
        case Some(appDescription) =>
          for {
            applicationName <- appDescription.getApplicationName
            _ <- Console
              .printLine(
                s"Got application description for $applicationName"
              )
              .ignore

            envStream = ElasticBeanstalk.describeEnvironments(
              DescribeEnvironmentsRequest(applicationName =
                Some(applicationName)
              )
            )

            _ <- envStream.run(ZSink.foreach { env =>
              env.getEnvironmentName.flatMap { environmentName =>
                (for {
                  environmentId <- env.getEnvironmentId
                  _ <- Console
                    .printLine(
                      s"Getting the EB resources of $environmentName"
                    )
                    .ignore

                  resourcesResult <-
                    ElasticBeanstalk.describeEnvironmentResources(
                      DescribeEnvironmentResourcesRequest(environmentId =
                        Some(environmentId)
                      )
                    )
                  resources <- resourcesResult.getEnvironmentResources
                  _ <- Console
                    .printLine(
                      s"Getting the EC2 instances in $environmentName"
                    )
                    .ignore
                  instances <- resources.getInstances
                  instanceIds <- ZIO.foreach(instances)(_.getId)
                  _ <- Console
                    .printLine(
                      s"Instance IDs are ${instanceIds.mkString(", ")}"
                    )
                    .ignore

                  reservationsStream = Ec2.describeInstances(
                    DescribeInstancesRequest(instanceIds = Some(instanceIds.map(id => zio.aws.ec2.model.primitives.InstanceId(ResourceId.unwrap(id)))))
                  )
                  _ <- reservationsStream.run(ZSink.foreach { reservation =>
                    reservation.getInstances
                      .flatMap { instances =>
                        ZIO.foreach(instances) { instance =>
                          for {
                            id <- instance.getInstanceId
                            typ <- instance.getInstanceType
                            launchTime <- instance.getLaunchTime
                            _ <- Console.printLine(s"  instance $id:").ignore
                            _ <- Console.printLine(s"    type: $typ").ignore
                            _ <- Console
                              .printLine(
                                s"    launched at: $launchTime"
                              )
                              .ignore
                          } yield ()
                        }
                      }
                  })
                } yield ()).catchAll { error =>
                  Console
                    .printLineError(
                      s"Failed to get info for $environmentName: $error"
                    )
                    .ignore
                }
              }
            })
          } yield ()
        case None =>
          ZIO.unit
      }
    } yield ()

  override def run: URIO[ZIOAppArgs, ExitCode] = {
    val httpClient = NettyHttpClient.default
    val awsConfig = httpClient >>> AwsConfig.default
    val aws = awsConfig >>> (Ec2.live ++ ElasticBeanstalk.live)

    program
      .provideLayer(aws)
      .either
      .flatMap {
        case Left(error) =>
          Console.printLineError(s"AWS error: $error").ignore.as(ExitCode.failure)
        case Right(_) =>
          ZIO.unit.as(ExitCode.success)
      }
  }
}
```

---

## Migration guide

# Migration from v3.x.x.x

There are some major changes compared to the ZIO 1 version (v3.x.x.x and v4.x.x.x). This section contains detailed information about what changed and how to modify existing code.

## New package names
Previously the zio-aws packages had the root package `io.github.vigoo.zioaws`. By moving to the ZIO organisation this changed to be simply `zio.aws`. 

So imports for something using the EC2 and ElasticBeanstalk APIs would change from:

```scala

```

to 

```scala

```

Some other changes to the imports may be necessary because of switching to the new service pattern of ZIO, described below.

## New getter names

In previous version of zio-aws the generated models provided to ways to access fields:

- a ZIO effect that failed if the value is `None` with the name of the field (for example `.instanceId`)
- a simple value accessor returning an `Option` value with the `Value` suffix (for example `.instanceIdValue`)

This has been changed to match the convention used in `zio-k8s` to the following:

- the ZIO effect for requiring that a field has value now has the `get` prefix (for example `.getInstanceId`)
- the value with the field name is the simple accessor returning the optional value: (for example `.instanceId`)

So for example the following code that prints information about EC2 instances:

```scala
for {
    id <- instance.instanceId
    typ <- instance.instanceType
    launchTime <- instance.launchTime
    _ <- console.putStrLn(s"  instance $id:").ignore
    _ <- console.putStrLn(s"    type: $typ").ignore
    _ <- console.putStrLn(s"    launched at: $launchTime").ignore
} yield ()
```

would have to be changed to:

```scala
for {
    id <- instance.getInstanceId
    typ <- instance.getInstanceType
    launchTime <- instance.getLaunchTime
    _ <- Console.printLine(s"  instance $id:").ignore
    _ <- Console.printLine(s"    type: $typ").ignore
    _ <- Console.printLine(s"    launched at: $launchTime").ignore
} yield ()
```

## New service pattern
Previous versions were following the _ZIO module pattern 1.0_, so all the generated AWS services consisted of a type alias using `Has`, a trait called `Service` and a package object with the service's name. With ZIO 2 we no longer have `Has` and the recommended way to structure services was changed to the simple new service pattern, where each service is just a `trait` and one or more implementation classes, constructed by layers.

In practice this means that:
- instead of the package object with lower-case name, the accessor functions are now in the service companion object (`ElasticBeanstalk.describeApplications` instead of `elasticbeanstalk.describeApplications`)
- same for the layers, for example `Ec2.live` instead of `ec2.live`
- the service trait has the name of the service (for example `Ec2`) instead of `Service` 

The new service pattern is also applied to the core services like `AwsConfig` and the http implementations.

## Optional parameters
A very large part of the fields of AWS models are optional. These previously had the type `Option`, and although for extracting data from them the library already had the generated getters, constructing these data types still required wrapping most of the parameters in `Some(...)`. To reduce this boilerplate zio-aws now uses the `Optional` type instead of `Option`, which was first used in [zio-k8s](https://coralogix.github.io/zio-k8s/docs/overview/overview_resources#model). Now the two libraries share the same type which was moved to [zio-prelude](https://zio.github.io/zio-prelude/).

The following example:

```scala
kinesis.describeStreamConsumer(
  DescribeStreamConsumerRequest(
    consumerName = Some(consumerName),
    streamARN = Some(streamDescription.streamDescriptionValue.streamARNValue)
  )
)
```

becomes

```scala
Kinesis.describeStreamConsumer(
  DescribeStreamConsumerRequest(
    consumerName = consumerName,
    streamARN = streamDescription.streamDescription.streamARN
  )
)
```

## Newtypes
Previously zio-aws generated simple Scala type aliases for primitive types in the AWS SDKs. For example the `TableName` type in `zio-aws-dynamodb` was just a type alias for `String`:

```scala
package io.github.vigoo.zioaws.dynamodb.model

package object primitives {
  type TableName = String
}
```

The new version uses [zio-prelude's newtype wrappers](https://zio.github.io/zio-prelude/docs/newtypes/) to provide better type safety:

```scala
package zio.aws.dynamodb.model

package object primitives {
  object TableName extends Subtype[String]
  type TableName = TableName.Type
}
```

In practice this means that we have to explicitly wrap these primitive values, for example instead of:

```scala
elasticbeanstalk.describeApplications(
  DescribeApplicationsRequest(applicationNames = Some(List("my-service")))
)
```

now we have to write

```scala
ElasticBeanstalk.describeApplications(
  DescribeApplicationsRequest(applicationNames = List(ApplicationName("my-service")))
)
```

and if we need to convert them to the underlying primitive type we need to call `unwrap`:

```scala
ResourceId.unwrap(id)
```

## Aspects
zio-aws introduced `AwsCallAspect` soon after Adam Fraser's [talk](https://www.youtube.com/watch?v=gcqWdNwNEPg&t=2s) but it was a custom implementation, defined as:

```scala
trait AwsCallAspect[-R] { self =>
  def apply[R1 <: R, A](
    f: ZIO[R1, AwsError, Described[A]]
  ): ZIO[R1, AwsError, Described[A]]

  // ...
}
```

These aspects can be applied to whole zio-aws service layers to add logging, metrics, retries etc for every AWS Java SDK call.

The ZIO 2.0.0 version is now using ZIO's built-in aspect support as a base:

```scala
type AwsCallAspect[-R] =
  ZIOAspect[Nothing, R, AwsError, AwsError, Nothing, Described[_]]
```

### New built-in aspects
With ZIO 2 we have logging and metrics support built-in, so zio-aws now provides ready to use aspects for logging and monitoring AWS calls:

```scala
val callLogging: AwsCallAspect[Any]
def callDuration(prefix: String, boundaries: MetricKeyType.Histogram.Boundaries): AwsCallAspect[Any]
```

### Changes in defining aspects
The following example aspect uses `rezilience` to add circuit breaking for an AWS service:

```scala
def circuitBreaking(cb: CircuitBreaker[AwsError]): AwsCallAspect[Any] =
  new AwsCallAspect[Any] {
    override final def apply[R1 <: Any, A](
      f: ZIO[R1, AwsError, Described[A]]
    ): ZIO[R1, AwsError, Described[A]] =
      cb(f).mapError(policyError =>
        AwsError.fromThrowable(policyError.toException)
      )
  }
```

because of the changed base type for `AwsCallAspect` with the new version the same aspect is defined like this:

```scala
def circuitBreaking(cb: CircuitBreaker[AwsError]): AwsCallAspect[Any] =
  new AwsCallAspect[Any] {
    override final def apply[R, E >: AwsError <: AwsError, A <: Described[_]](
      f: ZIO[R, E, A]
    )(implicit trace: Trace): ZIO[R, E, A] =
      cb(f).mapError(policyError =>
        AwsError.fromThrowable(policyError.toException)
      )
  }
```

The main differences:

- requiring the implicit `Trace` 
- the constraints on the error and result types are now expressed as type bounds because the ZIO aspect is more generic

## New config library

The 3.x series of zio-aws was using `zio-config 1.x`. The recently released 4.x series was using `zio-config 2.x` which is the ZIO 1 version of the config library's new API; The ZIO 2 version of zio-aws uses `zio-config 3.x` which is the same new config API but for ZIO 2.

This means that coming from zio-aws 3.x requires upgrading to the new config API. For more information about the changes in zio-config, [see it's release notes](https://github.com/zio/zio-config/releases/tag/v2.0.0-RC1).

---

## Wrappers

# Client wrappers

### Service modules
For each AWS Service the library defines a _ZIO service_ with wrapper functions for all the _operations_, a `live` 
implementation calling the Java SDK and a `mock` implementation using [zio-mock](https://github.com/zio/zio-mock).

The live implementation depends on a core _AWS configuration layer_:

```scala
val live: ZLayer[AwsConfig, Throwable, Ec2]
``` 

The `AwsConfig` layer defines how each service's async Java client gets configured, including the http client which is
provided by another layer `AwsConfig` is depending on.

Each module has accessor functions for _all operations_ of the given service.

### Operations

For simple request-response operations the library generates a very light wrapper:

```scala
def deleteVolume(request: DeleteVolumeRequest): ZIO[Ec2, AwsError, DeleteVolumeResponse.ReadOnly]
```

For operations where either the input or the output or both are _byte streams_, a `ZStream` wrapper is generated:

```scala
def getObject(request: GetObjectRequest): ZIO[S3, AwsError, StreamingOutputResult[Any, GetObjectResponse.ReadOnly, Byte]]
def putObject(request: PutObjectRequest, body: ZStream[Any, AwsError, Byte]): ZIO[S3, AwsError, PutObjectResponse.ReadOnly]
```

where the output is a stream packed together with additional response data:

```scala
case class StreamingOutputResult[R, Response, Item](
  response: Response,
  output: ZStream[R, AwsError, Item]
)
```

For operations with _event streams_ a `ZStream` of a model type gets generated:

```scala
def startStreamTranscription(request: StartStreamTranscriptionRequest, input: ZStream[Any, AwsError, AudioStream]): ZStream[TranscribeStreaming, AwsError, TranscriptEvent.ReadOnly]
```

And for all operations that supports _pagination_, streaming wrappers gets generated:

```scala
def scan(request: ScanRequest): ZStream[DynamoDb, AwsError, Map[AttributeName, AttributeValue.ReadOnly]]
```

Note that for event streaming or paginating operations returning a `ZStream` the actual AWS call happens when the stream gets pulled.

For use cases when calling the _paginating_ interface directly is necessary - for example when forwarding paginated results through a HTTP API, the library generates non-streaming wrappers as well for these methods.

For example the DynamoDB `scan` method's non-streaming variant is defined as:

```scala
def scanPaginated(request: ScanRequest): ZIO[DynamoDb, ScanResponse.ReadOnly]
```

### Model wrappers
For each model type a set of wrappers are generated, providing the following functionality:

- Case classes with default parameter values instead of the _builder pattern_
- [zio-prelude's newtype wrappers](https://zio.github.io/zio-prelude/docs/newtypes/) for primitive types
- Automatic conversion to Scala collection types
- ADTs instead of the Java enums 
- ZIO getter functions to "get or fail" the optional model fields
- Using zio-prelude's `Optional` type to eliminate boilerplate when constructing models with many optional fields

The following example from the `zio-aws-elasticsearch` library shows how the generated case classes look like, to be used as input for the service operations:

```scala
case class DescribePackagesFilter(name: Optional[DescribePackagesFilterName] = Optional.Absent, 
                                  value: Optional[Iterable[primitives.DescribePackagesFilterValue]] = Optional.Absent) {
    def buildAwsValue(): software.amazon.awssdk.services.elasticsearch.model.DescribePackagesFilter = {

      software.amazon.awssdk.services.elasticsearch.model.DescribePackagesFilter
        .builder()
        .optionallyWith(name.map(value => value.unwrap))(_.name)
        .optionallyWith(value.map(value => value.map { item => item: java.lang.String }.asJava))(_.value)
        .build()
    }

    def asReadOnly: DescribePackagesFilter.ReadOnly = DescribePackagesFilter.wrap(buildAwsValue())
}
```

When processing the _results_ of the operations (either directly or though the `ZStream` wrappers), the AWS Java model types are wrapped
by a _read-only wrapper interface_. The following example shows one from the `transcribe` module:

```scala
object CreateMedicalVocabularyResponse {
  private lazy val zioAwsBuilderHelper: BuilderHelper[software.amazon.awssdk.services.transcribe.model.CreateMedicalVocabularyResponse] = BuilderHelper.apply

  trait ReadOnly {
    def editable: CreateMedicalVocabularyResponse = CreateMedicalVocabularyResponse(vocabularyNameValue.map(value => value), languageCodeValue.map(value => value), vocabularyStateValue.map(value => value), lastModifiedTimeValue.map(value => value), failureReasonValue.map(value => value))
    def vocabularyName: Optional[VocabularyName]
    def languageCode: Optional[LanguageCode]
    def vocabularyState: Optional[VocabularyState]
    def lastModifiedTime: Optional[DateTime]
    def failureReason: Optional[FailureReason]
    def getVocabularyName: ZIO[Any, AwsError, VocabularyName] = AwsError.unwrapOptionField("vocabularyName", vocabularyNameValue)
    def getLanguageCode: ZIO[Any, AwsError, LanguageCode] = AwsError.unwrapOptionField("languageCode", languageCodeValue)
    def getVocabularyState: ZIO[Any, AwsError, VocabularyState] = AwsError.unwrapOptionField("vocabularyState", vocabularyStateValue)
    def getLastModifiedTime: ZIO[Any, AwsError, DateTime] = AwsError.unwrapOptionField("lastModifiedTime", lastModifiedTimeValue)
    def getFailureReason: ZIO[Any, AwsError, FailureReason] = AwsError.unwrapOptionField("failureReason", failureReasonValue)
  }

  private class Wrapper(impl: software.amazon.awssdk.services.transcribe.model.CreateMedicalVocabularyResponse) extends CreateMedicalVocabularyResponse.ReadOnly {
    // ... implements the ReadOnly interface by querying the underlying Java object
  }

  def wrap(impl: software.amazon.awssdk.services.transcribe.model.CreateMedicalVocabularyResponse): ReadOnly = new Wrapper(impl)
}
```

As a large part of the models in the AWS SDK are defined as _optional_, the generated wrapper also contains ZIO accessor functions,
which lift the option value to make it more comfortable to chain the AWS operations.

### Mocks
Each module also contains generated [_ZIO Test mocks_](https://github.com/zio/zio-mock) for the given service.

The following example shows how to use them with the `zio-aws-athena` library:

```scala
val athena = AthenaMock.StartQueryExecution(
  hasField(
    "queryString",
    (startQueryExecutionRequest: StartQueryExecutionRequest) =>
      startQueryExecutionRequest.queryString,
      equalTo(givenQuery)
  ),
  value(
    StartQueryExecutionResponse.wrap(
      software.amazon.awssdk.services.athena.model.StartQueryExecutionResponse
        .builder()
        .queryExecutionId(executionId)
        build()
    )
  )
)

codeUsingAthena.provide(athena)
```

---

## Compile-Time Resource Safety with Scope

Welcome to ZIO Blocks Scope—a library that makes resource management safe, composable, and verifiable at compile time. If you've ever struggled with `try/finally` chains, wondered when to close a database connection, or worried about resources outliving their owners, this tutorial is for you. You don't need any prior experience with Scope or advanced type system concepts to follow along.

## Learning Objectives

By the end of this tutorial, you will be able to:

- **Allocate resources safely** and trust the compiler to prevent use-after-close bugs
- **Understand the `$[A]` type and `scoped { }` block** — the two core mechanisms that make resource safety compile-time verifiable
- **Manage complex resource lifetimes** including cleanup with `defer`, nested scopes with `lower`, and reference counting with `Resource.shared`
- **Wire applications** using `Wire` and `Resource.from` for compile-time verified dependency injection

We'll learn through step-by-step examples that build from simple (allocating a single resource) to advanced patterns (nested scopes, shared resources, and dependency injection). Each section builds on the previous one, so we recommend reading from top to bottom.

## 1. The Problem: Why Resource Management Is Hard

Managing resources safely is deceptively difficult. When you open a database connection, read a file, or create a network socket, you must eventually close it—but only once, and only after you're done using it. Forget to close it, and you leak a resource. Close it too early, and you get a crash. Close it twice, and you get an error.

Nested resources make this worse. If you're reading a config file and then opening a database connection, you need nested `try/finally` blocks. If the inner resource throws an exception, the outer one may not close. Callbacks and closures can capture resources that outlive their intended scope, creating subtle use-after-free bugs.

Consider a typical `try/finally` pattern in Scala:

```scala
trait Connection extends AutoCloseable {
  def createStatement(): Statement
}
trait Statement extends AutoCloseable {
  def executeQuery(sql: String): ResultSet
}
trait ResultSet
def openConnection(): Connection = ???
def process(result: ResultSet): Unit = ???
def handleError(e: Exception): Unit = ???
val sql = ""

try {
  val connection = openConnection()
  try {
    val statement = connection.createStatement()
    try {
      val result = statement.executeQuery(sql)
      process(result)
    } finally {
      statement.close()
    }
  } finally {
    connection.close()
  }
} catch {
  case e: Exception => handleError(e)
}
```

This is hard to read, easy to get wrong, and doesn't compose—every additional resource adds another level of nesting. If an exception happens during cleanup, subsequent finalizers may not run. And if you pass a resource to another function, there's no compile-time guarantee that function won't use it after your scope ends.

Scope eliminates these problems by making resource ownership explicit and enforcing it at compile time.

## 2. Your First Scope

Let's start with the simplest possible example: allocating a single resource, using it, and letting it close automatically.

Scope builds on the concept of `Scope.global`—the root scope that outlives your entire program. You enter a scoped region using `scoped { }`, and inside that block, you can allocate resources. When the block exits, all resources close in reverse order (last allocated, first closed).

Here's a database connection that prints messages when opening and closing:

```scala

class Database extends AutoCloseable {
  def connect(): Unit = println("Database: connecting")
  def query(sql: String): String = s"Results of: $sql"
  override def close(): Unit = println("Database: closing")
}

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(Resource {
    val database = new Database()
    database.connect()
    database
  })

  $(db) { database =>
    val result = database.query("SELECT * FROM users")
    println(s"Query result: $result")
  }
}
```

Let's break down what happens:

- `Scope.global.scoped { scope => ... }` — Creates a scoped region. When the block exits, all allocated resources are closed.
- `import scope._` — Imports scope operations: `$`, `allocate`, and `defer`.
- `allocate(Resource { ... })` — Allocates a resource. Since `Database` extends `AutoCloseable`, its `close()` method is automatically registered as a finalizer.
- `$[Database]` — A scoped value of type `Database`. It can only be used within the scope where it was allocated.
- `$(db) { database => ... }` — Unwraps the scoped value and passes it to the block. This is the only way to access a resource.

When the scope exits, `database.close()` runs automatically, printing `"Database: closing"`.

With multiple resources, finalizers run in LIFO order (last allocated, first closed):

```scala

class Connection extends AutoCloseable {
  def name: String = this.getClass.getSimpleName
  override def close(): Unit = println(s"$name: closing")
}

Scope.global.scoped { scope =>

  val conn1 = allocate(Resource(new Connection() { override def name = "Connection-1" }))
  val conn2 = allocate(Resource(new Connection() { override def name = "Connection-2" }))
  val conn3 = allocate(Resource(new Connection() { override def name = "Connection-3" }))

  println("All connections allocated")
  println("Exiting scope - connections will close in reverse order (3, 2, 1)")
}
```

:::note
The `$[A]` type is central to Scope's compile-time safety: each scope instance has a unique `$` type that cannot be mixed with other scopes. Resources allocated in one scope literally cannot be used in another, even at the type level. This prevents entire classes of resource-lifetime bugs at compile time.
:::

## 3. The `$[A]` Type and the `$` Operator

To understand Scope, you need to understand `$[A]`. It is a marker type that says "this value is owned by a specific scope." At runtime, `$[A]` is erased to `A` (zero overhead), but at compile time, it enforces a fundamental rule: **you can only use a resource via the operator defined by the scope it belongs to**.

Every scope instance has its own unique `$` type. Two different scopes have incompatible `$` types, so you cannot accidentally mix them at the type level. For example, if you allocate a resource in an outer scope and try to use it directly in an inner scope without `lower`, you get a compile error because the types are incompatible.

To use a parent-scoped resource in a child scope safely, you must use the `lower` operator, which we'll cover in Section 7.

To use a resource, apply the `$(value)` operator (it's a macro) with a single-argument block. The parameter must be used as the receiver of all operations:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(msg)
  override def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val logger = allocate(Resource(new Logger()))

  // Correct: parameter used as receiver
  $(logger) { log =>
    log.log("Message 1")
    log.log("Message 2")
  }
}
```

The following patterns will not compile:
- `$(logger) { log => logger.log("X") }` — cannot use `logger` outside the `$()` operator
- `$(logger) { log => val x = log; x }` — result is `$[Logger]`, which is not `Unscoped`
- `$(logger) { log => (log, "data") }` — tuples containing `$[Logger]` are not `Unscoped`

The `$` operator automatically unwraps the result if it is an `Unscoped[B]` type. We'll cover `Unscoped` in detail in Section 5, but for now, know that primitives like `Int`, `String`, and `Unit` are always `Unscoped`:

```scala

class Calculator extends AutoCloseable {
  def add(a: Int, b: Int): Int = a + b
  override def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val calc = allocate(Resource(new Calculator()))

  // Result is Int, which is Unscoped, so it unwraps automatically
  val sum = $(calc)(_.add(3, 4))
  assert(sum == 7)
}
```

:::note
The `$` operator is not a regular method—it's a compile-time macro that inspects what you do with its parameter. This macro enforcement is what makes the rule "parameter must be receiver" checkable at compile time, not at runtime.
:::

## 4. Resources: Describing Acquisition and Cleanup

A `Resource[A]` is a lazy description of how to acquire a value and register any cleanup needed when the scope closes. Creating a resource does not acquire it—that only happens when you pass it to `scope.allocate()`.

There are several ways to construct a `Resource`:

**`Resource(value: => A)`** — The simplest form. Wraps a by-name value. If the value is `AutoCloseable`, its `close()` method is automatically registered:

```scala

class Connection extends AutoCloseable {
  override def close(): Unit = println("Connection closed")
}

Scope.global.scoped { scope =>

  // By-name: creates connection on demand, closes automatically
  val conn = allocate(Resource(new Connection()))

  $(conn) { c =>
    println("Using connection")
  }
}
```

**`Resource.acquireRelease(acquire)(release)`** — Explicit lifecycle control. Useful when cleanup is not a simple method call:

```scala

case class Connection(id: Int) {
  def query(sql: String): String = s"[$id] $sql"
}

Scope.global.scoped { scope =>

  // Explicit acquire and release
  val conn = allocate(Resource.acquireRelease {
    println("Opening connection...")
    Connection(42)
  } { c =>
    println(s"Closing connection $c")
  })

  $(conn) { c =>
    println(c.query("SELECT 1"))
  }
}
```

**`Resource.fromAutoCloseable(thunk)`** — Explicit wrapper for `AutoCloseable` subtypes. Type-safe and clear:

```scala

Scope.global.scoped { scope =>

  val file = allocate(Resource.fromAutoCloseable(new FileInputStream("/etc/hostname")))

  $(file) { f =>
    val bytes = Array.ofDim[Byte](100)
    val n = f.read(bytes)
    println(s"Read $n bytes")
  }
}
```

Resources compose: you can transform them with `map`, combine them with `zip`, or sequence them with `flatMap`:

```scala

case class Database(url: String) extends AutoCloseable {
  def getConnection(name: String): Connection =
    Connection(s"$url/$name")
  override def close(): Unit = println(s"Database closed: $url")
}

case class Connection(name: String) extends AutoCloseable {
  def query(sql: String): String = s"[$name] $sql"
  override def close(): Unit = println(s"Connection closed: $name")
}

Scope.global.scoped { scope =>

  val dbResource = Resource(new Database("localhost:5432"))

  // flatMap sequences: database opens first, then connection opens from it
  val connResource = dbResource.flatMap { db =>
    Resource(db.getConnection("myapp"))
  }

  val conn = allocate(connResource)

  $(conn) { c =>
    println(c.query("SELECT 1"))
  }
}
```

When you use `flatMap` to open a connection from an already-open database, the database stays open until the scope closes, ensuring the connection is always valid.

## 5. Returning Values: The `Unscoped[A]` Typeclass

When you exit a `scoped { }` block, the scope closes and all resources are finalized. But what can you return from a `scoped` block? A scoped value `$[A]` cannot escape—it would be used after the scope closes. That's where `Unscoped[A]` comes in.

`Unscoped[A]` is a typeclass that marks types as safe to return from a scoped block. It means "this type contains no scope-bound resources; it is pure data." The type system only allows returning a value if it has an `Unscoped` instance:

```scala

case class Config(host: String, port: Int)

Scope.global.scoped { scope =>

  // Config is a case class—it has Unscoped by default
  Config("localhost", 5432)
}
```

Built-in `Unscoped` instances include primitives (`Int`, `String`, `Boolean`), collections (`List[A]`, `Map[K, V]`), and common library types (`UUID`, `java.time.LocalDate`). If you define a case class with no resource fields, it automatically gets an `Unscoped` instance:

```scala

case class Result(count: Int, message: String)
case class ServerConfig(host: String, port: Int, timeout: Long)

Scope.global.scoped { scope =>

  // Both can be returned—they have implicit Unscoped instances
  Result(42, "success") -> ServerConfig("0.0.0.0", 8080, 30000)
}
```

If you define a custom class and want to return it from `scoped`, you need to either derive or provide an `Unscoped` instance. In Scala 3, case classes support automatic derivation via `derives`:

```scala

case class CustomData(x: Int, y: String) derives Unscoped

Scope.global.scoped { scope =>

  CustomData(10, "hello")
}
```

Alternatively, provide an instance explicitly using a `given`:

```scala

case class CustomData(x: Int, y: String)

given Unscoped[CustomData] = Unscoped.derived

Scope.global.scoped { scope =>

  CustomData(10, "hello")
}
```

If you try to return a scoped value without an `Unscoped` instance, you get a compile error:

```scala

class Connection extends AutoCloseable {
  override def close(): Unit = ()
}

// This does not compile because Connection has no Unscoped instance:
// val conn = Scope.global.scoped { scope =>
//   import scope._
//   allocate(Resource(new Connection()))
// }
```

This compile-time barrier prevents entire classes of resource-lifetime bugs—you cannot accidentally return a resource reference from a scoped block.

## 6. Finalizers and Error Handling

Sometimes you need to register cleanup that is not a simple resource `close()`. The `defer` operator lets you register arbitrary cleanup actions:

```scala

case class Transaction(id: Int) {
  def begin(): Unit = println(s"Transaction $id: begin")
  def commit(): Unit = println(s"Transaction $id: commit")
  def rollback(): Unit = println(s"Transaction $id: rollback")
}

Scope.global.scoped { scope =>

  val txn = Transaction(1)
  txn.begin()

  // Register rollback as cleanup
  scope.defer {
    txn.rollback()
  }

  // If we commit, cancel the rollback
  txn.commit()
}
```

`scope.defer()` returns a `DeferHandle` that lets you cancel the finalizer before it runs:

```scala

case class Transaction(id: Int) {
  def begin(): Unit = println(s"Transaction $id: begin")
  def commit(): Unit = println(s"Transaction $id: commit")
  def rollback(): Unit = println(s"Transaction $id: rollback")
}

Scope.global.scoped { scope =>

  val txn = Transaction(1)
  txn.begin()

  // Register rollback, but keep the handle so we can cancel it
  val rollbackHandle = scope.defer {
    txn.rollback()
  }

  // On success, cancel the rollback finalizer
  txn.commit()
  rollbackHandle.cancel()

  println("Scope exiting - rollback will NOT run because we cancelled it")
}
```

Finalizers run in **LIFO order** (last registered, first executed) and are guaranteed to run even if the scoped block throws an exception. If multiple finalizers throw, they are collected:

```scala

Scope.global.scoped { scope =>

  scope.defer { println("Finalizer 1") }
  scope.defer { println("Finalizer 2") }
  scope.defer { println("Finalizer 3") }

  println("Block executing")
}
```

When this code runs, the finalizers execute in reverse order of registration:

```
Block executing
Finalizer 3
Finalizer 2
Finalizer 1
```

:::note
The `DeferHandle.cancel()` operation is O(1)—it marks the finalizer as cancelled without traversing the entire registry. This makes it safe to use in performance-sensitive code, like transaction commits in tight loops.
:::

## 7. Nested Scopes and `lower`

Scopes form a tree: each scope can create child scopes via `scope.scoped { }`. Children are guaranteed to close before their parent, which is the foundation of hierarchical resource management.

But child scopes have a different `$[A]` type than their parent, so a parent-scoped value cannot be directly used in a child. That's where `lower` comes in. `Scope#lower` re-tags a parent-scoped value into a child scope, which is safe because the parent always outlives the child:

```scala

class Database(name: String) extends AutoCloseable {
  def query(sql: String): String = s"[$name] $sql"
  override def close(): Unit = println(s"Database [$name] closed")
}

class Connection(db: String, id: Int) extends AutoCloseable {
  def query(sql: String): String = s"[$db/$id] $sql"
  override def close(): Unit = println(s"Connection [$db/$id] closed")
}

Scope.global.scoped { parentScope =>

  // Open database in parent scope
  val db = allocate(Resource(new Database("maindb")))

  // Use database in parent scope
  $(db) { database =>
    println(s"Parent: ${database.query("SELECT 1")}")
  }

  // Create a child scope (e.g., for a request)
  parentScope.scoped { childScope =>

    // Lower the parent-scoped database into the child scope
    val dbInChild = childScope.lower(db)

    // Now we can use the database in the child scope
    val conn = allocate(Resource(new Connection("maindb", 1)))

    $(dbInChild) { database =>
      $(conn) { connection =>
        println(s"Child: ${connection.query("SELECT 2")}")
      }
    }

    println("Child scope exiting - connection closes first")
  }

  println("Parent scope exiting - database closes after children")
}
```

When the child scope exits, all resources allocated in it close first. Then the parent scope's finalizers run. This ensures that if a child holds a reference to a parent's resource, that resource is not closed until all children have finished.

## 8. Explicit Lifetime Management with `open()`

The `scoped { }` syntax ties resource lifetime to a lexical block. But sometimes you need explicit, decoupled lifetime management—for example, a request handler that opens a connection when processing begins and closes it when processing ends, independent of any fixed scope nesting.

Child scopes created via `scoped { }` are owned by the thread that creates them and must close within the creating thread. But `Scope.global.open()` creates an unowned scope that can be closed from any thread. This is useful for bridging structured scope-based resource management with callbacks or cross-thread communication:

```scala

class ConnectionPool extends AutoCloseable {
  def acquire(): String = "conn-001"
  override def close(): Unit = println("Pool closed")
}

// Simulate a request handler in an async framework
case class RequestContext(id: Int) {
  var connection: String = ""

  def setConnection(c: String): Unit = {
    connection = c
    println(s"Processing request ${id}, connection: $connection")
  }
}

// Using open() to manage lifetime explicitly, decoupled from lexical scope
val request = RequestContext(1)

Scope.global.scoped { scope =>

  // open() creates an unowned scope that can be closed explicitly
  $(open()) { handle =>
    val requestScope = handle.scope

    try {
      // Use the scope to allocate and manage the resource
      requestScope.scoped { innerScope =>

        val pool = allocate(Resource(new ConnectionPool()))

        $(pool) { p =>
          request.setConnection(p.acquire())
        }
      }
    } finally {
      // Close the open scope explicitly and handle any finalizer failures
      handle.close().orThrow()
    }
  }
}
```

The standard pattern for managing resource lifetimes in your application is to use `scoped { }` with careful nesting. The `open()` method is reserved for low-level integration points (like application startup/shutdown boundaries) and is not typically needed in application code.

:::note
Thread ownership is enforced for child scopes created with `scoped { }` but not for unowned scopes from `open()`. This difference allows Scope to prevent thread-related bugs in structured code while still supporting integration with callback-driven or asynchronous frameworks that require explicit lifetime management.
:::

## 9. Shared Resources and Reference Counting

When multiple parts of your application need the same heavyweight resource (like a database connection pool), you want to create it once and destroy it only when the last user is done. `Resource.shared` provides reference-counted sharing:

```scala

class ConnectionPool(id: Int) extends AutoCloseable {
  def getConnection(): String = s"conn-from-pool-$id"
  override def close(): Unit = println(s"Pool $id closed")
}

case class UserService(poolId: String)
case class OrderService(poolId: String)

Scope.global.scoped { scope =>

  // Create a shared resource: only one pool instance, reference-counted
  val sharedPool = Resource.shared[ConnectionPool] { _ =>
    println("Creating shared pool...")
    new ConnectionPool(42)
  }

  // Both services allocate from the same shared resource
  // The pool is created once, destroyed after both services release it
  val pool1 = allocate(sharedPool)
  val pool2 = allocate(sharedPool)

  $(pool1) { p1 =>
    $(pool2) { p2 =>
      // Both p1 and p2 point to the same pool instance
      println(s"Service 1 got: ${p1.getConnection()}")
      println(s"Service 2 got: ${p2.getConnection()}")
      println("Both services are using the same shared pool instance")
    }
  }

  println("Scope exiting - shared pool closed (once)")
}
```

`Resource.shared` is memoized: the first `allocate` creates the pool, and subsequent `allocate` calls get the same instance. The finalizer runs only after all allocations have released their reference (implicitly when the scope closes).

This pattern is essential for applications with a shared database connection pool, cache, or logging infrastructure.

## 10. Dependency Injection with Wire

Applications often have many services with interdependencies. Manual wiring is error-prone: forget a dependency, pass the wrong type, create a cycle, or accidentally duplicate an instance where sharing was intended.

`Wire` and `Resource.from` provide compile-time dependency injection. Wires are builders that describe how to construct instances, and `Resource.from` resolves the entire dependency graph:

```scala

case class DbConfig(url: String)
case class Database(config: DbConfig) extends AutoCloseable {
  override def close(): Unit = println("Database closed")
}

case class CacheService(db: Database)
case class AuthService(db: Database)
case class AppService(cache: CacheService, auth: AuthService)

Scope.global.scoped { scope =>

  // Wire.shared means all dependents get the same instance
  val configWire = Wire(DbConfig("localhost"))
  val dbWire = Wire.shared[Database]
  val cacheWire = Wire.shared[CacheService]
  val authWire = Wire.shared[AuthService]
  val appWire = Wire.shared[AppService]

  // Resource.from resolves all wires and returns the root app service
  val app = allocate(Resource.from[AppService](
    configWire, dbWire, cacheWire, authWire, appWire
  ))

  $(app) { a =>
    println("App service created: " + a.toString())
  }
}
```

The compiler verifies that:
- Every dependency can be satisfied.
- No unsatisfiable circular dependencies exist.
- Types match correctly.

If you violate any of these rules, you get a clear compile error before runtime.

## 11. Thread Ownership

On the JVM, Scope enforces a structured concurrency guarantee: each `Scope.Child` (any scope created with `scoped { }` or as a child of another scope) is owned by the thread that created it. This prevents a subtle class of bugs where a scope reference escapes to another thread and resources are used or closed on the wrong thread.

You can check ownership with `Scope#isOwner`:

```scala

Scope.global.scoped { scope =>
  println(s"Global scope owned by current thread: ${scope.isOwner}")

  scope.scoped { childScope =>
    println(s"Child scope owned by current thread: ${childScope.isOwner}")
  }
}
```

If you try to use a child scope from a different thread, operations like `allocate`, `defer`, and `$(value)()` throw an `IllegalStateException`:

```scala

class Database extends AutoCloseable {
  override def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val db = allocate(Resource(new Database()))

  // Attempting to use the scope from another thread will throw:
  // val thread = new Thread(() => {
  //   $(db) { _ => }  // throws IllegalStateException: ownership violation
  // })
  // thread.start()
}
```

To pass a resource to another thread safely, use `Scope.global.open()` to create an unowned scope, or redesign to keep all operations on the creating thread.

For platforms like Scala.js (single-threaded), thread ownership checks are disabled—ownership is always considered valid.

:::note
Thread ownership enforcement is not about thread safety in the traditional sense—it's about structured concurrency. It prevents subtle bugs where a scope escapes to another thread and resources are closed on a different thread than they were allocated.
:::

## 12. Common Errors and Troubleshooting

This section lists the most common runtime and compile errors, explains what caused them, and how to fix them.

### Runtime Errors

**`IllegalStateException: Scope is closed`** when calling `allocate`, `defer`, `$`, or `open` on a closed scope:

```scala

class Database extends AutoCloseable {
  override def close(): Unit = ()
}

var db: Option[Database] = None  // Wrong: trying to escape scoped value

Scope.global.scoped { scope =>

  // db = Some(allocate(Resource.fromAutoCloseable(new Database())))  // Error: can't assign scope.$[Database] to Option[Database]
}

// scope is now closed; this throws IllegalStateException:
// db.foreach(_.close())
```

**Fix:** Ensure all resource usage happens before the scoped block exits. If you need to return a resource reference, return only the underlying value (wrapped in an `Unscoped` type).

**`IllegalStateException: Thread ownership violation`** when calling operations on a child scope from a different thread:

```scala

var scope: Option[Scope] = None

Scope.global.scoped { s =>
  scope = Some(s)
}

Future {
  // This throws: the scope was created on the main thread, not this one
  // scope.foreach(_.scoped { _ => })
}
```

**Fix:** Use `Scope.global.open()` to create an unowned scope that can be shared across threads, or keep all operations on the creating thread.

### Compile Errors

**No `Unscoped` instance for type `T`** when trying to return a value from `scoped`:

```scala

class Connection extends AutoCloseable {
  override def close(): Unit = ()
}

// This does not compile:
// val conn = Scope.global.scoped { scope =>
//   import scope._
//   allocate(Resource(new Connection()))  // ERROR: $[Connection] has no Unscoped
// }
```

**Fix:** Only return types with an `Unscoped` instance (primitives, case classes, collections). If you need to return a resource reference, extract its underlying value first, or use `Wire` to manage the resource's lifetime.

**Cannot call method directly on `$[T]`**:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(msg)
  override def close(): Unit = ()
}

// This does not compile:
// Scope.global.scoped { scope =>
//   import scope._
//   val logger = allocate(Resource(new Logger()))
//   logger.log("test")  // ERROR: method log not visible on $[Logger]
// }
```

**Fix:** Use the `$` operator to unwrap: `$(logger)(_.log("test"))`.

**`Wire` cannot resolve dependency** when wiring fails due to missing constructor arguments:

```scala

case class Database(url: String) extends AutoCloseable {
  override def close(): Unit = ()
}

// If Database constructor is not satisfied by wires, compilation fails:
// val dbWire: Wire[Database] = Wire.shared[Database]  // ERROR: no Wire[String] for url
```

**Fix:** Provide a wire for every required dependency:

```scala

case class Database(url: String) extends AutoCloseable {
  override def close(): Unit = ()
}

// Correct: provide the String
val urlWire = Wire("localhost")
val dbWire = Wire.shared[Database]

Scope.global.scoped { scope =>

  val db = allocate(Resource.from[Database](urlWire, dbWire))
  $(db) { d => println("Connected to " + d.toString()) }
}
```

---

## Putting It Together

Now let's combine everything we've learned into a single, realistic example. This example demonstrates:

- Multiple allocated resources (database and logger)
- Wire-based dependency injection with shared and unique wires
- Automatic cleanup in reverse allocation order
- The complete interaction of all concepts

This example combines core concepts — allocation, cleanup, resource composition, and dependency injection:

```scala

class Database extends AutoCloseable {
  def query(sql: String): String = s"Results: $sql"
  override def close(): Unit = println("Closing database")
}

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(s"[LOG] $msg")
  override def close(): Unit = println("Closing logger")
}

class Application(database: Database, logger: Logger) extends AutoCloseable {
  def run(): String = {
    logger.log("Starting application")
    val result = database.query("SELECT * FROM users")
    logger.log(s"Query executed: $result")
    result
  }
  override def close(): Unit = logger.log("Shutting down application")
}

// Create an app using Scope with wire-based dependency injection
Scope.global.scoped { scope =>

  // Wire.shared means all dependents receive the same Logger instance
  // Wire.shared[Database] means all dependents receive the same Database
  // Resource.from[Application] constructs Application by resolving dependencies
  val app = allocate(
    Resource.from[Application](
      Wire.shared[Database],
      Wire.shared[Logger]
    )
  )

  // Use the application
  $(app) { a =>
    val result = a.run()
    println(s"Result: $result")

    // Register additional cleanup operations
    defer {
      println("Application cleanup complete")
    }
  }

  // All resources are closed in LIFO order: Application, Logger, then Database
  ()
}

println("Program finished")
```

This example demonstrates:
- **Wire-based DI**: `Wire.shared[T]` automatically derives how to construct `T` from its dependencies
- **Resource.from**: The macro analyzes the dependency graph and automatically allocates in correct order
- **Composition**: Multiple resources (Database, Logger, Application) with declared dependencies
- **Cleanup**: All resources close in reverse order when the scope exits (LIFO)

---

## What You've Learned

In this tutorial, you learned:

- What Scope is and why compile-time resource safety matters
- How to allocate, use, and manage resources with the `scoped { }` block
- The `$[A]` type and how it enforces resource ownership at the type level
- How to construct resources with `Resource` and its variants
- The `Unscoped[A]` typeclass and why returning resources is forbidden
- How to register cleanup with `defer` and manage finalizer order
- Nested scopes and the `lower` operator for hierarchical resource management
- Advanced patterns like `open()` for decoupled lifetime management, `Resource.shared` for reference counting, and `Wire` for dependency injection
- Common errors and how to fix them

You now have a solid foundation in Scope. The next step is to see how to apply these concepts in practice with realistic scenarios.

---

## Running the Examples

The code examples in this tutorial are embedded directly in the documentation and compile with mdoc. To run them locally:

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Compile and run the tutorial examples:**

All examples from the tutorial sections are compile-checked using mdoc. To verify they compile:

```bash
sbt "docs/mdoc --in docs/guides/compile-time-resource-safety-with-scope.md"
```

**3. Run standalone examples from the scope-examples module:**

The repository also includes additional companion examples in `scope-examples/`. For example:

```bash
sbt "scope-examples/runMain runDatabaseExample"
sbt "scope-examples/runMain runCachingExample"
sbt "scope-examples/runMain runThreadOwnershipExample"
```

To compile all examples:

```bash
sbt "scope-examples/compile"
```

---

## Where to Go Next

- **Ready to use this in practice?** Check out the how-to guides (coming soon) which walk through real-world examples.
- **Want to dive deeper into the API?** Read the [Scope Reference](../reference/resource-management/scope.md) for comprehensive API documentation.
- **Interested in related concepts?** Explore dependency injection with [Wire](../reference/resource-management/wire.md) or resource composition patterns.

---

## Summary

You now understand Scope's core concepts:

- **`$[A]`** — a type-level owner tag that prevents resources from escaping their scope.
- **`scoped { }`** — the syntax for entering and exiting a scope.
- **`allocate` and `defer`** — operations to register resources and cleanup.
- **`Resource`** — lazy descriptions of acquisition and cleanup.
- **`Unscoped`** — a compile-time guarantee that a type is safe to return from a scope.
- **Nesting and `lower`** — hierarchical resource management with compile-time parent-child relationships.
- **Shared resources** — reference counting for multiply-used resources.
- **`Wire` and dependency injection** — compile-time-verified wiring of complex applications.
- **Thread ownership** — JVM enforcement of structured concurrency.

For complete API documentation, see the [Scope Reference](../reference/resource-management/scope.md).

---

## Query DSL with Reified Optics — Part 3: Extending the Expression Language

In this guide, we will extend the ZIO Blocks query DSL with an expression language that goes beyond what `SchemaExpr` provides out of the box. By the end, you will have an `Expr` ADT that adds SQL-specific predicates (`IN`, `BETWEEN`, `IS NULL`, `LIKE`), type-safe aggregate functions (`COUNT`, `SUM`, `AVG`), and conditional expressions (`CASE WHEN`) — all composable with the built-in `SchemaExpr` operators from Parts 1 and 2.

This is Part 3 of the Query DSL series. [Part 1](./query-dsl-reified-optics.md) covered building query expressions, and [Part 2](./query-dsl-sql.md) covered translating them to SQL. Here, we handle the cases where the built-in expression language is not enough.

**What we'll cover:**

- Why `SchemaExpr` is deliberately closed and what that means for extension
- Designing an `Expr` ADT that is a superset of `SchemaExpr`
- Translating `SchemaExpr` into `Expr` via `fromSchemaExpr`
- Adding SQL-specific predicates: `IN`, `BETWEEN`, `IS NULL`, `LIKE`
- Writing bridge extension methods for seamless `SchemaExpr` + `Expr` composition
- Building a single unified SQL interpreter
- Adding type-safe aggregate functions and `CASE WHEN` for advanced SQL generation

## The Problem

The built-in `SchemaExpr` operators cover the fundamentals: equality, comparisons, boolean logic, arithmetic, and basic string operations. But real-world SQL requires more. Consider these common queries:

```sql
-- Membership test
SELECT * FROM products WHERE category IN ('Electronics', 'Books', 'Toys')

-- Range check
SELECT * FROM products WHERE price BETWEEN 10.0 AND 100.0

-- Null handling
SELECT * FROM products WHERE description IS NULL

-- Pattern matching with SQL wildcards
SELECT * FROM products WHERE name LIKE 'Lap%'

-- Aggregation
SELECT category, COUNT(*), AVG(price)
FROM products GROUP BY category HAVING COUNT(*) > 2

-- Conditional logic
SELECT name, CASE WHEN price > 100 THEN 'expensive' ELSE 'cheap' END AS tier
FROM products
```

None of these can be expressed with `SchemaExpr` alone. You could generate the SQL strings manually, but then you lose composability — you can no longer mix these operations with the type-safe `SchemaExpr` predicates from Parts 1 and 2.

Since `SchemaExpr` is a sealed trait, you cannot add new cases to it. Instead, we define an `Expr` ADT that is a superset of `SchemaExpr` — it includes equivalent nodes for everything `SchemaExpr` can express, plus our custom SQL-specific operations. A `fromSchemaExpr` function translates `SchemaExpr` values into `Expr`, enabling seamless interoperability with a single unified interpreter.

## Prerequisites

This guide builds on [Part 1: Expressions](./query-dsl-reified-optics.md) and [Part 2: SQL Generation](./query-dsl-sql.md). You should be comfortable building `SchemaExpr` values and translating them to SQL.

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

```scala

```

## Domain Setup

We reuse the product catalog domain from the earlier guides:

```scala
case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}
```

## Designing the Expr ADT

The key insight is the **translation pattern**: define your own sealed trait whose node types are a superset of `SchemaExpr`'s, then provide a `fromSchemaExpr` function that converts any `SchemaExpr` into your ADT. This gives you a single unified interpreter.

```
  Built-in (sealed, not extensible)          Your extension (superset)
┌───────────────────────────────┐       ┌───────────────────────────────────────┐
│  SchemaExpr[S, A]             │       │  Expr[S, A]                           │
│  ├── Literal                  │──────▶│  ├── Lit(value, schema)               │
│  ├── Optic                    │──────▶│  ├── Column(Optic)                    │
│  ├── Relational               │──────▶│  ├── Relational(left, right, RelOp)   │
│  ├── Logical (And/Or)         │──────▶│  ├── And / Or                         │
│  ├── Not                      │──────▶│  ├── Not                              │
│  ├── Arithmetic               │──────▶│  ├── Arithmetic(left, right, ArithOp) │
│  ├── StringConcat             │──────▶│  ├── StringConcat                     │
│  ├── StringRegexMatch         │──────▶│  ├── StringRegexMatch                 │
│  └── StringLength             │──────▶│  ├── StringLength                     │
└───────────────────────────────┘       │  ├── In(expr, values, schema)         ← new   │
         fromSchemaExpr ────────────────│  ├── Between(expr, low, high, schema) ← new   │
                                        │  ├── IsNull(expr)             ← new   │
                                        │  ├── Like(expr, pattern)      ← new   │
                                        │  ├── Agg(function, expr)      ← new   │
                                        │  └── CaseWhen(branches, else) ← new   │
                                        └───────────────────────────────────────┘
```

The `Expr` ADT includes mirrored nodes for every `SchemaExpr` case, plus SQL-specific extensions. It uses its own operator types (`RelOp`, `ArithOp`) and type-safe aggregate functions (`AggFunction[A, B]`).

Here is the full `Expr` ADT with its supporting types:

```scala
sealed trait Expr[S, A]

object Expr {

  // --- Core nodes (superset of SchemaExpr's nodes) ---
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  // Relational
  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]

  // Logical
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]

  // Arithmetic
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]

  // String
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  // --- SQL-specific extensions (no SchemaExpr equivalents) ---
  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  // --- Aggregates ---
  final case class Agg[S, A, B](function: AggFunction[A, B], expr: Expr[S, A]) extends Expr[S, B]

  // --- Conditional ---
  final case class CaseWhen[S, A](
    branches: List[(Expr[S, Boolean], Expr[S, A])],
    otherwise: Option[Expr[S, A]]
  ) extends Expr[S, A]

  // --- Factory methods ---
  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)

  def count[S, A](expr: Expr[S, A]): Expr[S, Long]   = Agg(AggFunction.Count(), expr)
  def sum[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Sum, expr)
  def avg[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Avg, expr)
  def min[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Min(), expr)
  def max[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Max(), expr)

  def caseWhen[S, A](branches: (Expr[S, Boolean], Expr[S, A])*): CaseWhenBuilder[S, A] =
    CaseWhenBuilder(branches.toList)

  case class CaseWhenBuilder[S, A](branches: List[(Expr[S, Boolean], Expr[S, A])]) {
    def otherwise(value: Expr[S, A]): Expr[S, A] = CaseWhen(branches, Some(value))
    def end: Expr[S, A] = CaseWhen(branches, None)
  }

  // --- Translation from SchemaExpr ---
  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = {
    val result = se match {
      case SchemaExpr.Optic(optic)       => Column(optic)
      case l: SchemaExpr.Literal[_, _]   => Lit(l.value, l.schema)

      case SchemaExpr.Relational(l, r, op) =>
        val relOp = op match {
          case SchemaExpr.RelationalOperator.Equal              => RelOp.Equal
          case SchemaExpr.RelationalOperator.NotEqual           => RelOp.NotEqual
          case SchemaExpr.RelationalOperator.LessThan           => RelOp.LessThan
          case SchemaExpr.RelationalOperator.LessThanOrEqual    => RelOp.LessThanOrEqual
          case SchemaExpr.RelationalOperator.GreaterThan        => RelOp.GreaterThan
          case SchemaExpr.RelationalOperator.GreaterThanOrEqual => RelOp.GreaterThanOrEqual
        }
        Relational(fromSchemaExpr(l), fromSchemaExpr(r), relOp)

      case SchemaExpr.Logical(l, r, op) => op match {
        case SchemaExpr.LogicalOperator.And => And(fromSchemaExpr(l), fromSchemaExpr(r))
        case SchemaExpr.LogicalOperator.Or  => Or(fromSchemaExpr(l), fromSchemaExpr(r))
      }

      case SchemaExpr.Not(inner) => Not(fromSchemaExpr(inner))

      case SchemaExpr.Arithmetic(l, r, op, _) =>
        val arithOp = op match {
          case SchemaExpr.ArithmeticOperator.Add      => ArithOp.Add
          case SchemaExpr.ArithmeticOperator.Subtract => ArithOp.Subtract
          case SchemaExpr.ArithmeticOperator.Multiply => ArithOp.Multiply
        }
        Arithmetic(fromSchemaExpr(l), fromSchemaExpr(r), arithOp)

      case SchemaExpr.StringConcat(l, r)              => StringConcat(fromSchemaExpr(l), fromSchemaExpr(r))
      case SchemaExpr.StringRegexMatch(regex, string) => StringRegexMatch(fromSchemaExpr(regex), fromSchemaExpr(string))
      case SchemaExpr.StringLength(string)            => StringLength(fromSchemaExpr(string))
    }
    result.asInstanceOf[Expr[S, A]]
  }
}

// --- Operators ---

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

// Typed aggregate functions
sealed trait AggFunction[A, B] {
  def name: String
}
object AggFunction {
  case class Count[A]() extends AggFunction[A, Long]         { val name = "COUNT" }
  case object Sum       extends AggFunction[Double, Double]  { val name = "SUM" }
  case object Avg       extends AggFunction[Double, Double]  { val name = "AVG" }
  case class Min[A]()   extends AggFunction[A, A]            { val name = "MIN" }
  case class Max[A]()   extends AggFunction[A, A]            { val name = "MAX" }
}
```

Here are some keynotes on the design:

- **Type-safe aggregates** — `AggFunction[A, B]` encodes the return type: `COUNT` returns `Long`, `SUM`/`AVG` return `Double`, `MIN`/`MAX` preserve the input type.
- **Typed literals and predicates** — `Lit(value, schema)`, `In(expr, values, schema)`, and `Between(expr, low, high, schema)` all carry a `Schema[A]` so the SQL renderer can format values correctly using the schema rather than runtime type checks.
- **`fromSchemaExpr`** — one-way translation recursively converts every `SchemaExpr` node into its `Expr` equivalent, mapping operators along the way.

## Extension Methods

To make the new operations feel natural, we define implicit classes on `Optic`, `Expr`, and `SchemaExpr`. The bridge implicit class on `SchemaExpr` auto-translates at the boundary via `fromSchemaExpr`, so `SchemaExpr` and `Expr` values compose seamlessly with `&&` and `||`:

```scala
implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

// Boolean combinators — accept both Expr and SchemaExpr on the right
implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

// Bridge: SchemaExpr on the left, Expr on the right
implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}
```

The bridge implicit classes are the key to ergonomic composition. When you write `Product.category.in("Electronics") && (Product.rating >= 4)`, the `&&` on `Expr[S, Boolean]` sees a `SchemaExpr[S, Boolean]` on the right and auto-translates it. Similarly, `(Product.rating >= 4) && Product.category.in("Electronics")` uses the `SchemaExpr` bridge to translate the left side. No explicit `.toExpr` is needed in most cases.

:::tip
The `.toExpr` method is still available for cases where you need to explicitly lift a `SchemaExpr[S, Boolean]` — for example, when building `CASE WHEN` branch conditions.
:::

## The Unified SQL Interpreter

With the `Expr` ADT, we write a single interpreter that handles all cases directly:

```scala
def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Column(optic)      => columnName(optic)
  case Expr.Lit(value, schema) => sqlLiteral(value, schema)

  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal              => "="
      case RelOp.NotEqual           => "<>"
      case RelOp.LessThan           => "<"
      case RelOp.LessThanOrEqual    => "<="
      case RelOp.GreaterThan        => ">"
      case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"

  case Expr.And(l, r) => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)  => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)    => s"NOT (${exprToSql(e)})"

  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add      => "+"
      case ArithOp.Subtract => "-"
      case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"

  case Expr.StringConcat(l, r)          => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s)  => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)             => s"LENGTH(${exprToSql(s)})"

  // SQL-specific
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)          => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern)   => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"

  // Aggregates
  case Expr.Agg(func, e) => s"${func.name}(${exprToSql(e)})"

  // CASE WHEN
  case Expr.CaseWhen(branches, otherwise) =>
    val cases = branches.map { case (cond, value) =>
      s"WHEN ${exprToSql(cond)} THEN ${exprToSql(value)}"
    }.mkString(" ")
    val elseClause = otherwise.map(e => s" ELSE ${exprToSql(e)}").getOrElse("")
    s"CASE $cases$elseClause END"
}
```

The typed `sqlLiteral[A](value, schema)` uses the `Schema` carried by `Lit`, `In`, and `Between` to format values correctly — strings get quoted, booleans become `TRUE`/`FALSE`, numbers stay as-is. Every AST node that holds literal values carries a `Schema[A]`, so a single `sqlLiteral` function handles all formatting with no untyped fallbacks.

## SQL-Specific Predicates

With the ADT, extensions, and interpreter in place, the new operators work directly on optics:

```scala
exprToSql(Product.category.in("Electronics", "Books", "Toys"))
// res0: String = "category IN ('Electronics', 'Books', 'Toys')"

exprToSql(Product.price.between(10.0, 100.0))
// res1: String = "(price BETWEEN 10.0 AND 100.0)"

exprToSql(Product.name.isNull)
// res2: String = "name IS NULL"

exprToSql(Product.name.isNotNull)
// res3: String = "NOT (name IS NULL)"

exprToSql(Product.name.like("Lap%"))
// res4: String = "name LIKE 'Lap%'"
```

Each extension method on `Optic` returns an `Expr` node. The interpreter handles it and produces the corresponding SQL fragment.

## Composing with SchemaExpr

The bridge implicit classes handle the translation automatically. You can freely mix `SchemaExpr` predicates (from `===`, `>`, etc.) with `Expr` predicates (from `.in`, `.between`, etc.) using `&&` and `||`:

```scala
// SchemaExpr values from built-in operators
val highRated: SchemaExpr[Product, Boolean] = Product.rating >= 4

// Expr values from extension methods
val inCategory: Expr[Product, Boolean] = Product.category.in("Electronics", "Books")
val priceRange: Expr[Product, Boolean] = Product.price.between(10.0, 500.0)

// Seamless composition — bridge auto-translates at the boundary
val combined: Expr[Product, Boolean] =
  inCategory && priceRange && highRated
```

```scala
exprToSql(combined)
// res5: String = "((category IN ('Electronics', 'Books') AND (price BETWEEN 10.0 AND 500.0)) AND (rating >= 4))"
```

The `&&` between `priceRange` (an `Expr`) and `highRated` (a `SchemaExpr`) triggers the overloaded `&&` that accepts `SchemaExpr` on the right. It calls `fromSchemaExpr` internally, so no explicit `.toExpr` is needed.

You can also start from a `SchemaExpr` on the left — the bridge implicit class handles it:

```scala
val query: Expr[Product, Boolean] =
  Product.category.in("Electronics", "Books") &&
  Product.price.between(10.0, 500.0) &&
  (Product.rating >= 4) &&
  Product.name.like("M%")
```

```scala
exprToSql(query)
// res6: String = "(((category IN ('Electronics', 'Books') AND (price BETWEEN 10.0 AND 500.0)) AND (rating >= 4)) AND name LIKE 'M%')"
```

A helper function to generate full SELECT statements from `Expr` predicates:

```scala
def selectWhere(table: String, predicate: Expr[_, Boolean]): String =
  s"SELECT * FROM $table WHERE ${exprToSql(predicate)}"
```

```scala
selectWhere("products", query)
// res7: String = "SELECT * FROM products WHERE (((category IN ('Electronics', 'Books') AND (price BETWEEN 10.0 AND 500.0)) AND (rating >= 4)) AND name LIKE 'M%')"
```

## Aggregate Expressions

The `Agg` node wraps any column expression with a type-safe aggregate function. The return type reflects SQL semantics: `COUNT` returns `Expr[S, Long]`, `SUM`/`AVG` return `Expr[S, Double]`, and `MIN`/`MAX` preserve the input type:

```scala
exprToSql(Expr.count(Expr.col(Product.name)))
// res8: String = "COUNT(name)"

exprToSql(Expr.avg(Expr.col(Product.price)))
// res9: String = "AVG(price)"

exprToSql(Expr.max(Expr.col(Product.rating)))
// res10: String = "MAX(rating)"
```

Aggregates compose with the rest of the ADT. Build a `GROUP BY` query by combining aggregate SQL fragments with a select builder:

```scala
def selectGroupBy(
  table: String,
  columns: List[String],
  groupBy: List[String],
  having: Option[String] = None
): String = {
  val base = s"SELECT ${columns.mkString(", ")} FROM $table GROUP BY ${groupBy.mkString(", ")}"
  having.fold(base)(h => s"$base HAVING $h")
}
```

```scala
selectGroupBy(
  "products",
  columns = List(
    "category",
    s"${exprToSql(Expr.count(Expr.col(Product.name)))} AS product_count",
    s"${exprToSql(Expr.avg(Expr.col(Product.price)))} AS avg_price"
  ),
  groupBy = List("category"),
  having = Some(s"${exprToSql(Expr.count(Expr.col(Product.name)))} > 2")
)
// res11: String = "SELECT category, COUNT(name) AS product_count, AVG(price) AS avg_price FROM products GROUP BY category HAVING COUNT(name) > 2"
```

## CASE WHEN Expressions

The `CaseWhen` node represents SQL's conditional expression. Use the `Expr.caseWhen` builder with `(condition -> result)` pairs and an optional `.otherwise` clause:

```scala
val priceLabel: Expr[Product, String] = Expr.caseWhen[Product, String](
  (Product.price > 100.0).toExpr  -> Expr.lit[Product, String]("expensive"),
  (Product.price > 10.0).toExpr   -> Expr.lit[Product, String]("moderate")
).otherwise(Expr.lit[Product, String]("cheap"))
```

```scala
exprToSql(priceLabel)
// res12: String = "CASE WHEN (price > 100.0) THEN 'expensive' WHEN (price > 10.0) THEN 'moderate' ELSE 'cheap' END"
```

`CASE WHEN` is useful for computed columns in SELECT lists:

```scala
val stockStatus: Expr[Product, String] = Expr.caseWhen[Product, String](
  (Product.inStock === true).toExpr -> Expr.lit[Product, String]("available")
).otherwise(Expr.lit[Product, String]("out of stock"))
```

```scala
val selectSql = s"SELECT name, price, ${exprToSql(priceLabel)} AS tier, ${exprToSql(stockStatus)} AS status FROM products"
// selectSql: String = "SELECT name, price, CASE WHEN (price > 100.0) THEN 'expensive' WHEN (price > 10.0) THEN 'moderate' ELSE 'cheap' END AS tier, CASE WHEN (inStock = TRUE) THEN 'available' ELSE 'out of stock' END AS status FROM products"
println(selectSql)
// SELECT name, price, CASE WHEN (price > 100.0) THEN 'expensive' WHEN (price > 10.0) THEN 'moderate' ELSE 'cheap' END AS tier, CASE WHEN (inStock = TRUE) THEN 'available' ELSE 'out of stock' END AS status FROM products
```

## Putting It Together

Here is a complete, self-contained example that defines the independent expression ADT, translates from `SchemaExpr`, and generates advanced SQL:

```scala

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}

// --- Independent Expr ADT ---

sealed trait Expr[S, A]

object Expr {
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  final case class Agg[S, A, B](function: AggFunction[A, B], expr: Expr[S, A]) extends Expr[S, B]
  final case class CaseWhen[S, A](
    branches: List[(Expr[S, Boolean], Expr[S, A])],
    otherwise: Option[Expr[S, A]]
  ) extends Expr[S, A]

  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)
  def count[S, A](expr: Expr[S, A]): Expr[S, Long]   = Agg(AggFunction.Count(), expr)
  def sum[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Sum, expr)
  def avg[S](expr: Expr[S, Double]): Expr[S, Double]  = Agg(AggFunction.Avg, expr)
  def min[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Min(), expr)
  def max[S, A](expr: Expr[S, A]): Expr[S, A]         = Agg(AggFunction.Max(), expr)

  def caseWhen[S, A](branches: (Expr[S, Boolean], Expr[S, A])*): CaseWhenBuilder[S, A] =
    CaseWhenBuilder(branches.toList)

  case class CaseWhenBuilder[S, A](branches: List[(Expr[S, Boolean], Expr[S, A])]) {
    def otherwise(value: Expr[S, A]): Expr[S, A] = CaseWhen(branches, Some(value))
    def end: Expr[S, A] = CaseWhen(branches, None)
  }

  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = {
    val result = se match {
      case SchemaExpr.Optic(optic)       => Column(optic)
      case l: SchemaExpr.Literal[_, _]   => Lit(l.value, l.schema)
      case SchemaExpr.Relational(l, r, op) =>
        val relOp = op match {
          case SchemaExpr.RelationalOperator.Equal              => RelOp.Equal
          case SchemaExpr.RelationalOperator.NotEqual           => RelOp.NotEqual
          case SchemaExpr.RelationalOperator.LessThan           => RelOp.LessThan
          case SchemaExpr.RelationalOperator.LessThanOrEqual    => RelOp.LessThanOrEqual
          case SchemaExpr.RelationalOperator.GreaterThan        => RelOp.GreaterThan
          case SchemaExpr.RelationalOperator.GreaterThanOrEqual => RelOp.GreaterThanOrEqual
        }
        Relational(fromSchemaExpr(l), fromSchemaExpr(r), relOp)
      case SchemaExpr.Logical(l, r, op) => op match {
        case SchemaExpr.LogicalOperator.And => And(fromSchemaExpr(l), fromSchemaExpr(r))
        case SchemaExpr.LogicalOperator.Or  => Or(fromSchemaExpr(l), fromSchemaExpr(r))
      }
      case SchemaExpr.Not(inner) => Not(fromSchemaExpr(inner))
      case SchemaExpr.Arithmetic(l, r, op, _) =>
        val arithOp = op match {
          case SchemaExpr.ArithmeticOperator.Add      => ArithOp.Add
          case SchemaExpr.ArithmeticOperator.Subtract => ArithOp.Subtract
          case SchemaExpr.ArithmeticOperator.Multiply => ArithOp.Multiply
        }
        Arithmetic(fromSchemaExpr(l), fromSchemaExpr(r), arithOp)
      case SchemaExpr.StringConcat(l, r)              => StringConcat(fromSchemaExpr(l), fromSchemaExpr(r))
      case SchemaExpr.StringRegexMatch(regex, string) => StringRegexMatch(fromSchemaExpr(regex), fromSchemaExpr(string))
      case SchemaExpr.StringLength(string)            => StringLength(fromSchemaExpr(string))
    }
    result.asInstanceOf[Expr[S, A]]
  }
}

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

sealed trait AggFunction[A, B] { def name: String }
object AggFunction {
  case class Count[A]() extends AggFunction[A, Long]        { val name = "COUNT" }
  case object Sum       extends AggFunction[Double, Double]  { val name = "SUM" }
  case object Avg       extends AggFunction[Double, Double]  { val name = "AVG" }
  case class Min[A]()   extends AggFunction[A, A]            { val name = "MIN" }
  case class Max[A]()   extends AggFunction[A, A]            { val name = "MAX" }
}

// --- Extension methods with bridge ---

implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}

// --- SQL rendering ---

def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Column(optic)      => columnName(optic)
  case Expr.Lit(value, schema) => sqlLiteral(value, schema)
  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal => "="; case RelOp.NotEqual => "<>"
      case RelOp.LessThan => "<"; case RelOp.LessThanOrEqual => "<="
      case RelOp.GreaterThan => ">"; case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.And(l, r)                   => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)                    => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)                      => s"NOT (${exprToSql(e)})"
  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add => "+"; case ArithOp.Subtract => "-"; case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.StringConcat(l, r)         => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s) => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)            => s"LENGTH(${exprToSql(s)})"
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)        => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern) => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"
  case Expr.Agg(func, e)     => s"${func.name}(${exprToSql(e)})"
  case Expr.CaseWhen(branches, otherwise) =>
    val cases = branches.map { case (cond, value) =>
      s"WHEN ${exprToSql(cond)} THEN ${exprToSql(value)}"
    }.mkString(" ")
    val elseClause = otherwise.map(e => s" ELSE ${exprToSql(e)}").getOrElse("")
    s"CASE $cases$elseClause END"
}

// --- Usage ---

// 1. SQL-specific predicates — seamless composition
val q1 = Product.category.in("Electronics", "Books") &&
  Product.price.between(10.0, 500.0) &&
  (Product.rating >= 4) &&
  Product.name.like("M%")

println(s"SELECT * FROM products WHERE ${exprToSql(q1)}")

// 2. Type-safe aggregation
val countExpr: Expr[Product, Long]   = Expr.count(Expr.col(Product.name))
val avgExpr: Expr[Product, Double]   = Expr.avg(Expr.col(Product.price))
val countSql = exprToSql(countExpr)
val avgSql   = exprToSql(avgExpr)
println(s"SELECT category, $countSql AS cnt, $avgSql AS avg_price FROM products GROUP BY category HAVING $countSql > 2")

// 3. CASE WHEN
val tier = Expr.caseWhen[Product, String](
  (Product.price > 100.0).toExpr -> Expr.lit[Product, String]("expensive"),
  (Product.price > 10.0).toExpr  -> Expr.lit[Product, String]("moderate")
).otherwise(Expr.lit[Product, String]("cheap"))

println(s"SELECT name, price, ${exprToSql(tier)} AS tier FROM products")
```

## Going Further

- **[Part 1: Expressions](./query-dsl-reified-optics.md)** -- Building query expressions with reified optics
- **[Part 2: SQL Generation](./query-dsl-sql.md)** -- Translating built-in expressions to SQL
- **[Part 4: A Fluent SQL Builder](./query-dsl-fluent-builder.md)** -- Type-safe SELECT, UPDATE, INSERT, DELETE with seamless condition mixing
- **[SchemaExpr Reference](../reference/schema-expr.md)** -- Full API coverage of expression types
- **[Optics Reference](../reference/optics.md)** -- Lens, Prism, Optional, and Traversal

The translation pattern shown here extends to any domain where `SchemaExpr` falls short. The same approach works for MongoDB operators (`$in`, `$exists`, `$elemMatch`), Elasticsearch queries (`terms`, `range`, `exists`), or GraphQL filters. Define an independent ADT, provide a `fromSchemaExpr` translation, add your domain-specific nodes, and write a single unified interpreter.

---

## Query DSL with Reified Optics — Part 4: A Fluent SQL Builder

In this guide, we will build a fluent, type-safe SQL statement builder on top of the query expression language from Parts 1–3. By the end, you will have builder functions for SELECT, UPDATE, INSERT, and DELETE that compose naturally with both built-in `SchemaExpr` operators and extended `Expr` operators — no manual `.toExpr` lifting required.

We'll take an incremental approach: starting with seamless condition composition, then adding table references and fluent statement builders for each SQL operation until we have a complete, native-feeling SQL DSL.

This is Part 4 of the Query DSL series. [Part 1](./query-dsl-reified-optics.md) covered expressions, [Part 2](./query-dsl-sql.md) covered SQL generation, and [Part 3](./query-dsl-extending.md) covered extending the expression language.

**What we'll cover:**

- Bridging `SchemaExpr` and `Expr` for seamless `&&` / `||` composition
- Defining table references with `Table[S]`
- Deriving table names from `Schema` metadata via `Modifier.config` and auto-pluralization
- Building fluent SELECT queries with `.columns()`, `.where()`, `.orderBy()`, `.limit()`
- Building UPDATE statements with `.set()` and `.where()`
- Building INSERT and DELETE statements
- Rendering all statement types to SQL strings

## The Problem

Parts 1–3 gave us a powerful expression language for building WHERE clauses. But constructing full SQL statements still requires string concatenation:

```scala
val whereSql = exprToSql(myCondition)
val sql = s"UPDATE products SET price = 9.99 WHERE $whereSql"
```

Column names in SET clauses are hand-written strings that can drift from your schema, table names are repeated magic strings, and there is no compile-time connection between the statement structure and your domain types.

Additionally, mixing built-in `SchemaExpr` operators (`===`, `>=`) with extended `Expr` operators (`between`, `like`) required explicit `.toExpr` calls in Part 3:

```scala
// Part 3: explicit .toExpr required
val q = (Product.rating >= 4).toExpr && Product.price.between(10.0, 500.0)
```

In a fluent builder, this friction breaks the flow. We want to write:

```scala
update(Product.table)
  .set(Product.price, 9.99)
  .where(
    Product.category === "Books" &&
      Product.rating >= 4 &&
      Product.price.between(10.0, 30.0) &&
      Product.name.like("M%")
  )
```

In this guide, we solve both problems: bridge extensions eliminate `.toExpr`, and type-safe statement builders eliminate string manipulation for SQL construction.

## Prerequisites

This guide builds on [Part 1: Expressions](./query-dsl-reified-optics.md), [Part 2: SQL Generation](./query-dsl-sql.md), and [Part 3: Extending the Expression Language](./query-dsl-extending.md).

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

## Domain Setup

We carry forward the product catalog domain and the Part 3 independent `Expr` ADT. The key additions in Part 4 are bridge extension methods, schema-driven table names, and statement builder types. The `Expr` ADT used here is the same independent design from Part 3.

```scala

// --- Table reference ---

case class Table[S](name: String)

object Table {
  def derived[S](implicit schema: Schema[S]): Table[S] = Table(tableName(schema))
}

// --- Table name derivation ---

def tableName[S](schema: Schema[S]): String =
  schema.reflect.modifiers.collectFirst {
    case Modifier.config(key, value) if key == "sql.table_name" => value
  }.getOrElse(pluralize(schema.reflect.typeId.name.toLowerCase))

def pluralize(word: String): String =
  if (word.endsWith("s") || word.endsWith("x") || word.endsWith("z") ||
      word.endsWith("ch") || word.endsWith("sh")) word + "es"
  else if (word.endsWith("y") && !word.endsWith("ay") && !word.endsWith("ey") &&
           !word.endsWith("oy") && !word.endsWith("uy")) word.dropRight(1) + "ies"
  else word + "s"

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val table: Table[Product] = Table.derived  // "Product" → "products"

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}

case class OrderItem(
  orderId: Int,
  productId: Int,
  quantity: Int,
  unitPrice: Double
)

object OrderItem extends CompanionOptics[OrderItem] {
  implicit val schema: Schema[OrderItem] = Schema.derived
    .modifier(Modifier.config("sql.table_name", "order_items"))

  val table: Table[OrderItem] = Table.derived  // annotation → "order_items"

  val orderId: Lens[OrderItem, Int]      = optic(_.orderId)
  val productId: Lens[OrderItem, Int]    = optic(_.productId)
  val quantity: Lens[OrderItem, Int]     = optic(_.quantity)
  val unitPrice: Lens[OrderItem, Double] = optic(_.unitPrice)
}

// --- Expr ADT (from Part 3) ---

sealed trait Expr[S, A]

object Expr {
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)

  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = {
    val result = se match {
      case SchemaExpr.Optic(optic)      => Column(optic)
      case l: SchemaExpr.Literal[_, _]  => Lit(l.value, l.schema)
      case SchemaExpr.Relational(l, r, op) =>
        val relOp = op match {
          case SchemaExpr.RelationalOperator.Equal              => RelOp.Equal
          case SchemaExpr.RelationalOperator.NotEqual           => RelOp.NotEqual
          case SchemaExpr.RelationalOperator.LessThan           => RelOp.LessThan
          case SchemaExpr.RelationalOperator.LessThanOrEqual    => RelOp.LessThanOrEqual
          case SchemaExpr.RelationalOperator.GreaterThan        => RelOp.GreaterThan
          case SchemaExpr.RelationalOperator.GreaterThanOrEqual => RelOp.GreaterThanOrEqual
        }
        Relational(fromSchemaExpr(l), fromSchemaExpr(r), relOp)
      case SchemaExpr.Logical(l, r, op) => op match {
        case SchemaExpr.LogicalOperator.And => And(fromSchemaExpr(l), fromSchemaExpr(r))
        case SchemaExpr.LogicalOperator.Or  => Or(fromSchemaExpr(l), fromSchemaExpr(r))
      }
      case SchemaExpr.Not(inner) => Not(fromSchemaExpr(inner))
      case SchemaExpr.Arithmetic(l, r, op, _) =>
        val arithOp = op match {
          case SchemaExpr.ArithmeticOperator.Add      => ArithOp.Add
          case SchemaExpr.ArithmeticOperator.Subtract => ArithOp.Subtract
          case SchemaExpr.ArithmeticOperator.Multiply => ArithOp.Multiply
        }
        Arithmetic(fromSchemaExpr(l), fromSchemaExpr(r), arithOp)
      case SchemaExpr.StringConcat(l, r)              => StringConcat(fromSchemaExpr(l), fromSchemaExpr(r))
      case SchemaExpr.StringRegexMatch(regex, string) => StringRegexMatch(fromSchemaExpr(regex), fromSchemaExpr(string))
      case SchemaExpr.StringLength(string)            => StringLength(fromSchemaExpr(string))
    }
    result.asInstanceOf[Expr[S, A]]
  }
}

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

// --- SQL helpers ---

def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

// --- Optic extension methods ---

implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

// --- Boolean combinators with bridge extensions ---

implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}

// --- Single unified SQL interpreter ---

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Column(optic)      => columnName(optic)
  case Expr.Lit(value, schema) => sqlLiteral(value, schema)
  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal              => "="
      case RelOp.NotEqual           => "<>"
      case RelOp.LessThan           => "<"
      case RelOp.LessThanOrEqual    => "<="
      case RelOp.GreaterThan        => ">"
      case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.And(l, r) => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)  => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)    => s"NOT (${exprToSql(e)})"
  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add      => "+"
      case ArithOp.Subtract => "-"
      case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.StringConcat(l, r)         => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s) => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)            => s"LENGTH(${exprToSql(s)})"
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)        => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern) => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"
}
```

## Seamless Condition Composition

In Part 3, composing `SchemaExpr` and `Expr` values required explicit `.toExpr` calls because `SchemaExpr.&&` is a direct method that only accepts other `SchemaExpr` values. Part 4 adds **bridge implicit classes** that handle the cross-type case automatically:

```scala
// When SchemaExpr.&& receives an Expr argument, the direct method
// doesn't match (Expr is not SchemaExpr), so this implicit class kicks in:
implicit final class SchemaExprBooleanBridge[S](
  private val self: SchemaExpr[S, Boolean]
) extends AnyVal {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = ...
}

// And vice versa — Expr.&& can accept SchemaExpr directly:
implicit final class ExprBooleanOps[S](
  private val self: Expr[S, Boolean]
) extends AnyVal {
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = ...
}
```

Scala's implicit conversion resolution makes this work:

1. **`SchemaExpr && SchemaExpr`** — the built-in direct method matches, returns `SchemaExpr`
2. **`SchemaExpr && Expr`** — the direct method doesn't match, the bridge implicit class kicks in, returns `Expr`
3. **`Expr && SchemaExpr`** — the bridge overload on `ExprBooleanOps` matches, returns `Expr`
4. **`Expr && Expr`** — the standard `ExprBooleanOps.&&` overload matches, returns `Expr`

The result type automatically widens to `Expr` whenever an `Expr` value enters the chain. No `.toExpr` needed:

```scala
val condition =
  Product.price.between(10.0, 500.0) &&
  (Product.category === "Electronics") &&
  (Product.rating >= 4) &&
  Product.name.like("L%")
// condition: Expr[Product, Boolean] = And(
//   left = And(
//     left = And(
//       left = Between(
//         expr = Column(
//           LensImpl(
//             sources = Array(
//               Record(
//                 fields = Vector(
//                   Term(
//                     name = "name",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "price",
//                     value = Primitive(
//                       primitiveType = Double(None),
//                       typeId = Double,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "category",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
// ...

exprToSql(condition)
// res0: String = "((((price BETWEEN 10.0 AND 500.0) AND (category = 'Electronics')) AND (rating >= 4)) AND name LIKE 'L%')"
```

The first two `&&` calls stay in `SchemaExpr` land (direct method). The third `&&` encounters `between` (returns `Expr`), triggering the bridge. From that point on, everything is `Expr`.

## Schema-Driven Table Names

The `Table[S]` type associates a SQL table name with a schema type. Rather than hand-writing each table name, `Table.derived` extracts it from `Schema` metadata:

```scala
case class Table[S](name: String)

object Table {
  def derived[S](implicit schema: Schema[S]): Table[S] = Table(tableName(schema))
}
```

The `tableName` function checks the schema's modifiers for an explicit `sql.table_name` annotation. If none is found, it lowercases the case class name and applies English pluralization:

```scala
def tableName[S](schema: Schema[S]): String =
  schema.reflect.modifiers.collectFirst {
    case Modifier.config(key, value) if key == "sql.table_name" => value
  }.getOrElse(pluralize(schema.reflect.typeId.name.toLowerCase))
```

This gives you two ways to control the table name:

**Auto-pluralization (zero configuration):**

```scala
object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived
  val table: Table[Product] = Table.derived  // "Product" → "products"
}
```

**Explicit annotation (for compound names or irregular plurals):**

```scala
object OrderItem extends CompanionOptics[OrderItem] {
  implicit val schema: Schema[OrderItem] = Schema.derived
    .modifier(Modifier.config("sql.table_name", "order_items"))
  val table: Table[OrderItem] = Table.derived  // annotation → "order_items"
}
```

Without the annotation, `OrderItem` would become `orderitems`. The `Modifier.config("sql.table_name", "order_items")` annotation gives you precise control when the default doesn't match your database convention.

```scala
Product.table.name
// res1: String = "products"

OrderItem.table.name
// res2: String = "order_items"
```

You can still use the explicit constructor `Table[S]("my_table")` when you need a name that doesn't come from a schema at all.

`Table[S]` is the entry point for every statement builder. The type parameter `S` connects the table to its domain type, ensuring you can only use `Product` lenses in queries against `Product.table`.

## The SELECT Builder

The SELECT builder uses immutable case classes with `copy` for a fluent API:

```scala
sealed trait SortOrder
object SortOrder {
  case object Asc  extends SortOrder
  case object Desc extends SortOrder
}

case class SelectStmt[S](
  table: Table[S],
  columnList: List[String] = List("*"),
  whereExpr: Option[Expr[S, Boolean]] = None,
  orderByList: List[(String, SortOrder)] = Nil,
  limitCount: Option[Int] = None
) {
  def columns(optics: Optic[S, _]*): SelectStmt[S] =
    copy(columnList = optics.map(columnName).toList)
  def where(cond: Expr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
  def orderBy(optic: Optic[S, _], order: SortOrder = SortOrder.Asc): SelectStmt[S] =
    copy(orderByList = orderByList :+ (columnName(optic), order))
  def limit(n: Int): SelectStmt[S] =
    copy(limitCount = Some(n))
}

def select[S](table: Table[S]): SelectStmt[S] = SelectStmt(table)

def renderSelect[S](stmt: SelectStmt[S]): String = {
  val cols = stmt.columnList.mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  val orderBy = if (stmt.orderByList.isEmpty) "" else {
    val orders = stmt.orderByList.map { case (col, order) =>
      val dir = order match { case SortOrder.Asc => "ASC"; case SortOrder.Desc => "DESC" }
      s"$col $dir"
    }.mkString(", ")
    s" ORDER BY $orders"
  }
  val limit = stmt.limitCount.map(n => s" LIMIT $n").getOrElse("")
  s"SELECT $cols FROM ${stmt.table.name}$where$orderBy$limit"
}
```

Each builder method returns a new `SelectStmt` with the updated field. The `.where()` method is overloaded to accept both `Expr` and `SchemaExpr` — `SchemaExpr` values are translated via `fromSchemaExpr`, so pure `SchemaExpr` chains and mixed chains both work:

```scala
// Pure SchemaExpr conditions
val basicSelect = select(Product.table)
  .columns(Product.name, Product.price)
  .where(Product.inStock === true)
// basicSelect: SelectStmt[Product] = SelectStmt(
//   table = Table("products"),
//   columnList = List("name", "price"),
//   whereExpr = Some(
//     Relational(
//       left = Column(
//         LensImpl(
//           sources = Array(
//             Record(
//               fields = Vector(
//                 Term(
//                   name = "name",
//                   value = Primitive(
//                     primitiveType = String(None),
//                     typeId = String,
//                     primitiveBinding = Primitive(),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List(),
//                     storedDefaultValue = None,
//                     storedExamples = List()
//                   ),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List()
//                 ),
//                 Term(
//                   name = "price",
//                   value = Primitive(
//                     primitiveType = Double(None),
//                     typeId = Double,
//                     primitiveBinding = Primitive(),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List(),
//                     storedDefaultValue = None,
//                     storedExamples = List()
//                   ),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List()
//                 ),
//                 Term(
//                   name = "category",
//                   value = Primitive(
//                     primitiveType = String(None),
//                     typeId = String,
//                     primitiveBinding = Primitive(),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List(),
//                     storedDefaultValue = None,
//                     storedExamples = List()
//                   ),
// ...

renderSelect(basicSelect)
// res3: String = "SELECT name, price FROM products WHERE (inStock = TRUE)"

// Mixed conditions with ordering and limit
val advancedSelect = select(Product.table)
  .columns(Product.name, Product.price, Product.category)
  .where(
    Product.price.between(10.0, 500.0) &&
    (Product.category === "Electronics") &&
    (Product.rating >= 4)
  )
  .orderBy(Product.price, SortOrder.Desc)
  .limit(10)
// advancedSelect: SelectStmt[Product] = SelectStmt(
//   table = Table("products"),
//   columnList = List("name", "price", "category"),
//   whereExpr = Some(
//     And(
//       left = And(
//         left = Between(
//           expr = Column(
//             LensImpl(
//               sources = Array(
//                 Record(
//                   fields = Vector(
//                     Term(
//                       name = "name",
//                       value = Primitive(
//                         primitiveType = String(None),
//                         typeId = String,
//                         primitiveBinding = Primitive(),
//                         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                         modifiers = List(),
//                         storedDefaultValue = None,
//                         storedExamples = List()
//                       ),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List()
//                     ),
//                     Term(
//                       name = "price",
//                       value = Primitive(
//                         primitiveType = Double(None),
//                         typeId = Double,
//                         primitiveBinding = Primitive(),
//                         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                         modifiers = List(),
//                         storedDefaultValue = None,
//                         storedExamples = List()
//                       ),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List()
//                     ),
//                     Term(
//                       name = "category",
//                       value = Primitive(
//                         primitiveType = String(None),
//                         typeId = String,
//                         primitiveBinding = Primitive(),
//                         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                         modifiers = List(),
//                         storedDefaultValue = None,
// ...

renderSelect(advancedSelect)
// res4: String = "SELECT name, price, category FROM products WHERE (((price BETWEEN 10.0 AND 500.0) AND (category = 'Electronics')) AND (rating >= 4)) ORDER BY price DESC LIMIT 10"
```

## The UPDATE Builder

The UPDATE builder accumulates `.set()` calls, each pairing an optic with a value:

```scala
case class Assignment(column: String, value: String)

case class UpdateStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil,
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): UpdateStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
  def where(cond: Expr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

def update[S](table: Table[S]): UpdateStmt[S] = UpdateStmt(table)

def renderUpdate[S](stmt: UpdateStmt[S]): String = {
  val sets = stmt.assignments.map(a => s"${a.column} = ${a.value}").mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"UPDATE ${stmt.table.name} SET $sets$where"
}
```

The `.set()` method uses the optic to extract the column name and `sqlLiteral` (via the implicit `Schema[A]`) to render the value. The type parameter `A` on `set[A](optic: Optic[S, A], value: A)` ensures you cannot assign a `String` to a `Double` field.

```scala
val basicUpdate =
  update(Product.table)
    .set(Product.price, 9.99)
    .where(
      Product.price.between(10.0, 30.0) &&
        Product.name.like("M%") &&
        (Product.category === "Books") &&
        (Product.rating >= 4) &&
        (Product.inStock === true)
    )
// basicUpdate: UpdateStmt[Product] = UpdateStmt(
//   table = Table("products"),
//   assignments = List(Assignment(column = "price", value = "9.99")),
//   whereExpr = Some(
//     And(
//       left = And(
//         left = And(
//           left = And(
//             left = Between(
//               expr = Column(
//                 LensImpl(
//                   sources = Array(
//                     Record(
//                       fields = Vector(
//                         Term(
//                           name = "name",
//                           value = Primitive(
//                             primitiveType = String(None),
//                             typeId = String,
//                             primitiveBinding = Primitive(),
//                             doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                             modifiers = List(),
//                             storedDefaultValue = None,
//                             storedExamples = List()
//                           ),
//                           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                           modifiers = List()
//                         ),
//                         Term(
//                           name = "price",
//                           value = Primitive(
//                             primitiveType = Double(None),
//                             typeId = Double,
//                             primitiveBinding = Primitive(),
//                             doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                             modifiers = List(),
//                             storedDefaultValue = None,
//                             storedExamples = List()
//                           ),
//                           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                           modifiers = List()
//                         ),
//                         Term(
//                           name = "category",
//                           value = Primitive(
//                             primitiveType = String(None),
//                             typeId = String,
//                             primitiveBinding = Primitive(),
//                             doc = Doc(blocks = IndexedSeq(), metadata = Map()),
// ...

renderUpdate(basicUpdate)
// res5: String = "UPDATE products SET price = 9.99 WHERE (((((price BETWEEN 10.0 AND 30.0) AND name LIKE 'M%') AND (category = 'Books')) AND (rating >= 4)) AND (inStock = TRUE))"
```

Multiple `.set()` calls accumulate:

```scala
val multiUpdate =
  update(Product.table)
    .set(Product.price, 19.99)
    .set(Product.inStock, false)
    .where(Product.category === "Clearance")
// multiUpdate: UpdateStmt[Product] = UpdateStmt(
//   table = Table("products"),
//   assignments = List(
//     Assignment(column = "price", value = "19.99"),
//     Assignment(column = "inStock", value = "FALSE")
//   ),
//   whereExpr = Some(
//     Relational(
//       left = Column(
//         LensImpl(
//           sources = Array(
//             Record(
//               fields = Vector(
//                 Term(
//                   name = "name",
//                   value = Primitive(
//                     primitiveType = String(None),
//                     typeId = String,
//                     primitiveBinding = Primitive(),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List(),
//                     storedDefaultValue = None,
//                     storedExamples = List()
//                   ),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List()
//                 ),
//                 Term(
//                   name = "price",
//                   value = Primitive(
//                     primitiveType = Double(None),
//                     typeId = Double,
//                     primitiveBinding = Primitive(),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List(),
//                     storedDefaultValue = None,
//                     storedExamples = List()
//                   ),
//                   doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                   modifiers = List()
//                 ),
//                 Term(
//                   name = "category",
//                   value = Primitive(
//                     primitiveType = String(None),
//                     typeId = String,
//                     primitiveBinding = Primitive(),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List(),
// ...

renderUpdate(multiUpdate)
// res6: String = "UPDATE products SET price = 19.99, inStock = FALSE WHERE (category = 'Clearance')"
```

## INSERT and DELETE

The INSERT builder collects column-value pairs via `.set()`:

```scala
case class InsertStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): InsertStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
}

def insertInto[S](table: Table[S]): InsertStmt[S] = InsertStmt(table)

def renderInsert[S](stmt: InsertStmt[S]): String = {
  val cols = stmt.assignments.map(_.column).mkString(", ")
  val vals = stmt.assignments.map(_.value).mkString(", ")
  s"INSERT INTO ${stmt.table.name} ($cols) VALUES ($vals)"
}
```

The DELETE builder takes an optional WHERE clause:

```scala
case class DeleteStmt[S](
  table: Table[S],
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def where(cond: Expr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

def deleteFrom[S](table: Table[S]): DeleteStmt[S] = DeleteStmt(table)

def renderDelete[S](stmt: DeleteStmt[S]): String = {
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"DELETE FROM ${stmt.table.name}$where"
}
```

```scala
val ins = insertInto(Product.table)
  .set(Product.name, "Wireless Mouse")
  .set(Product.price, 29.99)
  .set(Product.category, "Electronics")
  .set(Product.inStock, true)
  .set(Product.rating, 4)
// ins: InsertStmt[Product] = InsertStmt(
//   table = Table("products"),
//   assignments = List(
//     Assignment(column = "name", value = "'Wireless Mouse'"),
//     Assignment(column = "price", value = "29.99"),
//     Assignment(column = "category", value = "'Electronics'"),
//     Assignment(column = "inStock", value = "TRUE"),
//     Assignment(column = "rating", value = "4")
//   )
// )

renderInsert(ins)
// res7: String = "INSERT INTO products (name, price, category, inStock, rating) VALUES ('Wireless Mouse', 29.99, 'Electronics', TRUE, 4)"

val del = deleteFrom(Product.table)
  .where(
    Product.price.between(0.0, 1.0) &&
    (Product.inStock === false)
  )
// del: DeleteStmt[Product] = DeleteStmt(
//   table = Table("products"),
//   whereExpr = Some(
//     And(
//       left = Between(
//         expr = Column(
//           LensImpl(
//             sources = Array(
//               Record(
//                 fields = Vector(
//                   Term(
//                     name = "name",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "price",
//                     value = Primitive(
//                       primitiveType = Double(None),
//                       typeId = Double,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
//                     doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                     modifiers = List()
//                   ),
//                   Term(
//                     name = "category",
//                     value = Primitive(
//                       primitiveType = String(None),
//                       typeId = String,
//                       primitiveBinding = Primitive(),
//                       doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                       modifiers = List(),
//                       storedDefaultValue = None,
//                       storedExamples = List()
//                     ),
// ...

renderDelete(del)
// res8: String = "DELETE FROM products WHERE ((price BETWEEN 0.0 AND 1.0) AND (inStock = FALSE))"
```

:::tip
For batch inserts, create one `InsertStmt` per row and render each separately. The builder pattern keeps each row type-safe.
:::

## Putting It Together

Here is a complete example combining schema-driven table names, bridge extensions, all four statement builders, and the renderers. The `Expr` ADT, extension methods, SQL rendering, and builder types are defined in `Common.scala` and `package.scala` — the usage code stays focused on building queries:

```scala

// --- Table reference with schema-driven names ---

case class Table[S](name: String)

object Table {
  def derived[S](implicit schema: Schema[S]): Table[S] = Table(tableName(schema))
}

def tableName[S](schema: Schema[S]): String =
  schema.reflect.modifiers.collectFirst {
    case Modifier.config(key, value) if key == "sql.table_name" => value
  }.getOrElse(pluralize(schema.reflect.typeId.name.toLowerCase))

def pluralize(word: String): String =
  if (word.endsWith("s") || word.endsWith("x") || word.endsWith("z") ||
      word.endsWith("ch") || word.endsWith("sh")) word + "es"
  else if (word.endsWith("y") && !word.endsWith("ay") && !word.endsWith("ey") &&
           !word.endsWith("oy") && !word.endsWith("uy")) word.dropRight(1) + "ies"
  else word + "s"

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val table: Table[Product] = Table.derived  // "Product" → "products"

  val name: Lens[Product, String]     = optic(_.name)
  val price: Lens[Product, Double]    = optic(_.price)
  val category: Lens[Product, String] = optic(_.category)
  val inStock: Lens[Product, Boolean] = optic(_.inStock)
  val rating: Lens[Product, Int]      = optic(_.rating)
}

case class OrderItem(
  orderId: Int,
  productId: Int,
  quantity: Int,
  unitPrice: Double
)

object OrderItem extends CompanionOptics[OrderItem] {
  implicit val schema: Schema[OrderItem] = Schema.derived
    .modifier(Modifier.config("sql.table_name", "order_items"))

  val table: Table[OrderItem] = Table.derived  // annotation → "order_items"

  val orderId: Lens[OrderItem, Int]      = optic(_.orderId)
  val productId: Lens[OrderItem, Int]    = optic(_.productId)
  val quantity: Lens[OrderItem, Int]     = optic(_.quantity)
  val unitPrice: Lens[OrderItem, Double] = optic(_.unitPrice)
}

// --- Expr ADT ---

sealed trait Expr[S, A]

object Expr {
  final case class Column[S, A](optic: Optic[S, A]) extends Expr[S, A]
  final case class Lit[S, A](value: A, schema: Schema[A]) extends Expr[S, A]

  final case class Relational[S, A](left: Expr[S, A], right: Expr[S, A], op: RelOp) extends Expr[S, Boolean]
  final case class And[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Or[S](left: Expr[S, Boolean], right: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Not[S](expr: Expr[S, Boolean]) extends Expr[S, Boolean]
  final case class Arithmetic[S, A](left: Expr[S, A], right: Expr[S, A], op: ArithOp) extends Expr[S, A]
  final case class StringConcat[S](left: Expr[S, String], right: Expr[S, String]) extends Expr[S, String]
  final case class StringRegexMatch[S](regex: Expr[S, String], string: Expr[S, String]) extends Expr[S, Boolean]
  final case class StringLength[S](string: Expr[S, String]) extends Expr[S, Int]

  final case class In[S, A](expr: Expr[S, A], values: List[A], schema: Schema[A]) extends Expr[S, Boolean]
  final case class Between[S, A](expr: Expr[S, A], low: A, high: A, schema: Schema[A]) extends Expr[S, Boolean]
  final case class IsNull[S, A](expr: Expr[S, A]) extends Expr[S, Boolean]
  final case class Like[S](expr: Expr[S, String], pattern: String) extends Expr[S, Boolean]

  def col[S, A](optic: Optic[S, A]): Expr[S, A] = Column(optic)
  def lit[S, A](value: A)(implicit schema: Schema[A]): Expr[S, A] = Lit(value, schema)

  def fromSchemaExpr[S, A](se: SchemaExpr[S, A]): Expr[S, A] = {
    val result = se match {
      case SchemaExpr.Optic(optic)      => Column(optic)
      case l: SchemaExpr.Literal[_, _]  => Lit(l.value, l.schema)
      case SchemaExpr.Relational(l, r, op) =>
        val relOp = op match {
          case SchemaExpr.RelationalOperator.Equal              => RelOp.Equal
          case SchemaExpr.RelationalOperator.NotEqual           => RelOp.NotEqual
          case SchemaExpr.RelationalOperator.LessThan           => RelOp.LessThan
          case SchemaExpr.RelationalOperator.LessThanOrEqual    => RelOp.LessThanOrEqual
          case SchemaExpr.RelationalOperator.GreaterThan        => RelOp.GreaterThan
          case SchemaExpr.RelationalOperator.GreaterThanOrEqual => RelOp.GreaterThanOrEqual
        }
        Relational(fromSchemaExpr(l), fromSchemaExpr(r), relOp)
      case SchemaExpr.Logical(l, r, op) => op match {
        case SchemaExpr.LogicalOperator.And => And(fromSchemaExpr(l), fromSchemaExpr(r))
        case SchemaExpr.LogicalOperator.Or  => Or(fromSchemaExpr(l), fromSchemaExpr(r))
      }
      case SchemaExpr.Not(inner) => Not(fromSchemaExpr(inner))
      case SchemaExpr.Arithmetic(l, r, op, _) =>
        val arithOp = op match {
          case SchemaExpr.ArithmeticOperator.Add      => ArithOp.Add
          case SchemaExpr.ArithmeticOperator.Subtract => ArithOp.Subtract
          case SchemaExpr.ArithmeticOperator.Multiply => ArithOp.Multiply
        }
        Arithmetic(fromSchemaExpr(l), fromSchemaExpr(r), arithOp)
      case SchemaExpr.StringConcat(l, r)              => StringConcat(fromSchemaExpr(l), fromSchemaExpr(r))
      case SchemaExpr.StringRegexMatch(regex, string) => StringRegexMatch(fromSchemaExpr(regex), fromSchemaExpr(string))
      case SchemaExpr.StringLength(string)            => StringLength(fromSchemaExpr(string))
    }
    result.asInstanceOf[Expr[S, A]]
  }
}

sealed trait RelOp
object RelOp {
  case object Equal              extends RelOp
  case object NotEqual           extends RelOp
  case object LessThan           extends RelOp
  case object LessThanOrEqual    extends RelOp
  case object GreaterThan        extends RelOp
  case object GreaterThanOrEqual extends RelOp
}

sealed trait ArithOp
object ArithOp {
  case object Add      extends ArithOp
  case object Subtract extends ArithOp
  case object Multiply extends ArithOp
}

// --- Extension methods with bridge ---

implicit final class OpticExprOps[S, A](private val optic: Optic[S, A]) {
  def in(values: A*)(implicit schema: Schema[A]): Expr[S, Boolean]           = Expr.In(Expr.col(optic), values.toList, schema)
  def between(low: A, high: A)(implicit schema: Schema[A]): Expr[S, Boolean] = Expr.Between(Expr.col(optic), low, high, schema)
  def isNull: Expr[S, Boolean]                   = Expr.IsNull(Expr.col(optic))
  def isNotNull: Expr[S, Boolean]                = Expr.Not(Expr.IsNull(Expr.col(optic)))
}

implicit final class StringOpticExprOps[S](private val optic: Optic[S, String]) {
  def like(pattern: String): Expr[S, Boolean] = Expr.Like(Expr.col(optic), pattern)
}

implicit final class ExprBooleanOps[S](private val self: Expr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.And(self, other)
  def &&(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.And(self, Expr.fromSchemaExpr(other))
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean]      = Expr.Or(self, other)
  def ||(other: SchemaExpr[S, Boolean]): Expr[S, Boolean] = Expr.Or(self, Expr.fromSchemaExpr(other))
  def unary_! : Expr[S, Boolean]                          = Expr.Not(self)
}

implicit final class SchemaExprBooleanBridge[S](private val self: SchemaExpr[S, Boolean]) {
  def &&(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.And(Expr.fromSchemaExpr(self), other)
  def ||(other: Expr[S, Boolean]): Expr[S, Boolean] = Expr.Or(Expr.fromSchemaExpr(self), other)
  def toExpr: Expr[S, Boolean] = Expr.fromSchemaExpr(self)
}

// --- SQL rendering ---

def columnName(optic: zio.blocks.schema.Optic[_, _]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral[A](value: A, schema: Schema[A]): String = {
  val dv = schema.toDynamicValue(value)
  dv match {
    case p: DynamicValue.Primitive => p.value match {
      case _: PrimitiveValue.String  => s"'${value.toString.replace("'", "''")}'"
      case b: PrimitiveValue.Boolean => if (b.value) "TRUE" else "FALSE"
      case _                         => value.toString
    }
    case _ => value.toString
  }
}

def exprToSql[S, A](expr: Expr[S, A]): String = expr match {
  case Expr.Column(optic)      => columnName(optic)
  case Expr.Lit(value, schema) => sqlLiteral(value, schema)
  case Expr.Relational(left, right, op) =>
    val sqlOp = op match {
      case RelOp.Equal              => "="
      case RelOp.NotEqual           => "<>"
      case RelOp.LessThan           => "<"
      case RelOp.LessThanOrEqual    => "<="
      case RelOp.GreaterThan        => ">"
      case RelOp.GreaterThanOrEqual => ">="
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.And(l, r) => s"(${exprToSql(l)} AND ${exprToSql(r)})"
  case Expr.Or(l, r)  => s"(${exprToSql(l)} OR ${exprToSql(r)})"
  case Expr.Not(e)    => s"NOT (${exprToSql(e)})"
  case Expr.Arithmetic(left, right, op) =>
    val sqlOp = op match {
      case ArithOp.Add      => "+"
      case ArithOp.Subtract => "-"
      case ArithOp.Multiply => "*"
    }
    s"(${exprToSql(left)} $sqlOp ${exprToSql(right)})"
  case Expr.StringConcat(l, r)         => s"CONCAT(${exprToSql(l)}, ${exprToSql(r)})"
  case Expr.StringRegexMatch(regex, s) => s"(${exprToSql(s)} LIKE ${exprToSql(regex)})"
  case Expr.StringLength(s)            => s"LENGTH(${exprToSql(s)})"
  case Expr.In(e, values, schema) =>
    s"${exprToSql(e)} IN (${values.map(v => sqlLiteral(v, schema)).mkString(", ")})"
  case Expr.Between(e, low, high, schema) =>
    s"(${exprToSql(e)} BETWEEN ${sqlLiteral(low, schema)} AND ${sqlLiteral(high, schema)})"
  case Expr.IsNull(e)        => s"${exprToSql(e)} IS NULL"
  case Expr.Like(e, pattern) => s"${exprToSql(e)} LIKE '${pattern.replace("'", "''")}'"
}

// --- Statement builders ---

sealed trait SortOrder
object SortOrder {
  case object Asc  extends SortOrder
  case object Desc extends SortOrder
}

case class Assignment(column: String, value: String)

case class SelectStmt[S](
  table: Table[S],
  columnList: List[String] = List("*"),
  whereExpr: Option[Expr[S, Boolean]] = None,
  orderByList: List[(String, SortOrder)] = Nil,
  limitCount: Option[Int] = None
) {
  def columns(optics: Optic[S, _]*): SelectStmt[S] =
    copy(columnList = optics.map(columnName).toList)
  def where(cond: Expr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): SelectStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
  def orderBy(optic: Optic[S, _], order: SortOrder = SortOrder.Asc): SelectStmt[S] =
    copy(orderByList = orderByList :+ (columnName(optic), order))
  def limit(n: Int): SelectStmt[S] =
    copy(limitCount = Some(n))
}

case class UpdateStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil,
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): UpdateStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
  def where(cond: Expr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): UpdateStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

case class InsertStmt[S](
  table: Table[S],
  assignments: List[Assignment] = Nil
) {
  def set[A](optic: Optic[S, A], value: A)(implicit schema: Schema[A]): InsertStmt[S] =
    copy(assignments = assignments :+ Assignment(columnName(optic), sqlLiteral(value, schema)))
}

case class DeleteStmt[S](
  table: Table[S],
  whereExpr: Option[Expr[S, Boolean]] = None
) {
  def where(cond: Expr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(cond))
  def where(cond: SchemaExpr[S, Boolean]): DeleteStmt[S] =
    copy(whereExpr = Some(Expr.fromSchemaExpr(cond)))
}

def select[S](table: Table[S]): SelectStmt[S]       = SelectStmt(table)
def update[S](table: Table[S]): UpdateStmt[S]       = UpdateStmt(table)
def insertInto[S](table: Table[S]): InsertStmt[S]   = InsertStmt(table)
def deleteFrom[S](table: Table[S]): DeleteStmt[S]   = DeleteStmt(table)

// --- Renderers ---

def renderSelect[S](stmt: SelectStmt[S]): String = {
  val cols = stmt.columnList.mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  val orderBy = if (stmt.orderByList.isEmpty) "" else {
    val orders = stmt.orderByList.map { case (col, order) =>
      val dir = order match { case SortOrder.Asc => "ASC"; case SortOrder.Desc => "DESC" }
      s"$col $dir"
    }.mkString(", ")
    s" ORDER BY $orders"
  }
  val limit = stmt.limitCount.map(n => s" LIMIT $n").getOrElse("")
  s"SELECT $cols FROM ${stmt.table.name}$where$orderBy$limit"
}

def renderUpdate[S](stmt: UpdateStmt[S]): String = {
  val sets = stmt.assignments.map(a => s"${a.column} = ${a.value}").mkString(", ")
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"UPDATE ${stmt.table.name} SET $sets$where"
}

def renderInsert[S](stmt: InsertStmt[S]): String = {
  val cols = stmt.assignments.map(_.column).mkString(", ")
  val vals = stmt.assignments.map(_.value).mkString(", ")
  s"INSERT INTO ${stmt.table.name} ($cols) VALUES ($vals)"
}

def renderDelete[S](stmt: DeleteStmt[S]): String = {
  val where = stmt.whereExpr.map(c => s" WHERE ${exprToSql(c)}").getOrElse("")
  s"DELETE FROM ${stmt.table.name}$where"
}

// --- Usage ---

// Schema-driven table names
println(Product.table.name)    // "products" (auto-pluralized)
println(OrderItem.table.name)  // "order_items" (from annotation)

// SELECT with mixed conditions
val q = select(Product.table)
  .columns(Product.name, Product.price, Product.category)
  .where(
    Product.category.in("Electronics", "Books") &&
    Product.price.between(10.0, 500.0) &&
    (Product.rating >= 4).toExpr
  )
  .orderBy(Product.price, SortOrder.Desc)
  .limit(20)

println(renderSelect(q))

// UPDATE with seamless condition mixing
val u = update(Product.table)
  .set(Product.price, 9.99)
  .where(
    Product.price.between(10.0, 30.0) &&
      Product.name.like("M%") &&
      (Product.category === "Books") &&
      (Product.rating >= 4) &&
      (Product.inStock === true)
  )

println(renderUpdate(u))

// INSERT
val i = insertInto(Product.table)
  .set(Product.name, "Wireless Mouse")
  .set(Product.price, 29.99)
  .set(Product.category, "Electronics")
  .set(Product.inStock, true)
  .set(Product.rating, 4)

println(renderInsert(i))

// DELETE
val d = deleteFrom(Product.table)
  .where(Product.price.between(0.0, 1.0) && (Product.inStock === false))

println(renderDelete(d))

// Cross-table query with annotated table name
val orderQuery = select(OrderItem.table)
  .columns(OrderItem.orderId, OrderItem.productId, OrderItem.quantity)
  .where(OrderItem.quantity >= 3)
  .orderBy(OrderItem.unitPrice, SortOrder.Desc)

println(renderSelect(orderQuery))
```

## Going Further

- **[Part 1: Expressions](./query-dsl-reified-optics.md)** — Building query expressions with reified optics
- **[Part 2: SQL Generation](./query-dsl-sql.md)** — Translating built-in expressions to SQL
- **[Part 3: Extending the Expression Language](./query-dsl-extending.md)** — Adding custom operators beyond SchemaExpr
- **[SchemaExpr Reference](../reference/schema-expr.md)** — Full API coverage of expression types
- **[Optics Reference](../reference/optics.md)** — Lens, Prism, Optional, and Traversal

The builder pattern shown here extends naturally to JOIN clauses (using lenses from multiple table types), subqueries (nesting `SelectStmt` in WHERE conditions), and parameterized queries (collecting `?` placeholders and parameter values during rendering). Each of these builds on the same foundation: optics for column names, `Expr` for conditions, and immutable builders for statement structure.

---

## Query DSL with Reified Optics — Part 1: Expressions

In this guide, we will build a type-safe query DSL for filtering, comparing, and computing over domain data using ZIO Blocks' reified optics and schema expressions. By the end, you will have a composable query language that works on any schema-equipped data type, supporting equality checks, comparisons, boolean logic, arithmetic, and string operations.

We'll take an incremental approach: starting with simple field-level equality checks, then adding comparison operators, boolean combinators, arithmetic expressions, and string operations until we have a complete, expressive query DSL.

**What we'll cover:**

- Defining domain types with schemas and optics
- Building equality and comparison queries using `===`, `>`, `<`, `>=`, `<=`
- Combining queries with `&&` and `||`
- Using arithmetic operators (`+`, `-`, `*`) in expressions
- Working with string operations (`matches`, `concat`, `length`)
- Querying through nested structures and collections
- Evaluating queries against data

## The Problem

When you need to query or filter collections of structured data in Scala, you typically write ad-hoc predicate functions:

```scala
case class Product(name: String, price: Double, category: String, inStock: Boolean)

val products: List[Product] = loadProducts()

// Filtering with ad-hoc predicates
val results = products.filter(p =>
  p.category == "Electronics" && p.price < 500.0 && p.inStock
)
```

This works, but the predicate `p => p.category == "Electronics" && p.price < 500.0 && p.inStock` is an opaque function. You cannot inspect it, serialize it, translate it to SQL, send it to a remote service, or optimize it. It is a black box.

If you need to build a query builder for a database, a filter language for an API, or a rule engine, you need queries as **data** -- inspectable, composable, serializable expression trees. Building these by hand means defining an AST, writing an evaluator, and maintaining type safety across all operators -- a significant amount of boilerplate for every domain type.

In this guide, we'll solve this by using ZIO Blocks' `SchemaExpr` and reified optics, which give us composable, type-safe query expressions for free, derived directly from your data model's schema.

## Prerequisites

Add the ZIO Blocks Schema dependency:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

```scala

```

This guide assumes familiarity with ZIO Blocks schemas and basic optics. See the [Schema](../reference/schema.md) and [Optics](../reference/optics.md) reference pages for background.

## Defining Your Domain

We'll build a product catalog query DSL. First, define the domain types with schemas and optics:

```scala
case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}
```

Three things are required for each queryable type:

1. **`Schema.derived`** -- captures the type's structure at runtime
2. **`CompanionOptics[T]`** -- provides the `optic` macro for deriving lenses
3. **Named lenses** for each field you want to query on

Each lens is a *reified* field accessor: unlike `_.price`, the lens `Product.price` is a first-class value that carries the field name, the source schema, and the focus schema. This metadata is what makes the query DSL possible.

## Equality and Comparison Queries

Every optic in ZIO Blocks has built-in operators that create `SchemaExpr` values -- expression trees representing queries:

```scala
// Equality check
val isElectronics: SchemaExpr[Product, Boolean] =
  Product.category === "Electronics"

// Greater-than comparison
val expensiveItems: SchemaExpr[Product, Boolean] =
  Product.price > 100.0

// Less-than-or-equal
val budgetFriendly: SchemaExpr[Product, Boolean] =
  Product.price <= 50.0

// Comparison against a literal
val highRated: SchemaExpr[Product, Boolean] =
  Product.rating >= 4
```

The full set of comparison operators:

| Operator | Meaning                |
|----------|------------------------|
| `===`    | Equal to               |
| `!=`     | Not equal to           |
| `>`      | Greater than           |
| `>=`     | Greater than or equal  |
| `<`      | Less than              |
| `<=`     | Less than or equal     |

Each operator works in two forms:
- **Optic vs. literal**: `Product.price > 100.0` -- compare a field to a value
- **Optic vs. optic**: `Product.rating === Product.rating` -- compare two fields

## Evaluating Queries

A `SchemaExpr[A, Boolean]` is a predicate over `A`. Evaluate it with `.eval`:

```scala
val laptop = Product("Laptop", 999.99, "Electronics", true, 5)
val pen    = Product("Pen", 2.50, "Office", true, 3)
```

```scala
isElectronics.eval(laptop)
// res0: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
isElectronics.eval(pen)
// res1: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))

expensiveItems.eval(laptop)
// res2: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
expensiveItems.eval(pen)
// res3: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

The result type is `Either[OpticCheck, Seq[Boolean]]`:
- `Right(Seq(true))` or `Right(Seq(false))` for successful evaluation
- `Left(opticCheck)` if the optic path fails (e.g., a prism encounters the wrong variant case)

For lenses, evaluation always succeeds because lenses always resolve.

## Combining Queries with Boolean Logic

Combine queries with `&&` (and), `||` (or), and `!` (not):

```scala
// AND: electronics under $500
val affordableElectronics: SchemaExpr[Product, Boolean] =
  (Product.category === "Electronics") && (Product.price < 500.0)

// OR: either cheap or highly rated
val goodDeal: SchemaExpr[Product, Boolean] =
  (Product.price < 10.0) || (Product.rating >= 5)

// NOT: items that are out of stock
val outOfStock: SchemaExpr[Product, Boolean] =
  !Product.inStock
```

```scala
affordableElectronics.eval(laptop)
// res4: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
goodDeal.eval(laptop)
// res5: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
goodDeal.eval(pen)
// res6: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
outOfStock.eval(laptop)
// res7: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

Boolean combinators also work on `SchemaExpr` values, not just optics. This means you can build complex compound queries:

```scala
val complexQuery: SchemaExpr[Product, Boolean] =
  ((Product.category === "Electronics") && (Product.price < 500.0)) ||
  ((Product.category === "Office") && (Product.rating >= 4))
```

```scala
complexQuery.eval(laptop)
// res8: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
complexQuery.eval(pen)
// res9: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))
```

## Arithmetic Expressions

Optics on numeric fields support `+`, `-`, and `*`:

```scala
val discountedPrice: SchemaExpr[Product, Double] =
  Product.price * 0.9

val priceWithTax: SchemaExpr[Product, Double] =
  Product.price * 1.08
```

```scala
discountedPrice.eval(laptop)
// res10: Either[OpticCheck, Seq[Double]] = Right(IndexedSeq(899.991))
priceWithTax.eval(pen)
// res11: Either[OpticCheck, Seq[Double]] = Right(IndexedSeq(2.7))
```

Arithmetic operators are available for all numeric types: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`.

## String Operations

Optics on `String` fields provide string-specific operations:

```scala
// Regex matching
val startsWithL: SchemaExpr[Product, Boolean] =
  Product.name.matches("L.*")

// String concatenation
val labeledName: SchemaExpr[Product, String] =
  Product.name.concat(" [SALE]")

// String length
val nameLength: SchemaExpr[Product, Int] =
  Product.name.length
```

```scala
startsWithL.eval(laptop)
// res12: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
startsWithL.eval(pen)
// res13: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(false))

labeledName.eval(laptop)
// res14: Either[OpticCheck, Seq[String]] = Right(IndexedSeq("Laptop [SALE]"))

nameLength.eval(laptop)
// res15: Either[OpticCheck, Seq[Int]] = Right(IndexedSeq(6))
nameLength.eval(pen)
// res16: Either[OpticCheck, Seq[Int]] = Right(IndexedSeq(3))
```

| String Operation | Signature                          | Description               |
|------------------|------------------------------------|---------------------------|
| `matches`        | `(regex: String) => SchemaExpr[S, Boolean]` | Regex match        |
| `concat`         | `(suffix: String) => SchemaExpr[S, String]` | Append a string    |
| `length`         | `SchemaExpr[S, Int]`               | String length             |

## Dynamic Evaluation

Every `SchemaExpr` can also evaluate to `DynamicValue`, which is useful when you need format-agnostic results (e.g., serializing query results to JSON):

```scala
val priceExpr: SchemaExpr[Product, Double] = Product.price * 0.9
```

```scala
priceExpr.evalDynamic(laptop)
// res17: Either[OpticCheck, Seq[DynamicValue]] = Right(
//   IndexedSeq(Primitive(Double(899.991)))
// )
```

The `evalDynamic` method converts results to `DynamicValue` representations, enabling integration with serialization formats without knowing the concrete type.

## Querying Nested Structures

The real power of reified optics emerges with nested data. Define a richer domain:

```scala

case class Address(city: String, country: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Seller(name: String, address: Address, rating: Double)
object Seller extends CompanionOptics[Seller] {
  implicit val schema: Schema[Seller] = Schema.derived

  val name: Lens[Seller, String]       = optic(_.name)
  val rating: Lens[Seller, Double]     = optic(_.rating)
  // Compose through nested structure directly
  val city: Lens[Seller, String]       = optic(_.address.city)
  val country: Lens[Seller, String]    = optic(_.address.country)
}
```

The `optic(_.address.city)` macro composes a lens from `Seller` to `Address` with a lens from `Address` to `String`, producing a single `Lens[Seller, String]`. Now we can query nested fields as if they were top-level:

```scala
val localSeller: SchemaExpr[Seller, Boolean] =
  (Seller.city === "Berlin") && (Seller.rating >= 4.0)

val seller = Seller("TechShop", Address("Berlin", "Germany"), 4.5)
```

```scala
localSeller.eval(seller)
// res19: Either[OpticCheck, Seq[Boolean]] = Right(IndexedSeq(true))
```

## Querying Through Collections

For collection fields, use traversals to query across all elements:

```scala

case class LineItem(sku: String, price: Double, quantity: Int)
object LineItem {
  implicit val schema: Schema[LineItem] = Schema.derived
}

case class Order(id: String, items: List[LineItem])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived

  val id: Lens[Order, String]                  = optic(_.id)
  val allPrices: Traversal[Order, Double]      = optic(_.items.each.price)
  val allSkus: Traversal[Order, String]        = optic(_.items.each.sku)
  val allQuantities: Traversal[Order, Int]     = optic(_.items.each.quantity)
}
```

Traversals produce `SchemaExpr` values that evaluate to **multiple results**:

```scala
val order = Order("ORD-1", List(
  LineItem("SKU-A", 29.99, 2),
  LineItem("SKU-B", 149.99, 1),
  LineItem("SKU-C", 9.99, 5)
))

// Each element is evaluated independently
val hasExpensiveItem: SchemaExpr[Order, Boolean] =
  Order.allPrices > 100.0
```

```scala
hasExpensiveItem.eval(order)
// res21: Either[OpticCheck, Seq[Boolean]] = Right(List(false, true, false))
```

:::tip
When a traversal-based expression produces multiple results, each element in the sequence corresponds to one focused value from the traversal. This makes it straightforward to check whether *any* or *all* elements satisfy a condition by examining the result sequence.
:::

## Filtering a Collection

With `SchemaExpr` as a reified predicate, you can build a generic filter function that works with any schema-equipped type:

```scala

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}

def filter[A](items: List[A], predicate: SchemaExpr[A, Boolean]): List[A] =
  items.filter(item =>
    predicate.eval(item) match {
      case Right(results) => results.forall(_ == true)
      case Left(_)        => false
    }
  )

val catalog = List(
  Product("Laptop", 999.99, "Electronics", true, 5),
  Product("Mouse", 29.99, "Electronics", true, 4),
  Product("Pen", 2.50, "Office", true, 3),
  Product("Monitor", 349.99, "Electronics", false, 5),
  Product("Notebook", 5.99, "Office", true, 4)
)

val query = (Product.category === "Electronics") && (Product.inStock === true) && (Product.price < 500.0)
```

```scala
filter(catalog, query).map(_.name)
// res23: List[String] = List("Mouse")
```

The `filter` function knows nothing about `Product` -- it works with any `SchemaExpr[A, Boolean]`. The query is data, not a lambda, so it could be serialized, logged, or translated to a database query.

## Putting It Together

Here is a complete, self-contained example combining all the techniques from this guide:

```scala

// --- Domain ---

case class Address(city: String, country: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int,
  warehouse: Address
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
  val city: Lens[Product, String]      = optic(_.warehouse.city)
}

// --- Generic query filter ---

def filter[A](items: List[A], predicate: SchemaExpr[A, Boolean]): List[A] =
  items.filter(item =>
    predicate.eval(item) match {
      case Right(results) => results.forall(_ == true)
      case Left(_)        => false
    }
  )

// --- Usage ---

val catalog = List(
  Product("Laptop", 999.99, "Electronics", true, 5, Address("Berlin", "Germany")),
  Product("Mouse", 29.99, "Electronics", true, 4, Address("Berlin", "Germany")),
  Product("Pen", 2.50, "Office", true, 3, Address("London", "UK")),
  Product("Monitor", 349.99, "Electronics", false, 5, Address("Berlin", "Germany")),
  Product("Notebook", 5.99, "Office", true, 4, Address("London", "UK"))
)

// Compose a query: in-stock electronics under $500, from Berlin, highly rated
val query =
  (Product.category === "Electronics") &&
  (Product.inStock === true) &&
  (Product.price < 500.0) &&
  (Product.city === "Berlin") &&
  (Product.rating >= 4)

val results = filter(catalog, query)
// results: List(Product("Mouse", 29.99, "Electronics", true, 4, Address("Berlin", "Germany")))

// String operations
val searchQuery = Product.name.matches(".*top$")
val matches = filter(catalog, searchQuery)
// matches: List(Product("Laptop", ...))

// Arithmetic: compute discounted prices
val discounted = Product.price * 0.8
catalog.foreach { p =>
  println(s"${Product.name.get(p)}: ${discounted.eval(p)}")
}
```

## Going Further

- **[Part 2: SQL Generation](./query-dsl-sql.md)** -- Translating query expressions to SQL
- **[Part 3: Extending the Expression Language](./query-dsl-extending.md)** -- Adding custom operators (IN, BETWEEN, aggregates) beyond SchemaExpr
- **[Part 4: A Fluent SQL Builder](./query-dsl-fluent-builder.md)** -- Type-safe SELECT, UPDATE, INSERT, DELETE with seamless condition mixing
- **[Optics Reference](../reference/optics.md)** -- Full API coverage of Lens, Prism, Optional, and Traversal
- **[DynamicOptic Reference](../reference/dynamic-optic.md)** -- Runtime optic paths for programmatic query construction
- **[Schema Reference](../reference/schema.md)** -- Schema derivation and type-level metadata
- **[Path Interpolator](../path-interpolator.md)** -- String-based path construction with `p"..."` syntax

The `SchemaExpr` expression tree is a sealed trait, making it straightforward to write interpreters that translate queries to SQL, MongoDB filters, Elasticsearch queries, or any other target language. Because each optic carries its `DynamicOptic` path (via `toDynamic`), you can extract field names and paths programmatically for these translations.

---

## Query DSL with Reified Optics — Part 2: SQL Generation

In this guide, we will build a SQL query generator that translates ZIO Blocks' `SchemaExpr` expression trees into SQL WHERE clauses, SELECT statements, and parameterized queries. By the end, you will have an interpreter that takes any `SchemaExpr`-based query and produces executable SQL, covering comparisons, boolean logic, arithmetic, string operations, nested structures, and safe parameterization.

This is Part 2 of the Query DSL series. [Part 1](./query-dsl-reified-optics.md) covered building query expressions with reified optics. Here, we interpret those expressions as SQL.

**What we'll cover:**

- Interpreting `SchemaExpr` as a sealed AST via pattern matching
- Extracting column names from optic paths using `DynamicOptic`
- Translating relational, logical, arithmetic, and string operations to SQL
- Building complete `SELECT ... FROM ... WHERE ...` statements
- Generating parameterized queries for SQL injection safety
- Handling nested structures with table-qualified column names

## The Problem

In Part 1, we built composable query expressions as data -- `SchemaExpr` values that can be inspected, combined, and evaluated in-memory. But in real applications, data lives in databases. You need to translate those same queries into SQL.

The naive approach is to write SQL strings by hand for every query:

```scala
// Manual SQL for each query variant
def findProducts(category: Option[String], maxPrice: Option[Double], inStock: Option[Boolean]): String = {
  val conditions = List.newBuilder[String]
  category.foreach(c => conditions += s"category = '$c'")      // SQL injection!
  maxPrice.foreach(p => conditions += s"price < $p")
  inStock.foreach(s => conditions += s"in_stock = $s")
  val where = conditions.result().mkString(" AND ")
  s"SELECT * FROM products" + (if (where.nonEmpty) s" WHERE $where" else "")
}
```

This is fragile, repetitive, and vulnerable to SQL injection. Every new query shape requires new string-building code. The query logic is duplicated -- once as a `SchemaExpr` for in-memory filtering, and again as hand-written SQL for the database.

Since `SchemaExpr` is a sealed trait, we can write a single interpreter that translates *any* query expression into SQL. Write the interpreter once, and every query you build with the Part 1 DSL automatically gets a SQL translation.

## Prerequisites

This guide builds on [Part 1: Expressions](./query-dsl-reified-optics.md). You should be comfortable building `SchemaExpr` values with optic operators (`===`, `>`, `&&`, etc.).

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

```scala

```

## Domain Setup

We reuse the product catalog domain from Part 1:

```scala
case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}
```

## The SchemaExpr AST

Before we build the interpreter, let's understand the structure we are interpreting. `SchemaExpr` is a sealed trait with these cases:

```
SchemaExpr[A, B]
├── Literal[S, A](value, schema)                              -- a constant value
├── Optic[A, B](optic)                                        -- a field reference
├── StringRegexMatch[A](regex, string)                       -- regex pattern matching
├── StringLength[A](string)                                  -- string length calculation
├── UnaryOp[A, B]                                            -- abstract trait for unary operations
│   └── Not[A](expr)                                         -- boolean negation
└── BinaryOp[A, B, C]                                        -- abstract trait for binary operations
    ├── Relational[A, B](left, right, operator)              -- comparison operations
    ├── Logical[A](left, right, operator)                    -- boolean operations
    ├── Arithmetic[S, A](left, right, operator, isNumeric)   -- numeric operations
    └── StringConcat[A](left, right)                         -- string concatenation

RelationalOperator
├── LessThan
├── GreaterThan
├── LessThanOrEqual
├── GreaterThanOrEqual
├── Equal
└── NotEqual

LogicalOperator
├── And
└── Or

ArithmeticOperator
├── Add
├── Subtract
└── Multiply
```

Each case carries enough information to produce SQL: `Optic` nodes carry field paths, `Literal` nodes carry values, and operator nodes carry the operation type. Our interpreter walks this tree and emits SQL fragments.

## Extracting Column Names from Optics

The first challenge is turning a reified optic into a SQL column name. Every `Optic[S, A]` has a `toDynamic` method that returns a `DynamicOptic` -- a sequence of path nodes. For a lens like `Product.price`, the path is `[Field("price")]`. We extract the field name from the last `Field` node:

```scala
def columnName(optic: zio.blocks.schema.Optic[?, ?]): String = {
  val nodes = optic.toDynamic.nodes
  nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")
}
```

This converts the optic path to a column name. For a simple field like `Product.price`, it produces `"price"`. For a nested path, it joins field names with underscores (we will refine this for table-qualified names later).

```scala
columnName(Product.price)
// res0: String = "price"
columnName(Product.name)
// res1: String = "name"
columnName(Product.category)
// res2: String = "category"
```

## Translating Literals to SQL

Literal values need proper SQL formatting -- strings must be quoted, booleans converted to SQL syntax:

```scala
def sqlLiteral(value: Any): String = value match {
  case s: String     => s"'${s.replace("'", "''")}'"
  case b: Boolean    => if (b) "TRUE" else "FALSE"
  case n: Number     => n.toString
  case other         => other.toString
}
```

## Building the SQL Interpreter

Now we build the core interpreter. It pattern-matches on each `SchemaExpr` case and produces a SQL string:

```scala
def toSql[A, B](expr: SchemaExpr[A, B]): String = expr match {

  // Field reference → column name
  case SchemaExpr.Optic(optic) =>
    columnName(optic)

  // Constant value → SQL literal
  case SchemaExpr.Literal(value, _) =>
    sqlLiteral(value)

  // Comparison operators → SQL relational operators
  case SchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case SchemaExpr.RelationalOperator.Equal              => "="
      case SchemaExpr.RelationalOperator.NotEqual           => "<>"
      case SchemaExpr.RelationalOperator.LessThan           => "<"
      case SchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case SchemaExpr.RelationalOperator.GreaterThan        => ">"
      case SchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSql(left)} $sqlOp ${toSql(right)})"

  // Boolean operators → AND / OR
  case SchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case SchemaExpr.LogicalOperator.And => "AND"
      case SchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSql(left)} $sqlOp ${toSql(right)})"

  // Negation → NOT
  case SchemaExpr.Not(inner) =>
    s"NOT (${toSql(inner)})"

  // Arithmetic → SQL math operators
  case SchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case SchemaExpr.ArithmeticOperator.Add      => "+"
      case SchemaExpr.ArithmeticOperator.Subtract => "-"
      case SchemaExpr.ArithmeticOperator.Multiply => "*"
    }
    s"(${toSql(left)} $sqlOp ${toSql(right)})"

  // String concatenation → CONCAT()
  case SchemaExpr.StringConcat(left, right) =>
    s"CONCAT(${toSql(left)}, ${toSql(right)})"

  // Regex match → column LIKE pattern (simplified)
  case SchemaExpr.StringRegexMatch(regex, string) =>
    s"(${toSql(string)} LIKE ${toSql(regex)})"

  // String length → LENGTH()
  case SchemaExpr.StringLength(string) =>
    s"LENGTH(${toSql(string)})"
}
```

The mapping from `SchemaExpr` to SQL is direct:

| SchemaExpr Case     | SQL Output                           |
|---------------------|--------------------------------------|
| `Optic(optic)`      | Column name from `toDynamic`         |
| `Literal(v, _)`     | SQL literal (`'text'`, `42`, `TRUE`) |
| `Relational(_, _, op)` | `=`, `<>`, `<`, `>`, `<=`, `>=`   |
| `Logical(_, _, op)` | `AND`, `OR`                          |
| `Not(expr)`         | `NOT (...)`                          |
| `Arithmetic(_, _, op, _)` | `+`, `-`, `*`                  |
| `StringConcat`      | `CONCAT(a, b)`                       |
| `StringRegexMatch`  | `LIKE` (pattern matching)            |
| `StringLength`      | `LENGTH(col)`                        |

## Generating SQL from Queries

Now we can translate any query expression into a SQL WHERE clause. Let's try it with the queries from Part 1:

```scala
val isElectronics = Product.category === "Electronics"
val expensiveItems = Product.price > 100.0
val highRated = Product.rating >= 4
```

```scala
toSql(isElectronics)
// res3: String = "(category = 'Electronics')"
toSql(expensiveItems)
// res4: String = "(price > 100.0)"
toSql(highRated)
// res5: String = "(rating >= 4)"
```

## Compound Queries

Boolean combinators translate to `AND`, `OR`, and `NOT`:

```scala
val affordableElectronics =
  (Product.category === "Electronics") && (Product.price < 500.0)

val goodDeal =
  (Product.price < 10.0) || (Product.rating >= 5)

val outOfStock = !Product.inStock
```

```scala
toSql(affordableElectronics)
// res6: String = "((category = 'Electronics') AND (price < 500.0))"
toSql(goodDeal)
// res7: String = "((price < 10.0) OR (rating >= 5))"
toSql(outOfStock)
// res8: String = "NOT (inStock)"
```

Complex nested queries compose naturally:

```scala
val complexQuery =
  ((Product.category === "Electronics") && (Product.price < 500.0)) ||
  ((Product.category === "Office") && (Product.rating >= 4))
```

```scala
toSql(complexQuery)
// res9: String = "(((category = 'Electronics') AND (price < 500.0)) OR ((category = 'Office') AND (rating >= 4)))"
```

## Arithmetic in SQL

Arithmetic expressions translate directly to SQL math:

```scala
val discountedPrice = Product.price * 0.9
val priceWithTax = Product.price * 1.08
```

```scala
toSql(discountedPrice)
// res10: String = "(price * 0.9)"
toSql(priceWithTax)
// res11: String = "(price * 1.08)"
```

## String Operations in SQL

String operations map to SQL string functions:

```scala
// Regex match → LIKE
val startsWithL = Product.name.matches("L%")

// Concatenation → CONCAT()
val labeledName = Product.name.concat(" [SALE]")

// String length → LENGTH()
val nameLength = Product.name.length
```

```scala
toSql(startsWithL)
// res12: String = "(name LIKE 'L%')"
toSql(labeledName)
// res13: String = "CONCAT(name, ' [SALE]')"
toSql(nameLength)
// res14: String = "LENGTH(name)"
```

:::tip
The `matches` operator uses regex syntax in the `SchemaExpr` evaluator, but SQL's `LIKE` uses `%` and `_` wildcards. When building queries intended for SQL, use SQL-style patterns (`L%` instead of `L.*`). If you need full regex support, replace the `LIKE` translation with your database's regex function (e.g., `REGEXP` in MySQL, `~` in PostgreSQL).
:::

## Building Complete SELECT Statements

With the `toSql` interpreter, building complete SQL statements is straightforward:

```scala
def select(table: String, predicate: SchemaExpr[?, Boolean]): String =
  s"SELECT * FROM $table WHERE ${toSql(predicate)}"

def selectColumns(table: String, columns: List[String], predicate: SchemaExpr[?, Boolean]): String =
  s"SELECT ${columns.mkString(", ")} FROM $table WHERE ${toSql(predicate)}"

def selectWithLimit(
  table: String,
  predicate: SchemaExpr[?, Boolean],
  orderBy: Option[String] = None,
  limit: Option[Int] = None
): String = {
  val base = s"SELECT * FROM $table WHERE ${toSql(predicate)}"
  val ordered = orderBy.fold(base)(col => s"$base ORDER BY $col")
  limit.fold(ordered)(n => s"$ordered LIMIT $n")
}
```

```scala
val query = (Product.category === "Electronics") && (Product.inStock === true) && (Product.price < 500.0)
```

```scala
select("products", query)
// res15: String = "SELECT * FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0))"

selectColumns("products", List("name", "price"), query)
// res16: String = "SELECT name, price FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0))"

selectWithLimit("products", query, orderBy = Some("price ASC"), limit = Some(10))
// res17: String = "SELECT * FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0)) ORDER BY price ASC LIMIT 10"
```

## Parameterized Queries

The `toSql` function above inlines literal values directly into the SQL string. For production use, you need parameterized queries to prevent SQL injection. We modify the interpreter to collect parameters separately:

```scala
case class SqlQuery(sql: String, params: List[Any])

def toParameterized[A, B](expr: SchemaExpr[A, B]): SqlQuery = expr match {

  case SchemaExpr.Optic(optic) =>
    SqlQuery(columnName(optic), Nil)

  case SchemaExpr.Literal(value, _) =>
    SqlQuery("?", List(value))

  case SchemaExpr.Relational(left, right, op) =>
    val l = toParameterized(left)
    val r = toParameterized(right)
    val sqlOp = op match {
      case SchemaExpr.RelationalOperator.Equal              => "="
      case SchemaExpr.RelationalOperator.NotEqual           => "<>"
      case SchemaExpr.RelationalOperator.LessThan           => "<"
      case SchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case SchemaExpr.RelationalOperator.GreaterThan        => ">"
      case SchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)

  case SchemaExpr.Logical(left, right, op) =>
    val l = toParameterized(left)
    val r = toParameterized(right)
    val sqlOp = op match {
      case SchemaExpr.LogicalOperator.And => "AND"
      case SchemaExpr.LogicalOperator.Or  => "OR"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)

  case SchemaExpr.Not(inner) =>
    val i = toParameterized(inner)
    SqlQuery(s"NOT (${i.sql})", i.params)

  case SchemaExpr.Arithmetic(left, right, op, _) =>
    val l = toParameterized(left)
    val r = toParameterized(right)
    val sqlOp = op match {
      case SchemaExpr.ArithmeticOperator.Add      => "+"
      case SchemaExpr.ArithmeticOperator.Subtract => "-"
      case SchemaExpr.ArithmeticOperator.Multiply => "*"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)

  case SchemaExpr.StringConcat(left, right) =>
    val l = toParameterized(left)
    val r = toParameterized(right)
    SqlQuery(s"CONCAT(${l.sql}, ${r.sql})", l.params ++ r.params)

  case SchemaExpr.StringRegexMatch(regex, string) =>
    val s = toParameterized(string)
    val r = toParameterized(regex)
    SqlQuery(s"(${s.sql} LIKE ${r.sql})", s.params ++ r.params)

  case SchemaExpr.StringLength(string) =>
    val s = toParameterized(string)
    SqlQuery(s"LENGTH(${s.sql})", s.params)
}
```

Now literals become `?` placeholders, with the actual values collected in a parameter list:

```scala
val q = (Product.category === "Electronics") && (Product.price < 500.0) && (Product.rating >= 4)
val paramQuery = toParameterized(q)
```

```scala
paramQuery.sql
// res18: String = "(((category = ?) AND (price < ?)) AND (rating >= ?))"
paramQuery.params
// res19: List[Any] = List("Electronics", 500.0, 4)
```

You can use this with JDBC's `PreparedStatement`:

```scala
val ps = connection.prepareStatement(s"SELECT * FROM products WHERE ${paramQuery.sql}")
paramQuery.params.zipWithIndex.foreach { case (value, idx) =>
  value match {
    case s: String  => ps.setString(idx + 1, s)
    case d: Double  => ps.setDouble(idx + 1, d)
    case i: Int     => ps.setInt(idx + 1, i)
    case b: Boolean => ps.setBoolean(idx + 1, b)
    case l: Long    => ps.setLong(idx + 1, l)
  }
}
val rs = ps.executeQuery()
```

:::warning
Always use parameterized queries for user-supplied values. The inline `toSql` function is suitable for logging and debugging, but use `toParameterized` for actual database execution.
:::

## Nested Structures and Table-Qualified Columns

When domain types have nested structures, optic paths contain multiple `Field` nodes. For SQL, these often map to JOIN-based queries with table-qualified column names.

```scala

case class Address(city: String, country: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Seller(name: String, address: Address, rating: Double)
object Seller extends CompanionOptics[Seller] {
  implicit val schema: Schema[Seller] = Schema.derived

  val name: Lens[Seller, String]       = optic(_.name)
  val rating: Lens[Seller, Double]     = optic(_.rating)
  val city: Lens[Seller, String]       = optic(_.address.city)
  val country: Lens[Seller, String]    = optic(_.address.country)
}
```

The lens `Seller.city` has the path `[Field("address"), Field("city")]`. We can translate multi-segment paths into table-qualified column names:

```scala
def qualifiedColumnName(optic: zio.blocks.schema.Optic[?, ?]): String = {
  val fields = optic.toDynamic.nodes.collect {
    case f: DynamicOptic.Node.Field => f.name
  }
  // Single field: use as-is. Multiple fields: table.column convention
  if (fields.length <= 1) fields.mkString
  else s"${fields.init.mkString("_")}.${fields.last}"
}
```

```scala
qualifiedColumnName(Seller.name)
// res21: String = "name"
qualifiedColumnName(Seller.city)
// res22: String = "address.city"
qualifiedColumnName(Seller.country)
// res23: String = "address.country"
```

This produces `address.city` for nested fields, which maps naturally to a SQL JOIN:

```sql
SELECT sellers.*, address.city, address.country
FROM sellers
JOIN addresses AS address ON sellers.id = address.seller_id
WHERE address.city = 'Berlin' AND sellers.rating >= 4.0
```

To generate full JOIN queries, you would extend the interpreter to inspect the optic paths, detect multi-segment paths, and emit appropriate JOIN clauses. The path structure from `DynamicOptic` gives you all the information needed.

## Putting It Together

Here is a complete, self-contained example that defines a domain, builds queries, and generates both inline SQL and parameterized queries:

```scala

// --- Domain ---

case class Product(
  name: String,
  price: Double,
  category: String,
  inStock: Boolean,
  rating: Int
)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]      = optic(_.name)
  val price: Lens[Product, Double]     = optic(_.price)
  val category: Lens[Product, String]  = optic(_.category)
  val inStock: Lens[Product, Boolean]  = optic(_.inStock)
  val rating: Lens[Product, Int]       = optic(_.rating)
}

// --- SQL Interpreter ---

def columnName(optic: zio.blocks.schema.Optic[?, ?]): String =
  optic.toDynamic.nodes.collect { case f: DynamicOptic.Node.Field => f.name }.mkString("_")

def sqlLiteral(value: Any): String = value match {
  case s: String  => s"'${s.replace("'", "''")}'"
  case b: Boolean => if (b) "TRUE" else "FALSE"
  case n: Number  => n.toString
  case other      => other.toString
}

def toSql[A, B](expr: SchemaExpr[A, B]): String = expr match {
  case SchemaExpr.Optic(optic)                    => columnName(optic)
  case SchemaExpr.Literal(value, _)               => sqlLiteral(value)
  case SchemaExpr.Relational(left, right, op) =>
    val sqlOp = op match {
      case SchemaExpr.RelationalOperator.Equal              => "="
      case SchemaExpr.RelationalOperator.NotEqual           => "<>"
      case SchemaExpr.RelationalOperator.LessThan           => "<"
      case SchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case SchemaExpr.RelationalOperator.GreaterThan        => ">"
      case SchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    s"(${toSql(left)} $sqlOp ${toSql(right)})"
  case SchemaExpr.Logical(left, right, op) =>
    val sqlOp = op match {
      case SchemaExpr.LogicalOperator.And => "AND"
      case SchemaExpr.LogicalOperator.Or  => "OR"
    }
    s"(${toSql(left)} $sqlOp ${toSql(right)})"
  case SchemaExpr.Not(inner)                      => s"NOT (${toSql(inner)})"
  case SchemaExpr.Arithmetic(left, right, op, _) =>
    val sqlOp = op match {
      case SchemaExpr.ArithmeticOperator.Add      => "+"
      case SchemaExpr.ArithmeticOperator.Subtract => "-"
      case SchemaExpr.ArithmeticOperator.Multiply => "*"
    }
    s"(${toSql(left)} $sqlOp ${toSql(right)})"
  case SchemaExpr.StringConcat(left, right)       => s"CONCAT(${toSql(left)}, ${toSql(right)})"
  case SchemaExpr.StringRegexMatch(regex, string) => s"(${toSql(string)} LIKE ${toSql(regex)})"
  case SchemaExpr.StringLength(string)            => s"LENGTH(${toSql(string)})"
}

// --- Parameterized queries ---

case class SqlQuery(sql: String, params: List[Any])

def toParameterized[A, B](expr: SchemaExpr[A, B]): SqlQuery = expr match {
  case SchemaExpr.Optic(optic)      => SqlQuery(columnName(optic), Nil)
  case SchemaExpr.Literal(value, _) => SqlQuery("?", List(value))
  case SchemaExpr.Relational(left, right, op) =>
    val l = toParameterized(left); val r = toParameterized(right)
    val sqlOp = op match {
      case SchemaExpr.RelationalOperator.Equal              => "="
      case SchemaExpr.RelationalOperator.NotEqual           => "<>"
      case SchemaExpr.RelationalOperator.LessThan           => "<"
      case SchemaExpr.RelationalOperator.LessThanOrEqual    => "<="
      case SchemaExpr.RelationalOperator.GreaterThan        => ">"
      case SchemaExpr.RelationalOperator.GreaterThanOrEqual => ">="
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)
  case SchemaExpr.Logical(left, right, op) =>
    val l = toParameterized(left); val r = toParameterized(right)
    val sqlOp = op match {
      case SchemaExpr.LogicalOperator.And => "AND"
      case SchemaExpr.LogicalOperator.Or  => "OR"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)
  case SchemaExpr.Not(inner) =>
    val i = toParameterized(inner)
    SqlQuery(s"NOT (${i.sql})", i.params)
  case SchemaExpr.Arithmetic(left, right, op, _) =>
    val l = toParameterized(left); val r = toParameterized(right)
    val sqlOp = op match {
      case SchemaExpr.ArithmeticOperator.Add      => "+"
      case SchemaExpr.ArithmeticOperator.Subtract => "-"
      case SchemaExpr.ArithmeticOperator.Multiply => "*"
    }
    SqlQuery(s"(${l.sql} $sqlOp ${r.sql})", l.params ++ r.params)
  case SchemaExpr.StringConcat(left, right) =>
    val l = toParameterized(left); val r = toParameterized(right)
    SqlQuery(s"CONCAT(${l.sql}, ${r.sql})", l.params ++ r.params)
  case SchemaExpr.StringRegexMatch(regex, string) =>
    val s = toParameterized(string); val r = toParameterized(regex)
    SqlQuery(s"(${s.sql} LIKE ${r.sql})", s.params ++ r.params)
  case SchemaExpr.StringLength(string) =>
    val s = toParameterized(string)
    SqlQuery(s"LENGTH(${s.sql})", s.params)
}

// --- Complete SELECT builder ---

def select(table: String, predicate: SchemaExpr[?, Boolean]): String =
  s"SELECT * FROM $table WHERE ${toSql(predicate)}"

// --- Usage ---

val query =
  (Product.category === "Electronics") &&
  (Product.inStock === true) &&
  (Product.price < 500.0) &&
  (Product.rating >= 4)

// Inline SQL for debugging
println(select("products", query))
// SELECT * FROM products WHERE (((category = 'Electronics') AND (inStock = TRUE)) AND (price < 500.0)) AND (rating >= 4))

// Parameterized SQL for execution
val pq = toParameterized(query)
println(s"SQL:    ${pq.sql}")
println(s"Params: ${pq.params}")
// SQL:    (((category = ?) AND (inStock = ?)) AND (price < ?)) AND (rating >= ?))
// Params: List(Electronics, true, 500.0, 4)

// String operations in SQL
println(toSql(Product.name.matches("L%")))
// (name LIKE 'L%')

// Arithmetic in SQL
println(toSql(Product.price * 0.9))
// (price * 0.9)
```

## Going Further

- **[Part 1: Expressions](./query-dsl-reified-optics.md)** -- Building query expressions with reified optics
- **[Part 3: Extending the Expression Language](./query-dsl-extending.md)** -- Adding custom operators (IN, BETWEEN, aggregates) beyond SchemaExpr
- **[Part 4: A Fluent SQL Builder](./query-dsl-fluent-builder.md)** -- Type-safe SELECT, UPDATE, INSERT, DELETE with seamless condition mixing
- **[SchemaExpr Reference](../reference/schema-expr.md)** -- Full API coverage of expression types
- **[Optics Reference](../reference/optics.md)** -- Lens, Prism, Optional, and Traversal
- **[DynamicOptic Reference](../reference/dynamic-optic.md)** -- Runtime optic paths for programmatic field extraction

The interpreter pattern shown here extends naturally to other query targets. Because `SchemaExpr` is a sealed trait and `DynamicOptic` carries full path metadata, you can write interpreters for MongoDB filters, Elasticsearch queries, GraphQL filters, or any other query language using the same approach: pattern match on the AST, map operators, and extract field names from optic paths.

---

## Migrating from ZIO Schema to ZIO Blocks Schema

This guide helps you migrate an application that uses [ZIO Schema](https://github.com/zio/zio-schema) (version 1.x) to [ZIO Blocks Schema](https://github.com/zio/zio-blocks) (the schema module of ZIO Blocks). It covers the conceptual differences between the two libraries, provides a systematic mapping of data types, and shows how to rewrite the most common patterns in the idiomatic ZIO Blocks style.

**What we will cover:**

- Prerequisites and dependency changes
- The core architectural shift from `Schema[A]` as a sealed trait to `Schema[A]` as a thin wrapper over `Reflect[F, A]`
- Migrating schema definitions for primitives, records, enums, collections, optional values, and newtypes
- Replacing the annotation/modifier system
- Adapting codec derivation to the unified `Format + Deriver` model
- Replacing `DynamicValue` usage
- Migrating optics and accessor patterns
- Migrating diff, patch, and schema evolution patterns
- Handling types and features that no longer have a direct analogue

---

## Prerequisites

### Dependency Changes

Replace the ZIO Schema dependency group with ZIO Blocks Schema:

**Before (ZIO Schema 1.x):**

```scala
libraryDependencies += "dev.zio" %% "zio-schema"            % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-derivation" % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-json"       % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-protobuf"   % "1.x.x"
libraryDependencies += "dev.zio" %% "zio-schema-avro"       % "1.x.x"
```

**After (ZIO Blocks Schema):**

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema"            % "0.0.33"
// Optional codec modules:
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro"       % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift"     % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson"       % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon"       % "0.0.33"
```

Key points:
- JSON codec support is now built into `zio-blocks-schema` — no separate JSON module.
- There is no separate `zio-blocks-schema-derivation` dependency; derivation is built in.
- The `scala-reflect` provided dependency (required in ZIO Schema for Scala 2) is still needed for Scala 2 macro derivation — add it the same way as before.
- ZIO Blocks Schema has **zero runtime dependency on ZIO itself**. You do not need `zio` on your classpath for schema operations.

### Package Rename

All imports change from `zio.schema` to `zio.blocks.schema`:

```scala
// Before

// After

```

---

## The Core Architecture Shift

The most important thing to understand when migrating is that `Schema[A]` is no longer a sealed trait hierarchy — it is a thin case class:

```scala
// ZIO Schema 1.x: Schema is a sealed trait with ~20 concrete cases
sealed trait Schema[A] {
  def annotations: Chunk[Any]
  def defaultValue: Either[String, A]
  // ...
}

// ZIO Blocks Schema: Schema is a case class wrapping Reflect
final case class Schema[A](reflect: Reflect.Bound[A])
```

The structural description lives in `Reflect[F[_, _], A]`, a sealed trait with eight node types. The `F` type parameter distinguishes a *bound* reflect (with runtime constructors and deconstructors) from an *unbound* one (structural information only):

```
Reflect[F, A]
├── Reflect.Record[F, A]         — case classes and other product types
├── Reflect.Variant[F, A]        — sealed traits, enums, Option, Either
├── Reflect.Sequence[F, A, C[_]] — List, Vector, Set, Chunk, etc.
├── Reflect.Map[F, K, V, M[_,_]] — Map[K, V]
├── Reflect.Primitive[F, A]      — Int, String, UUID, java.time.*, etc.
├── Reflect.Wrapper[F, A, B]     — opaque types and validated newtypes
├── Reflect.Dynamic[F]           — escape hatch for schema-agnostic data
└── Reflect.Deferred[F, A]       — recursive (self-referential) types
```

As a result, you will rarely pattern match on `Schema[A]` directly — instead, you work through `schema.reflect` when you need to inspect structure.

---

## Migrating Schema Definitions

### Schema Derivation

Automatic derivation syntax is essentially unchanged:

```scala
// ZIO Schema 1.x — Scala 2: type inferred from ascription; Scala 3: type param required

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
}

// ZIO Blocks Schema — identical call in Scala 2 and Scala 3

final case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}
```

`Schema.derived[A]` works identically in both Scala 2 and Scala 3 in ZIO Blocks Schema. There is no separate `DeriveSchema` import and no arity limit.

### Primitives

All 30 primitive types from ZIO Schema are present in ZIO Blocks Schema with the same coverage: `Unit`, `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`, `String`, `BigInt`, `BigDecimal`, all `java.time.*` types, `Currency`, and `UUID`.

Implicit schemas are available in the same way:

```scala
// ZIO Schema 1.x
val s: Schema[Int]              = Schema[Int]
val s: Schema[java.time.Instant] = Schema[java.time.Instant]

// ZIO Blocks Schema — identical call sites
val s: Schema[Int]               = Schema[Int]
val s: Schema[java.time.Instant]  = Schema[java.time.Instant]
```

The underlying representation changes: ZIO Schema uses `Schema.Primitive[A](standardType: StandardType[A])`, while ZIO Blocks Schema uses `Reflect.Primitive[F, A](primitiveType: PrimitiveType[A], ...)`. Both carry default values and ordering, but in ZIO Blocks the primitive type also carries an embedded `Validation[A]` constraint (see the [Migrating Validation](#migrating-validation) section below).

### Records (Case Classes)

**Before (ZIO Schema 1.x):**

```scala

final case class Address(street: String, city: String, postCode: String)
final case class Person(name: String, age: Int, address: Address)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
}
```

**After (ZIO Blocks Schema):**

```scala

final case class Address(street: String, city: String, postCode: String)
final case class Person(name: String, age: Int, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}
```

The derivation call is identical. Internally, ZIO Blocks generates a `Reflect.Record` node (a single generic type, not the arity-specialised `CaseClass1`..`CaseClass22` of ZIO Schema), so there is no 22-field arity limit.

If you were writing schemas manually using `Schema.CaseClass2[...]` or similar, you will need to rewrite those. The equivalent in ZIO Blocks is to write the `Reflect.Record` directly or, preferably, just use `Schema.derived[A]`:

```scala
// ZIO Schema 1.x — manual construction for a 2-field record
val personSchema: Schema[Person] =
  Schema.CaseClass2[String, Int, Person](
    id0 = TypeId.fromTypeName("Person"),
    field01 = Schema.Field("name", Schema[String], get0 = _.name, set0 = (p, v) => p.copy(name = v)),
    field02 = Schema.Field("age",  Schema[Int],    get0 = _.age,  set0 = (p, v) => p.copy(age = v)),
    construct0 = Person(_, _)
  )

// ZIO Blocks Schema — prefer derivation; no manual CaseClass* required
val personSchema: Schema[Person] = Schema.derived[Person]
```

Manual construction is still possible in ZIO Blocks Schema (by assembling a `Reflect.Record` directly), but it is substantially more involved because you must supply a `Binding.Record` with explicit `Constructor[A]` and `Deconstructor[A]` implementations that use the unboxed register system. Automatic derivation is strongly preferred.

### Sealed Traits / Enums (Sum Types)

**Before (ZIO Schema 1.x):**

```scala

sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = DeriveSchema.gen
}
```

**After (ZIO Blocks Schema):**

```scala

sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
}
```

Again, the call is identical. Internally, ZIO Blocks generates a `Reflect.Variant` node. There is no 22-case arity limit.

### Optional Values

ZIO Schema has a first-class `Schema.Optional[A]` node. In ZIO Blocks, `Option[A]` is modeled as a `Reflect.Variant` with two cases (`None` and `Some`). From a user perspective this is transparent — implicit schemas for `Option[A]` exist in the same form:

```scala
// ZIO Schema 1.x
val optSchema: Schema[Option[String]] = Schema[Option[String]]

// ZIO Blocks Schema — identical
val optSchema: Schema[Option[String]] = Schema[Option[String]]
```

For value-type `Option` variants (e.g., `Option[Int]`), ZIO Blocks provides specialised implicit instances (`Schema.optionInt`, `Schema.optionLong`, etc.) that avoid boxing. These are resolved automatically by the compiler — no code change required.

:::tip
The internal modeling difference (variant vs. dedicated node) is only relevant if you are pattern-matching on the raw `Schema` or `Reflect` structure. In that case, replace any match on `Schema.Optional(inner, _)` with a check on `reflect.isOption` and use `reflect.optionInnerType` to retrieve the inner reflect:

```scala
// ZIO Schema 1.x — pattern matching on Optional
schema match {
  case Schema.Optional(inner, _) => // use inner
  case _                         => // ...
}

// ZIO Blocks Schema — use the isOption predicate
// optionInnerType returns Option[Reflect[F, ?]] where F matches the enclosing Reflect's binding
val r = schema.reflect
if (r.isOption) {
  val inner: Option[Reflect[binding.Binding, ?]] = r.optionInnerType
  // use inner
}
```
:::

### Either

In ZIO Schema, `Either[A, B]` is a first-class `Schema.Either[A, B]` node. In ZIO Blocks, it is modeled as a two-case `Reflect.Variant`. The implicit schema is provided automatically:

```scala
// ZIO Schema 1.x
val eitherSchema: Schema[Either[String, Int]] = Schema.either[String, Int]

// ZIO Blocks Schema
val eitherSchema: Schema[Either[String, Int]] = Schema[Either[String, Int]]
```

:::warning
ZIO Blocks Schema does not have a `Fallback[A, B]` type. If you were using `Schema.Fallback` for partial decoding, you will need to model that with a custom `Reflect.Variant` or handle it in your codec logic directly.
:::

### Collections

All standard collection types are supported with the same implicit schema pattern:

```scala
// ZIO Schema 1.x
Schema[List[String]]
Schema[Vector[Int]]
Schema[Chunk[Double]]
Schema[Set[String]]
Schema[Map[String, Int]]

// ZIO Blocks Schema — identical call sites
Schema[List[String]]
Schema[Vector[Int]]
Schema[Chunk[Double]]   // uses zio.blocks.chunk.Chunk
Schema[Set[String]]
Schema[Map[String, Int]]
```

Note that `Chunk` is now `zio.blocks.chunk.Chunk` (not `zio.Chunk`). This is a zero-dependency replacement with the same API surface for typical usage.

ZIO Schema's `NonEmptyChunk` and `NonEmptyMap` schemas do not have direct equivalents in ZIO Blocks Schema. The recommended approach is to model them as wrapper types:

```scala
// ZIO Schema 1.x — NonEmptyChunk implicit schema
val schema: Schema[NonEmptyChunk[String]] = Schema[NonEmptyChunk[String]]

// ZIO Blocks Schema — model as a validated wrapper

final case class NonEmptyList[A] private (values: List[A])
object NonEmptyList {
  def apply[A](head: A, tail: A*): NonEmptyList[A] = new NonEmptyList(head :: tail.toList)

  implicit def schema[A](implicit element: Schema[A]): Schema[NonEmptyList[A]] =
    Schema[List[A]].transform(
      to   = list =>
        if (list.nonEmpty) new NonEmptyList(list)
        else throw SchemaError.validationFailed("List must not be empty"),
      from = _.values
    )
}
```

### Newtypes and Opaque Types

ZIO Schema uses `Schema.transform` (which wraps a `Transform` node) for both validated newtypes and lossless wrappers:

```scala
// ZIO Schema 1.x
implicit val bigDecimalSchema: Schema[BigDecimal] =
  Schema.primitive[java.math.BigDecimal].transform(BigDecimal(_), _.bigDecimal)
```

ZIO Blocks Schema has `Schema[A].transform(to: A => B, from: B => A)` which produces a `Reflect.Wrapper` node. The `to` and `from` functions are total but can throw to indicate failure. The method also requires an implicit `TypeId[B]`, which is derived automatically by the macro system for any concrete named type — you will not need to supply it manually for ordinary case classes:

```scala
// ZIO Blocks Schema
case class Email(value: String)
object Email {
  // TypeId[Email] is resolved implicitly from the macro-derived instance
  implicit val schema: Schema[Email] =
    Schema[String].transform(
      to   = str =>
        if (str.contains('@')) Email(str)
        else throw SchemaError.validationFailed("Not a valid email address"),
      from = _.value
    )
}
```

For simple lossless wrappers where no validation is needed, the pattern is the same but without the error throw:

```scala
// ZIO Blocks Schema — simple newtype wrapper
case class UserId(value: Long)
object UserId {
  implicit val schema: Schema[UserId] =
    Schema[Long].transform(UserId(_), _.value)
}
```

:::warning
In ZIO Schema, `transformOrFail` accepted `A => Either[String, B]` return types. In ZIO Blocks, `transform` uses total functions that throw on failure — use `throw SchemaError.validationFailed(message)` in the `to` function to signal failure. There is no `transformOrFail` method.

If you encounter a "could not find implicit value for parameter typeId: TypeId[B]" error, ensure the target type `B` is a concrete, named class or object (not an anonymous structural type or a type alias to a primitive). For primitive-backed aliases such as `type Meters = Double`, wrap in a `case class` instead.
:::

### Lazy / Recursive Schemas

**Before (ZIO Schema 1.x):**

```scala

case class Tree(value: Int, children: List[Tree])
object Tree {
  implicit lazy val schema: Schema[Tree] = DeriveSchema.gen
  // Or manually with Schema.defer:
  // implicit lazy val schema: Schema[Tree] = Schema.CaseClass2(
  //   ..., field02 = Schema.Field("children", Schema.defer(Schema.list(schema)), ...)
  // )
}
```

**After (ZIO Blocks Schema):**

```scala

case class Tree(value: Int, children: List[Tree])
object Tree {
  implicit val schema: Schema[Tree] = Schema.derived[Tree]
}
```

Recursive types are handled automatically by the macro. Internally, ZIO Blocks generates a `Reflect.Deferred` node that uses thread-local cycle detection — you do not need to use `Schema.defer` manually. The `implicit val` (not `lazy val`) is sufficient.

If you were wrapping a recursive reference manually with `Schema.defer(...)`, simply remove that wrapper — recursive references inside `Schema.derived` are handled for you.

---

## Migrating Annotations and Modifiers

ZIO Schema uses an open `Chunk[Any]` annotation system. ZIO Blocks Schema replaces this with a strongly-typed, sealed `Modifier` hierarchy.

### Transient Fields

```scala
// ZIO Schema 1.x

final case class User(name: String, @transientField password: String)
object User {
  implicit val schema: Schema[User] = DeriveSchema.gen
}

// ZIO Blocks Schema

// Transient fields must have a default value in ZIO Blocks Schema.
// Because transient fields are excluded from serialization, the decoder
// needs a default to reconstruct the object without that field in the input.
final case class User(name: String, @Modifier.transient() password: String = "")
object User {
  implicit val schema: Schema[User] = Schema.derived[User]
}
```

### Field Renaming

```scala
// ZIO Schema 1.x — @fieldName annotation

final case class Product(@fieldName("product_name") name: String, price: Double)

// ZIO Blocks Schema — @Modifier.rename annotation

final case class Product(@Modifier.rename("product_name") name: String, price: Double)
```

### Field Aliases (for Decoding)

```scala
// ZIO Schema 1.x — @fieldNameAliases annotation

final case class Config(@fieldNameAliases("max-size", "max_size") maxSize: Int)

// ZIO Blocks Schema — @Modifier.alias annotation (one alias per annotation)

final case class Config(
  @Modifier.alias("max-size")
  @Modifier.alias("max_size")
  maxSize: Int
)
```

### Codec-Specific Configuration

```scala
// ZIO Schema 1.x — no standard mechanism; each codec module defines its own
// e.g., @fieldDefaultValue, @optionalField, or codec-specific annotations

// ZIO Blocks Schema — use @Modifier.config with convention "format.property"

final case class Message(
  @Modifier.config("protobuf.field-id", "1") id: Long,
  @Modifier.config("protobuf.field-id", "2") content: String
)
```

### Discriminator and Case Name Annotations

```scala
// ZIO Schema 1.x

@discriminatorName("type")
sealed trait Event
@caseName("user_created")
case class UserCreated(userId: String) extends Event

// ZIO Blocks Schema — use Modifier.rename on the case, Modifier.config for discriminator

sealed trait Event
@Modifier.rename("user_created")
case class UserCreated(userId: String) extends Event
```

For discriminator key configuration on the enclosing sealed trait, use `Modifier.config` on the reflect node after derivation:

```scala
implicit val schema: Schema[Event] =
  Schema.derived[Event].modifier(Modifier.config("json.discriminator", "type"))
```

### Programmatic Annotation

ZIO Schema allows adding annotations at any time via `schema.annotate(annotation)`. In ZIO Blocks, you add modifiers:

```scala
// ZIO Schema 1.x
val schema2 = schema.annotate(someAnnotation)

// ZIO Blocks Schema
val schema2 = schema.modifier(Modifier.config("key", "value"))
```

---

## Migrating Codec Derivation

### The Unified Format Model

ZIO Schema has separate codec APIs in each codec sub-module (e.g., `JsonCodec.jsonCodec`, `ProtobufCodec.protobufCodec`). ZIO Blocks Schema introduces a unified `codec.Format` interface that all codec modules implement. Codecs are derived via a consistent call:

```scala
// ZIO Schema 1.x — each codec module has its own factory
// JsonCodec.jsonCodec returns a zio.json.JsonCodec (a text codec from the zio-json library)

val jsonCodec = JsonCodec.jsonCodec(Person.schema)

// ProtobufCodec.protobufCodec returns a BinaryCodec[A] (Chunk[Byte] in / out)

val protoCodec: BinaryCodec[Person] = ProtobufCodec.protobufCodec(Person.schema)

// ZIO Blocks Schema — all codecs via schema.derive(Format); return type inferred

val jsonCodec = Person.schema.derive(JsonFormat)   // inferred: JsonCodec[Person]

val avroCodec = Person.schema.derive(AvroFormat)
```

Derived codecs are cached per `(Schema, Format)` pair — subsequent calls to `schema.derive(JsonFormat)` return the same instance.

### Encoding and Decoding

The codec interface differs significantly between the two libraries:

```scala
// ZIO Schema 1.x — Protobuf (true BinaryCodec: Chunk[Byte] in/out)

val codec: BinaryCodec[Person] = ProtobufCodec.protobufCodec(Person.schema)
val encoded: Chunk[Byte]              = codec.encode(Person("Alice", 30))
val decoded: Either[DecodeError, Person] = codec.decode(encoded)

// ZIO Schema 1.x — JSON (zio-json JsonCodec: String in/out)

val jsonCodec = JsonCodec.jsonCodec(Person.schema)
val json: String                       = jsonCodec.encodeJson(Person("Alice", 30), None).toString
val fromJson: Either[String, Person]   = jsonCodec.decodeJson(json)
```

```scala
// ZIO Blocks Schema — all formats use ByteBuffer (binary) or CharBuffer (text)

val person = Person("Alice", 30)

// Encode
val buffer = ByteBuffer.allocate(1024)
Person.schema.encode(JsonFormat)(buffer)(person)

// Decode
buffer.flip()
val result: Either[SchemaError, Person] = Person.schema.decode(JsonFormat)(buffer)
```

ZIO Blocks codecs use `java.nio.ByteBuffer` for binary formats and `java.nio.CharBuffer` for text formats, and do not depend on `zio-json` or any other external codec library.

### JSON Codec

JSON support is built into the core `zio-blocks-schema` module — no separate dependency is needed:

```scala
// ZIO Schema 1.x — requires a separate zio-schema-json module
libraryDependencies += "dev.zio" %% "zio-schema-json" % "1.x.x"

val codec = JsonCodec.jsonCodec(Person.schema)  // returns zio.json.JsonCodec

// ZIO Blocks Schema — built into zio-blocks-schema; no extra dependency

val codec = Person.schema.derive(JsonFormat)  // returns JsonCodec[Person]
```

### Streaming Codecs

ZIO Schema's streaming codec methods (`streamEncoder`, `streamDecoder`) integrated with `ZStream`. ZIO Blocks Schema codecs are format-level `encode`/`decode` operations over `ByteBuffer` or `CharBuffer` — they do not depend on ZIO's streaming primitives. If you need streaming, wrap the codec in your effect system's streaming abstraction.

---

## Migrating DynamicValue

### Structure Changes

The `DynamicValue` ADT is significantly simplified in ZIO Blocks — from 15 cases down to 6. The key differences:

- ZIO Schema's `DynamicValue.Primitive[A](value: A, standardType: StandardType[A])` stores the raw value and its `StandardType` inline. ZIO Blocks wraps the scalar in a `PrimitiveValue` case class instead.
- `DynamicValue.Record` drops the `TypeId` parameter and uses `Chunk[(String, DynamicValue)]` instead of `ListMap[String, DynamicValue]`.
- `Option`, `Either`, and `Tuple` are no longer dedicated ADT cases — they are represented structurally using `Variant` and `Record`.

| ZIO Schema | ZIO Blocks Schema |
|---|---|
| `Primitive[A](value: A, standardType: StandardType[A])` | `Primitive(value: PrimitiveValue)` |
| `Record(id: TypeId, values: ListMap[String, DynamicValue])` | `Record(fields: Chunk[(String, DynamicValue)])` — no TypeId |
| `Enumeration(id: TypeId, value: (String, DynamicValue))` | `Variant(caseName: String, value: DynamicValue)` |
| `Sequence(values: Chunk[DynamicValue])` | `Sequence(elements: Chunk[DynamicValue])` |
| `Dictionary(entries: Chunk[(DynamicValue, DynamicValue)])` | `Map(entries: Chunk[(DynamicValue, DynamicValue)])` |
| `SomeValue(value: DynamicValue)` | `Variant("Some", Record(Chunk("value" -> ...)))` |
| `NoneValue` | `Variant("None", Null)` |
| `LeftValue(value: DynamicValue)` | `Variant("Left", Record(Chunk("value" -> ...)))` |
| `RightValue(value: DynamicValue)` | `Variant("Right", Record(Chunk("value" -> ...)))` |
| `Tuple(left, right)` | `Record(Chunk("_1" -> left, "_2" -> right))` |
| `SetValue(values: Set[DynamicValue])` | `Sequence(elements: Chunk[DynamicValue])` |
| `BothValue(left, right)` | No direct equivalent (used by `Fallback`, which is removed) |
| `DynamicAst(ast: MetaSchema)` | No direct equivalent |
| `Singleton[A](instance: A)` | No direct equivalent |
| `Error(message: String)` | No direct equivalent — use `SchemaError` |

### Primitive Values

In ZIO Schema, primitive values are stored inline in `DynamicValue.Primitive[A](value: A, standardType: StandardType[A])`. There is no separate `PrimitiveValue` type. In ZIO Blocks, a sealed `PrimitiveValue` ADT wraps each primitive:

```scala
// ZIO Schema 1.x — value and StandardType are separate constructor arguments

val pv: DynamicValue = DynamicValue.Primitive(42, StandardType[Int])
val ps: DynamicValue = DynamicValue.Primitive("hello", StandardType[String])

// ZIO Blocks Schema — value is wrapped in a PrimitiveValue case class

val pv: DynamicValue = DynamicValue.Primitive(PrimitiveValue.Int(42))
val ps: DynamicValue = DynamicValue.Primitive(PrimitiveValue.String("hello"))
```

The `PrimitiveValue` case names (`Int`, `Long`, `String`, `Boolean`, `Double`, etc.) match the Scala primitive names.

### Converting Between Typed Values and DynamicValue

```scala
// ZIO Schema 1.x
// toDynamic is a method on Schema[A], not on the value itself
val dv: DynamicValue         = Person.schema.toDynamic(person)
// toTypedValue requires an implicit Schema[Person] in scope
val back: Either[String, Person] = dv.toTypedValue[Person]

// ZIO Blocks Schema
val dv: DynamicValue                  = Person.schema.toDynamicValue(person)
val back: Either[SchemaError, Person] = Person.schema.fromDynamicValue(dv)
```

Two things change: `toDynamic` is renamed `toDynamicValue` (still on `Schema[A]`), and `toTypedValue` is replaced by `schema.fromDynamicValue`. The error type changes from `String` to `SchemaError`.

### DynamicValue Operations

ZIO Blocks `DynamicValue` has a rich operation API that was absent in ZIO Schema. Where ZIO Schema required you to convert back to a typed value to manipulate data, you can now operate directly on `DynamicValue`:

```scala

val record = DynamicValue.Record(
  Chunk(
    "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
    "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(30))
  )
)

// Navigate — get(fieldName) returns DynamicValueSelection (supports chaining)
// Call .one to extract a single value as Either[SchemaError, DynamicValue]
val name: Either[SchemaError, DynamicValue] = record.get("name").one

// Modify — set returns DynamicValue directly (silent no-op if path not found)
// Use setOrFail to get an Either on missing paths
val updated: DynamicValue = record.set(
  DynamicOptic.root.field("name"),
  DynamicValue.Primitive(PrimitiveValue.String("Bob"))
)

// Diff
val other = DynamicValue.Record(Chunk(
  "name" -> DynamicValue.Primitive(PrimitiveValue.String("Bob")),
  "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(31))
))
val patch = record.diff(other)
```

---

## Migrating Schema Introspection

### MetaSchema → DynamicSchema

ZIO Schema has `MetaSchema` (a type-erased structural description of a schema) and `schema.ast` to convert to it. ZIO Blocks uses `DynamicSchema` for the same purpose:

```scala
// ZIO Schema 1.x
val meta: MetaSchema = schema.ast
val back: Schema[_]  = meta.toSchema

// ZIO Blocks Schema
val dynamic: DynamicSchema = schema.toDynamicSchema
```

`DynamicSchema` wraps a `Reflect[NoBinding, _]` — the full structural description without runtime constructors or deconstructors. Use it for:

- Runtime structural validation of `DynamicValue` instances
- Dynamic schema loading from configuration or network
- Schema inspection without compile-time type information

```scala
// ZIO Blocks Schema — validate a DynamicValue against a schema
val personSchema: Schema[Person]   = Schema.derived[Person]
val dynSchema: DynamicSchema       = personSchema.toDynamicSchema

val value = DynamicValue.Record(Chunk(
  "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
  "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(30))
))

dynSchema.conforms(value)         // true
dynSchema.check(value)            // None (no error)
```

:::warning
ZIO Schema's `Migration` system for schema-to-schema migration (i.e., automatically migrating values from one version of a type to another) is **not yet available** in ZIO Blocks Schema. The `schema.migrate[B](newSchema)` and `schema.coerce[B](newSchema)` methods do not exist. If your application relies on schema migration, you have two options:

1. Implement migration logic manually using `DynamicValue` transformations and `DynamicSchema` for validation.
2. Wait for schema migration support to be added to ZIO Blocks Schema (it is on the roadmap).
:::

### Schema Serialization

ZIO Schema supports serializing a schema itself (via `schema.serializable`). ZIO Blocks Schema does not have a direct equivalent at this time. All schema metadata types (`DynamicOptic`, `DynamicPatch`, `Modifier`, `Validation`, etc.) have `Schema` instances and are individually serializable, but there is no single `Schema[Schema[A]]` that round-trips the full structural description.

---

## Migrating Optics

ZIO Schema uses an `AccessorBuilder` pattern that delegates optic creation to an external `zio-schema-optics` module. ZIO Blocks Schema includes a complete, first-class optics system in the core module.

### Generating Optics

**Before (ZIO Schema 1.x with `zio-schema-optics`):**

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
  val (name, age) = schema.makeAccessors(ZioOpticsBuilder)
}
```

**After (ZIO Blocks Schema):**

Optics are generated by the macro derivation and placed directly in the companion object as `Lens` instances via a `CompanionOptics` mechanism. In Scala 3, they are generated automatically. In Scala 2, use `Schema.derived[Person]` and access fields by calling `schema.reflect.asRecord.get.lensByName[String]("name")`, or use the macro-derived companion optics pattern:

```scala
// Scala 3 — optics generated in companion via macro

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  // Scala 3 macro generates: val name: Lens[Person, String] = ...
  //                          val age:  Lens[Person, Int]    = ...
}

// Usage
val lens: Lens[Person, String] = Person.name
val person = Person("Alice", 30)
lens.modify(person, _.toUpperCase)  // Person("ALICE", 30)
```

```scala
// Scala 2 — obtain lenses from the schema

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]

  val name: Lens[Person, String] =
    schema.reflect.asRecord.get.lensByName[String]("name").get
  val age: Lens[Person, Int] =
    schema.reflect.asRecord.get.lensByName[Int]("age").get
}
```

### Using Optics

The four optic types in ZIO Blocks are `Lens`, `Prism`, `Optional`, and `Traversal`. Their usage API is similar to standard optics libraries:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

// Obtain lens (Scala 2 example)
val nameLens: Lens[Person, String] =
  Person.schema.reflect.asRecord.get.lensByName[String]("name").get

val person = Person("Alice", 30)

// Get
val name: String = nameLens.get(person)             // "Alice"

// Modify
val upper: Person = nameLens.modify(person, _.toUpperCase)   // Person("ALICE", 30)

// Replace — note: ZIO Blocks uses replace, not set (unlike Monocle and many other optics libraries)
val renamed: Person = nameLens.replace(person, "Bob")        // Person("Bob", 30)
```

For sealed traits, use `Prism`:

```scala

sealed trait Shape
case class Circle(radius: Double)    extends Shape
case class Rectangle(w: Double, h: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]

  val circlePrism: Prism[Shape, Circle] =
    schema.reflect.asVariant.get.prismByName[Circle]("Circle").get
}

val shape: Shape = Circle(5.0)
Shape.circlePrism.getOption(shape)          // Some(Circle(5.0))
Shape.circlePrism.reverseGet(Circle(3.0))   // Circle(3.0): Shape
```

### Schema Expressions (New in ZIO Blocks)

ZIO Blocks introduces `SchemaExpr[S, A]`, a typed expression language built on top of optics. There is no equivalent in ZIO Schema. These allow you to build inspectable, composable predicates and computations:

```scala

case class Product(name: String, price: Double, inStock: Boolean)
object Product {
  implicit val schema: Schema[Product] = Schema.derived[Product]
  val priceLens: Lens[Product, Double] =
    schema.reflect.asRecord.get.lensByName[Double]("price").get
  val inStockLens: Lens[Product, Boolean] =
    schema.reflect.asRecord.get.lensByName[Boolean]("inStock").get
}

// Build a typed predicate expression
val cheapAndInStock: SchemaExpr[Product, Boolean] =
  (Product.priceLens < 100.0) && (Product.inStockLens === true)

// Evaluate against data — eval returns Either[OpticCheck, Seq[A]]
// Right(Seq(true))  on success
// Left(OpticCheck)  if a prism in the path did not match
val p = Product("Widget", 49.99, inStock = true)
cheapAndInStock.eval(p)  // Right(Seq(true))
```

---

## Migrating Diff and Patch

### Diff

ZIO Schema uses `Differ.fromSchema(schema).diff(a, b)` or the convenience method `schema.diff(a, b)`. ZIO Blocks Schema uses the same convenience method:

```scala
// ZIO Schema 1.x
val patch: Patch[Person] = Person.schema.diff(person1, person2)

// ZIO Blocks Schema
val patch: Patch[Person] = Person.schema.diff(person1, person2)
```

The call site is identical, but the underlying `Patch` types are different.

### Patch Application

```scala
// ZIO Schema 1.x
val result: Either[String, Person] = Person.schema.patch(person, patch)

// ZIO Blocks Schema
val result: Either[SchemaError, Person] = Person.schema.patch(person, patch)
// or equivalently:
val result: Either[SchemaError, Person] = patch.apply(person, PatchMode.Strict)
```

The error type changes from `String` to `SchemaError`.

### Creating Patches Programmatically

ZIO Schema has no structured API for creating patches programmatically. ZIO Blocks Schema provides one through `Patch` smart constructors:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  val nameLens: Lens[Person, String] =
    schema.reflect.asRecord.get.lensByName[String]("name").get
  val ageLens: Lens[Person, Int] =
    schema.reflect.asRecord.get.lensByName[Int]("age").get
}

// Set a field
val renamePatch: Patch[Person] = Patch.set(Person.nameLens, "Bob")

// Compose patches
val combined: Patch[Person] = renamePatch ++ Patch.set(Person.ageLens, 31)

// Apply
val updated: Either[SchemaError, Person] = combined(Person("Alice", 30), PatchMode.Strict)
```

---

## Migrating Type Class Derivation

### Before (ZIO Schema 1.x)

ZIO Schema does not have a general `Deriver[TC]` interface. Each codec module implements its own derivation logic independently. There is no way to derive an arbitrary user-defined type class from a `Schema[A]`.

### After (ZIO Blocks Schema)

ZIO Blocks Schema introduces `Deriver[TC]`, a unified interface for deriving any type class `TC[_]` from a schema. This replaces ad-hoc codec-specific derivation:

```scala

// Define a type class
trait Show[A] {
  def show(a: A): String
}

// Implement Deriver[Show]
object DeriveShow extends Deriver[Show] {

  def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding[BindingType.Primitive, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Show[A]] = Lazy {
    new Show[A] {
      def show(a: A): String = a.toString
    }
  }

  def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, _]],
    typeId: TypeId[A],
    binding: Binding[BindingType.Record, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    val recordBinding = binding.asInstanceOf[Binding.Record[A]]
    val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, _]]]
    val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
    Lazy {
      new Show[A] {
        private lazy val resolvedShows: IndexedSeq[Show[Any]] =
          fields.map(f => D.instance(f.value.metadata).asInstanceOf[Lazy[Show[Any]]].force)
        def show(a: A): String = {
          val regs = Registers(recordReflect.usedRegisters)
          recordBinding.deconstructor.deconstruct(regs, RegisterOffset.Zero, a)
          val fieldStrs = fields.indices.map { i =>
            val v = recordReflect.registers(i).get(regs, RegisterOffset.Zero)
            s"${fields(i).name} = ${resolvedShows(i).show(v)}"
          }
          s"${typeId.name}(${fieldStrs.mkString(", ")})"
        }
      }
    }
  }

  // ... deriveVariant, deriveSequence, deriveMap, deriveDynamic, deriveWrapper
  // (see the DeriveShowExample in the examples module for full implementation)
}

// Derive Show for any type
case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  implicit val show: Show[Person]     = schema.derive(DeriveShow)
}

Person.show.show(Person("Alice", 30))  // Person(name = "Alice", age = 30)
```

---

## Migrating Validation

### Before (ZIO Schema 1.x)

ZIO Schema uses a composable `Validation[A]` ADT as an annotation, attached via `@validate(...)` or `.validation(...)`:

```scala

case class User(
  @validate(Validation.greaterThan(0)) age: Int,
  @validate(Validation.minLength(3))   name: String
)
object User {
  implicit val schema: Schema[User] = DeriveSchema.gen
}

schema.validate(User(-1, "Al"))
// Returns Chunk[ValidationError] with violations
```

### After (ZIO Blocks Schema)

ZIO Blocks Schema has a simpler, non-composable `Validation[A]` that is embedded inside `PrimitiveType[A]` and checked during `DynamicSchema.check`. It is not composable with `And`/`Or`/`Not`:

```scala

// Validation is checked during DynamicSchema.check — not during fromDynamicValue
val dynSchema = Schema[Int].toDynamicSchema

val valid   = DynamicValue.Primitive(PrimitiveValue.Int(5))
val invalid = DynamicValue.Primitive(PrimitiveValue.Int(-1))

dynSchema.conforms(valid)    // true
dynSchema.conforms(invalid)  // true (no validation constraint on the base Int schema)
```

For validated types, use `Schema[A].transform` with a throwing `to` function, which signals failure during `fromDynamicValue`:

```scala

// Validated positive integer
val positiveIntSchema: Schema[Int] =
  Schema[Int].transform(
    to   = n => if (n > 0) n else throw SchemaError.validationFailed("Must be positive"),
    from = identity
  )

positiveIntSchema.fromDynamicValue(
  DynamicValue.Primitive(PrimitiveValue.Int(-1))
)
// Left(SchemaError: Must be positive)
```

For struct-level validation across multiple fields, implement validation in the `to` function of a wrapper:

```scala

final case class AgeRange(min: Int, max: Int)
object AgeRange {
  implicit val schema: Schema[AgeRange] = Schema.derived[AgeRange]
  // Schema-level validation is handled through the derived schema's
  // DynamicSchema.check, or by adding custom validation in a wrapping transform.
}
```

:::info
If you rely heavily on ZIO Schema's composable validation (chaining `And`, `Or`, `Not`, `Transform` validators), you will need to implement that logic in the `to` function of a `Schema.transform` wrapper, or in application-level validation code. ZIO Blocks Schema's built-in `Validation` is deliberately simpler: it covers the most common primitive constraints without the complexity of a full combinator library.
:::

---

## Migrating the Fail Schema

ZIO Schema provides `Schema.fail[A](message: String)` to represent the absence of schema information:

```scala
// ZIO Schema 1.x
val missing: Schema[MyType] = Schema.fail("No schema available for MyType")
```

ZIO Blocks Schema has no equivalent `Fail` schema node. The recommended approach is to leave the implicit schema undefined and let the compiler report the missing instance, or to throw from a type class derivation:

```scala
// ZIO Blocks Schema — no Schema.fail; use a compile error or a runtime exception approach
// If you need a runtime sentinel, use Schema[DynamicValue] or create a minimal placeholder:
val placeholder: Schema[DynamicValue] = Schema[DynamicValue]
```

---

## Summary of Missing Features

The following ZIO Schema features do not yet have equivalents in ZIO Blocks Schema:

| Feature | Status |
|---|---|
| `Schema.fail` / fail schemas | Not available |
| `Schema.migrate[B]` / `Schema.coerce[B]` | Not available — schema migration is planned |
| `MetaSchema` / schema serialization | Partial — `DynamicSchema` covers structural inspection; full schema round-trip is not available |
| `Fallback[A, B]` schema | Not available |
| `NonEmptyChunk` / `NonEmptyMap` schemas | Not available — use wrapper types |
| `Schema.Singleton` / singleton schemas | Not available |
| `DynamicValue.BothValue` / `DynamicValue.DynamicAst` | Not available |
| Composable `Validation` (`And`, `Or`, `Not`) | Not available — use `transform` with throwing functions |
| Streaming codec methods (`streamEncoder`, `streamDecoder`) | Not available — wrap codecs in your effect system |
| ZIO `Chunk` (from `zio-core`) | Replaced by `zio.blocks.chunk.Chunk` |

---

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

```bash
# Step 1: Schema derivation, primitives, and DynamicValue roundtrip
sbt "schema-examples/runMain ziosschemamigration.Step1SchemaDerivedAndPrimitives"

# Step 2: Modifiers and transform (annotations, newtypes)
sbt "schema-examples/runMain ziosschemamigration.Step2ModifiersAndTransform"

# Step 3: Optics (Lens, Prism) and DynamicSchema validation
sbt "schema-examples/runMain ziosschemamigration.Step3OpticsAndDynamicSchema"

# Step 4: Diff and patch
sbt "schema-examples/runMain ziosschemamigration.Step4DiffAndPatch"

# Complete example: end-to-end e-commerce domain
sbt "schema-examples/runMain ziosschemamigration.CompleteMigrationExample"

# Type class derivation — deriving Show from a Schema
sbt "schema-examples/runMain typeclassderivation.DeriveShowExample"

# Type class derivation — deriving a random generator from a Schema
sbt "schema-examples/runMain typeclassderivation.DeriveGenExample"
```

**3. Or compile all examples at once:**

```bash
sbt "schema-examples/compile"
```

---

## Going Further

- [Schema Reference](../reference/schema.md) — full `Schema[A]` API
- [Reflect Reference](../reference/reflect.md) — the `Reflect[F, A]` node types
- [Binding Reference](../reference/binding.md) — constructors, deconstructors, and the register system
- [Optics Reference](../reference/optics.md) — `Lens`, `Prism`, `Optional`, `Traversal`
- [Type Class Derivation Guide](../reference/type-class-derivation.md) — implementing `Deriver[TC]`
- [Codec Reference](../reference/codec.md) — the `Format` and `Codec` infrastructure
- [DynamicValue Reference](../reference/dynamic-value.md) — the `DynamicValue` API
- [Validation Reference](../reference/validation.md) — built-in validation constraints

---

## ZIO Blocks

**Modular, zero-dependency building blocks for modern Scala applications.**

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-blocks/workflows/CI/badge.svg) [![ZIO Blocks](https://img.shields.io/github/stars/zio/zio-blocks?style=social)](https://github.com/zio/zio-blocks)

## What Is ZIO Blocks?

ZIO Blocks is a **family of type-safe, modular building blocks** for Scala applications. Each block is a standalone library with zero or minimal dependencies, designed to work with *any* Scala stack—ZIO, Cats Effect, Kyo, Ox, Akka, or plain Scala.

The philosophy is simple: **use what you need, nothing more**. Each block is independently useful, cross-platform (JVM, JS), and designed to compose with other blocks or your existing code.

## The Blocks

| Block | Description | Status |
|-------|-------------|--------|
| **Schema** | Type-safe schemas with automatic codec derivation | ✅ Available |
| **Chunk** | High-performance immutable indexed sequences | ✅ Available |
| **Scope** | Compile-time safe resource management and DI | ✅ Available |
| **Docs** | GitHub Flavored Markdown parsing and rendering | ✅ Available |
| **TypeId** | Compile-time type identity with rich metadata | ✅ Available |
| **Context** | Type-indexed heterogeneous collections | ✅ Available |
| **MediaType** | Type-safe IANA media types with 2,600+ predefined types | ✅ Available |
| **Ring Buffer** | High-performance bounded ring buffers (SPSC, MPSC, SPMC, MPMC) | ✅ Available |
| **Streams** | Pull-based streaming primitives | 🚧 In Development |

## Core Principles

- **Zero Lock-In**: No dependencies on ZIO, Cats Effect, or any effect system. Use with whatever stack you prefer.
- **Modular**: Each block is a separate artifact. Import only what you need.
- **Cross-Platform**: Full support for JVM and Scala.js.
- **Cross-Version**: Full support for Scala 2.13 and Scala 3.x with source compatibility—adopt Scala 3 on your timeline, not ours.
- **High Performance**: Optimized implementations that avoid boxing, minimize allocations, and leverage platform-specific features.
- **Type Safety**: Leverage Scala's type system for correctness without runtime overhead.

---

## Schema

The Schema block brings dynamic-language productivity to statically-typed Scala. Define your data types once, and derive codecs, validators, optics, and more automatically.

### The Problem

In statically-typed languages, you often maintain separate codec implementations for each data format (JSON, Avro, Protobuf, etc.). Meanwhile, dynamic languages handle data effortlessly:

```javascript
// JavaScript: one line and done
const data = await res.json();
```

In Scala, you'd typically need separate codecs for each format—a significant productivity gap.

### The Solution

ZIO Blocks Schema derives everything from a single schema definition:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive codecs for any format:
val jsonCodec    = Schema[Person].derive(JsonFormat)        // JSON
val avroCodec    = Schema[Person].derive(AvroFormat)        // Avro
val toonCodec    = Schema[Person].derive(ToonFormat)        // TOON (LLM-optimized)
val msgpackCodec = Schema[Person].derive(MessagePackFormat) // MessagePack
val thriftCodec  = Schema[Person].derive(ThriftFormat)      // Thrift
```

### Key Features

- **Universal Data Formats**: JSON, Avro, TOON (compact LLM-optimized format), MessagePack, Thrift, and BSON, with Protobuf planned.
- **High Performance**: Register-based design stores primitives directly in byte arrays, enabling zero-allocation serialization.
- **Reflective Optics**: Type-safe lenses, prisms, and traversals with embedded structural metadata.
- **Automatic Derivation**: Derive type class instances for any type with a schema.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"

// Optional format modules:
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro" % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon" % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift" % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson" % "0.0.33"
```

### Example: Optics

```scala

case class Address(street: String, city: String)
case class Person(name: String, age: Int, address: Address)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int] = $(_.age)
  val streetName: Lens[Person, String] = $(_.address.street)
}

val person = Person("Alice", 30, Address("123 Main St", "Springfield"))
val updated = Person.age.replace(person, 31)
```

---

## Chunk

A high-performance, immutable indexed sequence optimized for the patterns common in streaming, parsing, and data processing. Think of it as `Vector` but faster for the operations that matter most.

### Why Chunk?

Standard library collections make trade-offs that aren't ideal for streaming and binary data processing:

- `Vector` is general-purpose but not optimized for concatenation patterns
- `Array` is mutable and boxes primitives when used generically
- `List` has O(n) random access

Chunk is designed for:

- **Fast concatenation** via balanced trees (Conc-Trees)
- **Zero-boxing** for primitive types with specialized builders
- **Efficient slicing** without copying
- **Seamless interop** with `ByteBuffer`, `Array`, and standard collections

### Key Features

- **Specialized Builders**: Dedicated builders for `Byte`, `Int`, `Long`, `Double`, etc. avoid boxing overhead.
- **Balanced Concatenation**: Based on Conc-Trees for O(log n) concatenation while maintaining O(1) indexed access.
- **Bit Operations**: First-class support for bit-level operations, bit chunks backed by `Byte`, `Int`, or `Long` arrays.
- **NonEmptyChunk**: A statically-guaranteed non-empty variant for APIs that require at least one element.
- **Full Scala Collection Integration**: Implements `IndexedSeq` for seamless interop.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-chunk" % "0.0.33"
```

### Example

```scala

// Create chunks
val bytes = Chunk[Byte](1, 2, 3, 4, 5)
val moreBytes = Chunk.fromArray(Array[Byte](6, 7, 8))

// Efficient concatenation (O(log n))
val combined = bytes ++ moreBytes

// Zero-copy slicing
val slice = combined.slice(2, 6)

// Bit operations
val bits = bytes.asBitsByte
val masked = bits & Chunk.fill(bits.length)(true)

// NonEmptyChunk for type-safe non-emptiness
val nonEmpty = NonEmptyChunk(1, 2, 3)
val head: Int = nonEmpty.head  // Always safe, no Option needed
```

---

## Scope

Compile-time verified resource safety for synchronous Scala code. Scope prevents resource leaks at compile time by tagging values with an unnameable type-level identity—values allocated in a scope can only be used within that scope. Child scope values cannot escape to parent scopes, enforced by both the abstract scope-tagged type and the `Unscoped` constraint on `scoped`.

### The Problem

Resource management in Scala is error-prone:

```scala
// Classic try/finally - verbose and easy to get wrong
val db = openDatabase()
try {
  val tx = db.beginTransaction()
  try {
    doWork(tx)
    tx.commit()
  } finally tx.close()  // What if commit() throws?
} finally db.close()

// Using - better, but doesn't prevent returning resources
Using(openDatabase()) { db =>
  db  // Oops! Returned the resource - use after close!
}
```

### The Solution

Scope makes resource leaks a **compile error**, not a runtime bug:

```scala

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(Resource(openDatabase()))

  // Methods are hidden - can't call db.query() directly
  // Must use $ to access:
  val result: String = $(db)(_.query("SELECT 1"))

  // Trying to return `db` would be a compile error!
  result  // Only pure data (String) escapes
}
// db.close() called automatically
```

### Key Features

- **Compile-Time Leak Prevention**: Values of type `scope.$[A]` are opaque and unique to each scope instance. Returning a scoped value from its scope is a type error.
- **Zero Runtime Overhead**: `$[A]` erases to `A` at runtime—zero allocation overhead.
- **Structured Scopes**: Child scopes nest within parents; resources clean up LIFO when scopes exit.
- **Built-in Dependency Injection**: Wire up your application with `Resource.from[T](wires*)` for automatic constructor-based DI.
- **AutoCloseable Integration**: Resources implementing `AutoCloseable` have `close()` registered automatically.
- **Unscoped Constraint**: The `scoped` method requires `Unscoped[A]` evidence on the return type, ensuring only pure data (not resources or closures) can escape.
- **Actionable Runtime Errors**: If a scope reference escapes and is used after closing, `allocate`, `open()`, and `$` throw `IllegalStateException` with a detailed message explaining what went wrong, the common causes, and how to fix it—no silent null returns.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.33"
```

### Example: Basic Resource Management

```scala

final class Database extends AutoCloseable:
  def query(sql: String): String = s"Result: $sql"
  def close(): Unit = println("Database closed")

Scope.global.scoped { scope =>

  // Allocate returns $[Database] (scoped value)
  val db: $[Database] = allocate(Resource(new Database))

  // Access via $ - result (String) escapes, db does not
  val result: String = $(db)(_.query("SELECT * FROM users"))

  println(result)
}
// Output: Result: SELECT * FROM users
//         Database closed
```

### Example: Dependency Injection

```scala

case class Config(dbUrl: String)
class Database(config: Config) extends AutoCloseable { ... }
class UserRepo(db: Database) { ... }
class UserService(repo: UserRepo) extends AutoCloseable { ... }

// Resource.from auto-wires the dependency graph
// Only provide leaf values - concrete classes are auto-wired
val serviceResource: Resource[UserService] = Resource.from[UserService](
  Wire(Config("jdbc:postgresql://localhost/mydb"))
)

Scope.global.scoped { scope =>

  val service = allocate(serviceResource)

  $(service)(_.createUser("Alice"))
}
// Cleanup runs LIFO: UserService → Database (UserRepo has no cleanup)
```

### Example: Nested Scopes with Transactions

```scala
Scope.global.scoped { connScope =>

  val conn = allocate(Resource.fromAutoCloseable(new Connection))

  // Transaction lives in child scope - cleaned up before connection
  val result: String = scoped { txScope =>

    val c  = lower(conn)
    val tx = $(c)(_.beginTransaction()).allocate
    $(tx)(_.execute("INSERT INTO users VALUES (1, 'Alice')"))
    $(tx)(_.commit())
    "success"
  }
  // Transaction closed here, connection still open

  println(result)
}
// Connection closed here
```

### Getting Started

New to Scope? Check out the [Scope Tutorial](./guides/compile-time-resource-safety-with-scope.md) for a comprehensive step-by-step guide that walks you through the concepts, patterns, and real-world examples. The tutorial is designed for newcomers and covers everything from basic resource management to advanced dependency injection.

For detailed API documentation, see the [Scope Reference](./reference/resource-management/scope.md).

---

## Docs

A zero-dependency GitHub Flavored Markdown library for parsing, rendering, and programmatic construction of Markdown documents.

### Why Docs?

Generating documentation, README files, or any Markdown content programmatically is common but error-prone with string concatenation. Docs provides:

- **Type-safe AST**: Build Markdown documents with compile-time guarantees
- **Compile-time validation**: The `md"..."` interpolator validates syntax at compile time
- **Multiple renderers**: Output to Markdown, HTML, or ANSI terminal
- **Round-trip parsing**: Parse Markdown to AST and render back to Markdown

### Key Features

- **GFM Compliant**: Tables, strikethrough, autolinks, task lists, fenced code blocks
- **Zero Dependencies**: Only depends on zio-blocks-chunk
- **Cross-Platform**: Full support for JVM and Scala.js
- **Type-Safe Interpolator**: `md"# Hello $name"` with compile-time validation
- **Multiple Renderers**: Markdown, HTML (full document or fragment), ANSI terminal

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-docs" % "0.0.33"
```

### Example

```scala

// Parse Markdown
val doc = Parser.parse("# Hello\n\nThis is **bold** text.")
// Right(Doc(Chunk(Heading(H1, "Hello"), Paragraph(...))))

// Render to HTML
val html = doc.map(_.toHtml)
// Full HTML5 document with <html>, <head>, <body>

// Render to HTML fragment (just the content)
val fragment = doc.map(_.toHtmlFragment)
// "HelloThis is bold text."

// Render to terminal with ANSI colors
val terminal = doc.map(_.toTerminal)

// Use the type-safe interpolator
val name = "World"
val greeting = md"# Hello $name"
// Doc containing: Heading(H1, Chunk(Text("Hello World")))

// Build documents programmatically

val manual = Doc(Chunk(
  Block.Heading(HeadingLevel.H1, Chunk(Inline.Text("API Reference"))),
  Block.Paragraph(Chunk(
    Inline.Text("See "),
    Inline.Link(Chunk(Inline.Text("docs")), "/docs", None),
    Inline.Text(" for details.")
  ))
))

// Render back to Markdown
val markdown = Renderer.render(manual)
```

### Supported GFM Features

| Feature | Supported |
|---------|-----------|
| Headings (ATX) | ✅ |
| Paragraphs | ✅ |
| Emphasis/Strong | ✅ |
| Code (inline & fenced) | ✅ |
| Links & Images | ✅ |
| Lists (bullet, ordered, task) | ✅ |
| Blockquotes | ✅ |
| Tables | ✅ |
| Strikethrough | ✅ |
| Autolinks | ✅ |
| Hard/Soft breaks | ✅ |
| HTML (passthrough) | ✅ |

### Limitations

- **No frontmatter**: YAML/TOML headers are not parsed
- **No HTML entity decoding**: `&amp;` stays as-is
- **No footnotes**: GFM footnote extension not supported
- **No emoji shortcodes**: `:smile:` not converted to emoji

---

## TypeId

Compile-time type identity with rich metadata. TypeId captures comprehensive information about Scala types including name, owner, type parameters, variance, parent types, and annotations.

### Key Features

- **Rich Metadata**: Captures type name, owner, kind (class/trait/object/enum), parent types, and annotations
- **Higher-Kinded Support**: Works with proper types and type constructors via `AnyKind`
- **Subtype Checking**: Runtime subtype/supertype relationship checks using compile-time extracted information
- **Cross-Platform**: Works identically on JVM and Scala.js

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-typeid" % "0.0.33"
```

### Example

```scala

// Get TypeId for any type
val listId = TypeId.of[List[Int]]
println(listId.name)       // "List"
println(listId.fullName)   // "scala.collection.immutable.List"
println(listId.arity)      // 1 (type constructor)

// Check type relationships
trait Animal
case class Dog(name: String) extends Animal

val dogId = TypeId.of[Dog]
val animalId = TypeId.of[Animal]
dogId.isSubtypeOf(animalId)  // true

// Access structural information
dogId.isCaseClass  // true
dogId.isSealed     // false
```

---

## Context

A type-indexed heterogeneous collection that stores values by their types with compile-time type safety.

### Key Features

- **Type-Safe Lookup**: Retrieve values by type with compile-time guarantees
- **Covariant**: `Context[Specific]` is a subtype of `Context[General]`
- **Subtype Matching**: Lookup by supertype finds matching subtypes
- **Cached Access**: O(1) subsequent lookups after first retrieval

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-context" % "0.0.33"
```

### Example

```scala

case class Config(debug: Boolean)
case class Metrics(count: Int)

// Create a context with multiple values
val ctx: Context[Config & Metrics] = Context(
  Config(debug = true),
  Metrics(count = 42)
)

// Retrieve values by type
val config: Config = ctx.get[Config]
val metrics: Metrics = ctx.get[Metrics]

// Add or update values
val updated = ctx.update[Metrics](m => m.copy(count = m.count + 1))

// Combine contexts
val ctx1 = Context(Config(false))
val ctx2 = Context(Metrics(0))
val merged: Context[Config & Metrics] = ctx1 ++ ctx2
```

---

## Ring Buffer

High-performance, bounded ring buffers for inter-thread communication. Four lock-free variants cover every producer/consumer pattern (SPSC, MPSC, SPMC, MPMC).

### Why Ring Buffer?

Standard `java.util.concurrent` queues use node allocation (`ConcurrentLinkedQueue`) or coarse locking (`ArrayBlockingQueue`). Ring buffers avoid both:

- **Zero allocation** on the hot path—pre-allocated circular array
- **Lock-free** on the fast path—CAS or release/acquire semantics only
- **Cache-friendly**—sequential memory access with 128-byte padding between producer/consumer fields

### Key Features

- **Four concurrency patterns**: SPSC, SPMC, MPSC, MPMC—pick the most constrained variant for your use case
- **Cross-platform**: Same API on JVM and Scala.js (JS uses sequential implementations)

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-ringbuffer" % "0.0.33"
```

### Example

```scala

// SPSC: fastest, for dedicated producer-consumer pairs
val spsc = SpscRingBuffer[String](1024)
spsc.offer("hello") // true
spsc.take()          // "hello"

// MPMC: general-purpose, any number of threads
val mpmc = MpmcRingBuffer[String](1024)
mpmc.offer("hello") // false if full
mpmc.take()          // null if empty
```

---

## Streams (In Development)

A pull-based streaming library for composable, backpressure-aware data processing.

```scala

// Coming soon: efficient pull-based streams
// that compose with any effect system
```

---

## Compatibility

ZIO Blocks works with any Scala stack:

| Stack | Compatible |
|-------|------------|
| ZIO 2.x | ✅ |
| Cats Effect 3.x | ✅ |
| Kyo | ✅ |
| Ox | ✅ |
| Akka | ✅ |
| Plain Scala | ✅ |

Each block has zero dependencies on effect systems. Use the blocks directly, or integrate them with your effect system of choice.

## Scala & Platform Support

ZIO Blocks supports **Scala 2.13** and **Scala 3.x** with full source compatibility. Write your code once and compile it against either version—migrate to Scala 3 when your team is ready, not when your dependencies force you.

| Platform | Schema | Chunk | Scope | Docs | TypeId | Context | Ring Buffer | Streams |
|----------|--------|-------|-------|------|--------|---------|-------------|---------|
| JVM | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Scala.js | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

## Documentation

### Core Schema Concepts

- [Schema](./reference/schema.md) - Core schema definitions and derivation
- [Allows](./reference/allows.md) - Compile-time structural grammar constraints
- [Reflect](./reference/reflect.md) - Structural reflection API
- [Binding](./reference/binding.md) - Runtime constructors and deconstructors
- [BindingResolver](./reference/binding-resolver.md) - Binding lookup and schema rebinding
- [Registers](./reference/registers.md) - Register-based primitive storage

### Optics & Navigation

- [Optics](./reference/optics.md) - Lenses, prisms, and traversals
- [SchemaExpr](./reference/schema-expr.md) - Schema-aware expressions for queries and validation
- [Path Interpolator](./path-interpolator.md) - Type-safe path construction
- [DynamicValue](./reference/dynamic-value.md) - Schema-less dynamic values
- [DynamicSchema](./reference/dynamic-schema.md) - Type-erased schemas for validation and cross-process transport

### Serialization

- [Codec & Format](./reference/codec.md) - Codec, Format, BinaryCodec & TextCodec
- [JSON](./reference/json.md) - JSON codec and parsing
- [JsonPatch](./reference/json-patch.md) - Diff and patch JSON values
- [JsonDiffer](./reference/json-differ.md) - Compute minimal diffs between JSON values
- [JSON Schema](./reference/json-schema.md) - JSON Schema generation and validation
- [Formats](./reference/formats.md) - Avro, TOON, MessagePack, BSON, Thrift
- [Extension Syntax](./reference/syntax.md) - `.toJson`, `.fromJson`, and more

### Data Operations

- [Patching](./reference/patch.md) - Serializable data transformations
- [SchemaError](./reference/schema-error.md) - Structured error type for schema operations
- [Validation](./reference/validation.md) - Data validation and error handling
- [Schema Evolution](./reference/schema-evolution/index.md) - One-way and bidirectional type-safe conversions
  - [Into](./reference/schema-evolution/into.md) - One-way conversion with validation
  - [As](./reference/schema-evolution/as.md) - Bidirectional round-trip conversion

### Other Blocks

- [Chunk](./reference/chunk.md) - High-performance immutable sequences
- [Scope](./reference/resource-management/scope.md) - Compile-time safe resource management and DI
- [Wire](./reference/resource-management/wire.md) - Recipes for constructing services and dependencies
- [TypeId](./reference/typeid.md) - Type identity and metadata
- [Context](./reference/context.md) - Type-indexed heterogeneous collections
- [Docs (Markdown)](./reference/docs.md) - Markdown parsing and rendering
- [MediaType](./reference/media-type.md) - Type-safe IANA media types
- [HTTP Model](./reference/http-model.md) - Pure HTTP data model with URL parsing, headers, cookies, and forms
- [Ring Buffer](./ringbuffer.md) - High-performance bounded ring buffers

### Guides

- [Migrating from ZIO Schema](./guides/zio-schema-migration.md) - Step-by-step guide to migrating from ZIO Schema 1.x to ZIO Blocks Schema
- [Query DSL Part 1: Expressions](./guides/query-dsl-reified-optics.md) - Build type-safe, composable query expressions
- [Query DSL Part 2: SQL Generation](./guides/query-dsl-sql.md) - Translate query expressions into SQL
- [Query DSL Part 3: Extending the Expression Language](./guides/query-dsl-extending.md) - Add custom operators beyond SchemaExpr
- [Query DSL Part 4: A Fluent SQL Builder](./guides/query-dsl-fluent-builder.md) - Build type-safe SELECT, UPDATE, INSERT, DELETE statements

---

## Path Interpolator

The path interpolator `p"..."` is a compile-time string interpolator for constructing `DynamicOptic` instances in ZIO Blocks. It provides a clean, concise syntax for building optic paths that navigate through complex data structures, with all parsing and validation happening at compile time for zero runtime overhead.

**Why use the path interpolator?**

Instead of manually constructing optics like this:

```scala
DynamicOptic(Vector(
  DynamicOptic.Node.Field("users"),
  DynamicOptic.Node.Elements,
  DynamicOptic.Node.Field("email")
))
```

You can write:

```scala
p".users[*].email"
```

The interpolator is **type-safe**, **compile-time validated**, and **performance-optimized** with zero runtime parsing overhead.

## Getting Started

Import the schema package to enable the path interpolator:

```scala

// Now you can use p"..." anywhere
val path = p".users[0].name"
```

## Key Features

- **✅ Zero Runtime Overhead**: All parsing happens at compile time
- **✅ Cross-Platform**: Works on Scala 2.13.x and Scala 3.x
- **✅ Compile-Time Safety**: Invalid paths are rejected during compilation
- **✅ No Runtime Interpolation**: Prevents accidental use of runtime values
- **✅ Rich Syntax**: Supports all `DynamicOptic` operations

## Syntax Reference

### Field Access

Access fields in records using dot notation. The leading dot is optional.

```scala
// With leading dot
p".name"           // Field("name")
p".firstName"      // Field("firstName")

// Without leading dot
p"name"            // Field("name")
p"firstName"       // Field("firstName")

// Chained fields
p".user.address.street"
// Equivalent to: Field("user") → Field("address") → Field("street")
```

**Special cases:**

```scala
p"._private"       // Fields starting with underscore
p".field123"       // Fields with digits
p".café"           // Unicode field names
p".true"           // Keywords as field names (true, false, null)
```

### Index Access

Access sequence elements by index, multiple indices, or ranges.

**Single index:**

```scala
p"[0]"             // AtIndex(0)
p"[42]"            // AtIndex(42)
p"[2147483647]"    // AtIndex(Int.MaxValue)
```

**Multiple indices:**

```scala
p"[0,1,2]"         // AtIndices(Seq(0, 1, 2))
p"[0, 2, 5]"       // AtIndices(Seq(0, 2, 5)) - spaces allowed
p"[5,2,8,1]"       // Order preserved
```

**Ranges:**

```scala
p"[0:5]"           // AtIndices(Seq(0, 1, 2, 3, 4))
p"[5:8]"           // AtIndices(Seq(5, 6, 7))
p"[3:4]"           // AtIndices(Seq(3)) - single element
p"[5:5]"           // AtIndices(Seq.empty) - empty range
p"[10:5]"          // AtIndices(Seq.empty) - inverted range
```

### Element Selectors

Select all elements in a sequence using wildcard syntax.

```scala
p"[*]"             // Elements - all elements
p"[:*]"            // Elements - alternative syntax
```

**Chained selectors:**

```scala
p"[*][*]"          // Nested sequences: all elements of all elements
p"[*][0]"          // First element of each sequence
```

### Map Access

Access map values by key, where keys can be strings, integers, booleans, or characters.

**String keys:**

```scala
p"""{"host"}"""              // AtMapKey(String("host"))
p"""{"foo bar"}"""           // Keys with spaces
p"""{"日本語"}"""             // Unicode keys
p"""{"🎉"}"""                 // Emoji keys
p"""{""}"""                  // Empty string key
```

**Integer keys:**

```scala
p"{42}"                      // AtMapKey(Int(42))
p"{0}"                       // AtMapKey(Int(0))
p"{-42}"                     // AtMapKey(Int(-42))
p"{2147483647}"              // AtMapKey(Int.MaxValue)
p"{-2147483648}"             // AtMapKey(Int.MinValue)
```

**Boolean keys:**

```scala
p"{true}"                    // AtMapKey(Boolean(true))
p"{false}"                   // AtMapKey(Boolean(false))
```

**Char keys:**

```scala
p"{'a'}"                     // AtMapKey(Char('a'))
p"{' '}"                     // AtMapKey(Char(' '))
p"{'9'}"                     // AtMapKey(Char('9'))
```

**Multiple keys:**

```scala
p"""{"foo", "bar", "baz"}""" // AtMapKeys(Seq(...))
p"{1, 2, 3}"                 // Multiple integer keys
p"{true, false}"             // Multiple boolean keys

// Mixed types
p"""{"foo", 42}"""           // AtMapKeys(Seq(String("foo"), Int(42)))
p"""{"s", 'c', 42, true}"""  // All supported types
```

### Map Selectors

Select all keys or all values in a map.

```scala
p"{*}"             // MapValues - all values
p"{:*}"            // MapValues - alternative syntax
p"{*:}"            // MapKeys - all keys
```

**Examples:**

```scala
p"{*}{*}"          // Nested maps: all values of all values
p"{*:}{*:}"        // All keys of all keys
```

### Variant Case Access

Navigate into a specific variant case using angle brackets.

```scala
p"<Left>"          // Case("Left")
p"<Right>"         // Case("Right")
p"<Some>"          // Case("Some")
p"<None>"          // Case("None")
```

**Special cases:**

```scala
p"<_Empty>"        // Cases starting with underscore
p"<Case1>"         // Cases with digits
p"<café>"          // Unicode case names
```

**Chained cases:**

```scala
p"<C>"       // Nested variants
```

### Schema Search

Search for values matching a schema pattern anywhere in a data structure using the `#` prefix.

**Nominal types:**

```scala
p"#Person"         // Find all values of type Person
p"#User"           // Find all values of type User
p"#Address"        // Find all values of type Address
```

**Primitive types:**

```scala
p"#string"         // Find all string values
p"#int"            // Find all integer values
p"#boolean"        // Find all boolean values
p"#uuid"           // Find all UUID values
```

**Structural records:**

```scala
p"#record { name: string }"                // Find records with a string 'name' field
p"#record { name: string, age: int }"      // Find records with both fields
p"#record { items: list(Person) }"         // Nested schema
```

**Structural variants:**

```scala
p"#variant { Left: int, Right: string }"   // Find Either-like variants
```

**Collections:**

```scala
p"#list(string)"                           // Find lists of strings
p"#list(Person)"                           // Find lists of Person
p"#map(string, int)"                       // Find maps from string to int
p"#option(Person)"                         // Find optional Person values
```

**Wildcard:**

```scala
p"#_"              // Find any value (matches everything)
```

**Combined paths with search:**

```scala
p".users#Person"               // Search for Person in users field
p"#Person.name"                // Find all Person values, then get name
p".items[*]#Person.email"      // Elements then search then field
p"#list(Person)#Person"        // Chained searches
```

## Escape Sequences

String and character literals support standard escape sequences:

| Escape | Result  | Description     |
|--------|---------|-----------------|
| `\n`   | newline | Line feed       |
| `\t`   | tab     | Horizontal tab  |
| `\r`   | return  | Carriage return |
| `\'`   | `'`     | Single quote    |
| `\"`   | `"`     | Double quote    |
| `\\`   | `\`     | Backslash       |

**Examples:**

```scala
p"""{"foo\nbar"}"""          // String key with newline
p"""{"foo\tbar"}"""          // String key with tab
p"""{'\n'}"""                // Char key with newline
p"""{"foo\"bar"}"""          // Escaped quote in string
p"""{"foo\\bar"}"""          // Escaped backslash in string
```

## Combined Paths

Combine different path elements to navigate complex nested structures.

### Field → Sequence

```scala
p".items[0]"                 // First item
p".items[*]"                 // All items
p".items[0,1,2]"             // Items at indices 0, 1, 2
p".items[0:5]"               // Items 0 through 4
```

### Field → Map

```scala
p""".config{"host"}"""       // Map lookup
p".settings{42}"             // Integer key
p".lookup{*}"                // All map values
p".lookup{*:}"               // All map keys
```

### Field → Variant

```scala
p".result<Success>"          // Variant case
p".response<Ok>"             // HTTP response variant
```

### Nested Structures

```scala
// Record in sequence
p".users[0].name"
// Equivalent to: Field("users") → AtIndex(0) → Field("name")

// All elements then field
p".users[*].email"
// Equivalent to: Field("users") → Elements → Field("email")

// Map values then field
p".lookup{*}.value"
// Equivalent to: Field("lookup") → MapValues → Field("value")

// Variant then field
p".response<Ok>.body"
// Equivalent to: Field("response") → Case("Ok") → Field("body")
```

### Deeply Nested Paths

```scala
// Complex nested navigation
p""".root.children[*].metadata{"tags"}[0]"""
// Field("root") → Field("children") → Elements → 
// Field("metadata") → AtMapKey("tags") → AtIndex(0)

// All node types in one path
p""".a[0]{"k"}<V>.b[*]{*}.c{*:}"""
// Field("a") → AtIndex(0) → AtMapKey("k") → Case("V") →
// Field("b") → Elements → MapValues → Field("c") → MapKeys
```

## Root and Empty Paths

```scala
p""                          // Empty path = root
// Equivalent to: DynamicOptic.root
// Equivalent to: DynamicOptic(Vector.empty)
```

## Compile-Time Safety

The path interpolator **rejects runtime interpolation** to prevent unsafe dynamic path construction.

**❌ This will fail to compile:**

```scala
val fieldName = "email"
val path = p".$fieldName"
// Error: Path interpolator does not support runtime arguments.
//        Use only literal strings like p".field[0]"
```

**❌ This will also fail:**

```scala
val idx = 5
val path = p"[$idx]"
// Error: Path interpolator does not support runtime arguments.
//        Use only literal strings like p".field[0]"
```

**✅ Use only literal strings:**

```scala
val path = p".users[0].email"  // ✓ Works
```

### Parse Error Examples

Invalid syntax is caught at compile time:

```scala
// Unterminated string
p"""{"foo"""
// Error: Unterminated string literal starting at position 1

// Invalid escape sequence
p"""{"foo\x"}"""
// Error: Invalid escape sequence '\x' at position 6

// Unexpected character
p".field@"
// Error: Unexpected character '@' at position 6

// Invalid identifier
p"."
// Error: Invalid identifier at position 1
```

## Performance

**Zero Runtime Overhead**

All path parsing and validation occurs at **compile time**. The interpolator generates the exact same bytecode as manual `DynamicOptic` construction:

```scala
// These produce identical bytecode:
p".users[*].email"

DynamicOptic(Vector(
  DynamicOptic.Node.Field("users"),
  DynamicOptic.Node.Elements,
  DynamicOptic.Node.Field("email")
))
```

There is **no runtime parsing**, **no reflection**, and **no performance penalty**.

## Examples

### Accessing Nested Fields

```scala

case class Address(street: String, city: String, zipCode: String)
case class Person(name: String, age: Int, address: Address)

// Access nested street field
val streetPath = p".address.street"

// Use with DynamicValue
val person = DynamicValue.fromPerson(...)
val street = person.get(streetPath)
```

### Working with Collections

```scala
case class User(id: Int, email: String, tags: Seq[String])
case class Company(name: String, users: Seq[User])

// Get all user emails
val emailsPath = p".users[*].email"

// Get first user's first tag
val firstTagPath = p".users[0].tags[0]"

// Get specific users by index
val specificUsersPath = p".users[0,2,5]"
```

### Map Lookups

```scala
case class Config(
  settings: Map[String, String],
  ports: Map[Int, String]
)

// Lookup by string key
val hostPath = p"""settings{"host"}"""

// Lookup by integer key
val httpPortPath = p"ports{80}"

// Get all config values
val allValuesPath = p"settings{*}"

// Get all port numbers (keys)
val allPortsPath = p"ports{*:}"
```

### Variant Case Handling

```scala
sealed trait Result[+A]
case class Success[A](value: A) extends Result[A]
case class Failure(error: String) extends Result[Nothing]

case class Response(result: Result[User])

// Navigate into Success case
val successValuePath = p".result<Success>.value"

// Navigate into Failure case
val errorPath = p".result<Failure>.error"
```

### Real-World Example: API Response

```scala
case class Metadata(tags: Seq[String], version: Int)
case class Item(id: String, data: String, metadata: Metadata)
case class ApiResponse(
  status: String,
  items: Seq[Item],
  config: Map[String, String]
)

// Get the version from the first item's metadata
val versionPath = p".items[0].metadata.version"

// Get all item IDs
val allIdsPath = p".items[*].id"

// Get the first tag of each item
val firstTagsPath = p".items[*].metadata.tags[0]"

// Lookup config value
val apiKeyPath = p"""config{"api_key"}"""
```

## Before & After Comparison

### Manual Construction (Before)

```scala

// Simple path - verbose and error-prone
val path1 = DynamicOptic(Vector(
  Node.Field("users"),
  Node.AtIndex(0),
  Node.Field("email")
))

// Complex path - extremely verbose
val path2 = DynamicOptic(Vector(
  Node.Field("root"),
  Node.Field("children"),
  Node.Elements,
  Node.Field("metadata"),
  Node.AtMapKey(DynamicValue.Primitive(PrimitiveValue.String("tags"))),
  Node.AtIndex(0)
))

// Map with multiple keys
val path3 = DynamicOptic(Vector(
  Node.Field("data"),
  Node.AtMapKeys(Seq(
    DynamicValue.Primitive(PrimitiveValue.String("foo")),
    DynamicValue.Primitive(PrimitiveValue.String("bar")),
    DynamicValue.Primitive(PrimitiveValue.Int(42))
  ))
))
```

### Path Interpolator (After)

```scala

// Simple path - clean and readable
val path1 = p".users[0].email"

// Complex path - still clean and readable
val path2 = p""".root.children[*].metadata{"tags"}[0]"""

// Map with multiple keys - concise
val path3 = p"""data{"foo", "bar", 42}"""
```

**Benefits:**

- **90% less code** for typical paths
- **Easier to read** and understand intent
- **Easier to write** and maintain
- **Compile-time validated** - catches errors immediately
- **No performance difference** - identical bytecode

## Practical Usage Patterns

### Building Paths Dynamically (at Compile Time)

```scala
// You can't use runtime variables, but you can compose literal paths:
val basePath = p".data.items"
val emailPath = basePath(p"[*].email")
// Same as: p".data.items[*].email"
```

### Working with DynamicValue

```scala

val data: DynamicValue = ...

// Navigate and extract
val value = data.get(p".users[0].email")

// Update at path
val updated = data.set(p".users[0].age", DynamicValue.fromInt(30))
```

### Integration with Schema Optics

```scala

case class User(name: String, email: String)
object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived
  
  // Use path interpolator for complex lenses
  val email = $(_.email)
}

// DynamicOptic can be used for runtime path resolution
val dynamicPath = p".email"
```

## Tips and Best Practices

1. **Use the leading dot for clarity**: While optional, `p".field"` is more explicit than `p"field"`

2. **Leverage compile-time validation**: Let the compiler catch typos and syntax errors early

3. **Compose paths when needed**: Break complex paths into reusable components
   ```scala
   val userPath = p".users[0]"
   val emailPath = userPath(p".email")
   ```

4. **Use raw strings for map keys**: Triple-quoted strings avoid escape hell
   ```scala
   p"""config{"api.key"}"""  // Better than p"config{\"api.key\"}"
   ```

5. **Document complex paths**: Add comments explaining what nested paths navigate
   ```scala
   // Get the first tag from each user's metadata
   val tagsPath = p".users[*].metadata.tags[0]"
   ```

## Limitations

- **No runtime interpolation**: You cannot use variables in paths (this is by design for safety)
- **No arithmetic in ranges**: Ranges must be literal integers (e.g., `[0:5]` not `[0:n]`)
- **No string interpolation**: Only literal strings work with the interpolator
- **Map keys limited to primitives**: Only String, Int, Char, and Boolean keys are supported

These limitations ensure compile-time safety and zero runtime overhead.

## Debug-Friendly toString

`DynamicOptic` instances have a custom `toString` that produces output matching the `p"..."` interpolator syntax. This makes debugging easier because you can copy the output directly into your code:

```scala
val optic = DynamicOptic.root.field("users").elements.field("email")
println(optic)  // Output: .users[*].email

// The output can be copy-pasted into p"..."
val same = p".users[*].email"
```

**Examples:**

| DynamicOptic Construction                            | toString Output   |
|------------------------------------------------------|-------------------|
| `DynamicOptic.root.field("name")`                    | `.name`           |
| `DynamicOptic.root.field("address").field("street")` | `.address.street` |
| `DynamicOptic.root.caseOf("Some")`                                      | `<Some>`          |
| `DynamicOptic.root.at(0)`                                               | `[0]`             |
| `DynamicOptic.root.atIndices(0, 2, 5)`                                  | `[0,2,5]`         |
| `DynamicOptic.elements`                                                 | `[*]`             |
| `DynamicOptic.root.atKey("host")`                                       | `{"host"}`        |
| `DynamicOptic.root.atKey(80)`                                           | `{80}`            |
| `DynamicOptic.mapValues`                                                | `{*}`             |
| `DynamicOptic.mapKeys`                                                  | `{*:}`            |
| `DynamicOptic.wrapped`                                                  | `.~`              |
| `DynamicOptic.root.searchSchema(SchemaRepr.Nominal("Person"))`          | `#Person`         |
| `DynamicOptic.root.searchSchema(SchemaRepr.Primitive("string"))`        | `#string`         |

## Summary

The `p"..."` path interpolator provides:

- **Concise syntax** for building optic paths
- **Compile-time parsing** with zero runtime overhead  
- **Type-safe navigation** through complex data structures
- **Cross-platform support** for Scala 2 and Scala 3
- **Rich feature set** covering all DynamicOptic operations

Use it whenever you need to construct `DynamicOptic` paths for navigating dynamic data structures in ZIO Blocks.

---

## Allows

`Allows[A, S]` is a compile-time capability token that proves, at the call site, that type `A` satisfies the structural grammar `S`. A capability token is a compile-time phantom proof value — it carries no runtime data and exists solely to pass evidence through the type system that a structural constraint has been satisfied.

`Allows` does **not** require or use `Schema[A]`. It inspects the Scala type structure of `A` directly at compile time, using nothing but the Scala type system. Any `Schema[A]` that appears alongside `Allows` in examples is the library author's own separate constraint — it is not imposed by `Allows` itself.

```scala
sealed abstract class Allows[A, S <: Allows.Structural]
```

## Overview

The gap `Allows` fills is **structural preconditions** at the call site, at compile time, with precise error messages. Structural preconditions are constraints on the shape of a type's fields (e.g., "all fields must be scalars"), unlike runtime checks which happen during execution and produce exceptions or errors.

## Motivation

ZIO Blocks gives library authors a powerful way to build data-oriented DSLs. A library can accept `A: Schema` and use the schema at runtime to serialize, deserialize, query, or transform values of `A`. A data-oriented DSL is a generic API built around a data description (Schema) rather than a fixed interface, allowing one function to serialize, validate, or transform any conforming type. Many generic functions have **structural preconditions** that don't require a schema.

Consider these real-world scenarios:

- A CSV serializer requires flat records of scalars — nested records should fail at the call site, not deep inside the serializer.
- An RDBMS layer cannot handle nested records as column values — the error should name the problematic field.
- An event bus expects a sealed trait of flat record cases — violations should be caught before publishing.
- A JSON document store allows arbitrarily nested records but not `DynamicValue` leaves — the schema validation should be precise. DynamicValue is the schema-less escape hatch that can hold arbitrary data — a DynamicValue leaf bypasses compile-time checking entirely, making it impossible for the compiler to enforce any structural grammar.

Without `Allows`, these constraints can only be checked at runtime, producing confusing errors deep inside library internals. With `Allows[A, S]`, the constraint is verified at the **call site**, at compile time, with precise, path-aware error messages and concrete fix suggestions.

## The Upper Bound Semantics

`Allows[A, S]` is an upper bound. A type `A` that uses only a strict subset of what `S` permits also satisfies it — just as `A <: Foo` does not require that `A` uses every method of `Foo`. Upper bound semantics is the right choice because a lower bound would require using every shape (impractical), exact matching would require naming every shape used (too rigid), whereas upper bound says "your type may use any of these shapes" — a permission, not a mandate.

```scala

// Both satisfy Record[Primitive | Optional[Primitive]] — the upper bound

case class UserRow(name: String, age: Int)
// UserRow satisfies the grammar: all fields are Primitive

case class UserRowOpt(name: String, age: Int, email: Option[String])
// UserRowOpt also satisfies the grammar: all fields are Primitive or Optional[Primitive]

val ev1: Allows[UserRow, Record[Primitive | Optional[Primitive]]] = implicitly
val ev2: Allows[UserRowOpt, Record[Primitive | Optional[Primitive]]] = implicitly
```

## Creating Instances

`Allows[A, S]` is not instantiated directly. Instead, you summon an evidence value at the point where you need the constraint. The macro automatically verifies the constraint at compile time.

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

```scala

def toJson[A](doc: A)(implicit ev: Allows[A, Record[Primitive]]): String = ???

// Or summon at the call site:
val evidence = implicitly[Allows[Int, Primitive]]
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

```scala

def toJson[A](doc: A)(using Allows[A, Record[Primitive]]): String = ???

// Calling the function:
case class Person(name: String, age: Int)
val json = toJson(Person("Alice", 30))  // Compiles if Person satisfies Record[Primitive]
```

  </TabItem>
</Tabs>

The constraint is checked once, at the call site. If the type `A` does not satisfy `S`, you get a compile-time error with a precise message showing exactly which field violates the grammar.

## Grammar Nodes

All grammar nodes extend `Allows.Structural`.

| Node | Matches |
|---|---|
| `Primitive` | **Any** scalar — catch-all for all 30 Schema 2 primitive types |
| `Primitive.Boolean` | `scala.Boolean` only |
| `Primitive.Int` | `scala.Int` only |
| `Primitive.Long` | `scala.Long` only |
| `Primitive.Double` | `scala.Double` only |
| `Primitive.Float` | `scala.Float` only |
| `Primitive.String` | `java.lang.String` only |
| `Primitive.BigDecimal` | `scala.BigDecimal` only |
| `Primitive.BigInt` | `scala.BigInt` only |
| `Primitive.Unit` | `scala.Unit` only |
| `Primitive.Byte` | `scala.Byte` only |
| `Primitive.Short` | `scala.Short` only |
| `Primitive.Char` | `scala.Char` only |
| `Primitive.UUID` | `java.util.UUID` only |
| `Primitive.Currency` | `java.util.Currency` only |
| `Primitive.Instant` / `LocalDate` / `LocalDateTime` / … | Each specific `java.time.*` type |
| | |
| `Record[A]` | A case class / product type whose every field satisfies `A`. Vacuously true for zero-field records. Sealed traits and enums are **automatically unwrapped**: each case is checked individually, so no `Variant` node is needed. |
| `Sequence[A]` | Any collection (`List`, `Vector`, `Set`, `Array`, `Chunk`, …) whose element type satisfies `A` |
| `Map[K, V]` | `Map`, `HashMap`, … whose key satisfies `K` and value satisfies `V` |
| `Optional[A]` | `Option[X]` where the inner type `X` satisfies `A` |
| `Wrapped[A]` | A ZIO Prelude `Newtype`/`Subtype` wrapper whose underlying type satisfies `A` |
| | |
| `Self` | Recursive self-reference back to the entire enclosing `Allows[A, S]` grammar |
| `Dynamic` | `DynamicValue` — the schema-less escape hatch |
| `IsType[A]` | Exact nominal type match: satisfied only when the checked type is exactly `A` (`=:=`) |
| `` `\|` `` | Union of two grammar nodes: `A \| B`. In Scala 2 write `` A `\|` B `` in infix position. |

Every specific `Primitive.Xxx` node also satisfies the top-level `Primitive` node (which matches any of the 30 primitive types). This means a type annotated with `Primitive.Int` is valid wherever `Primitive` or `Primitive | Primitive.Long` is required.

## Core Operations

`Allows[A, S]` is a **proof token**, not an ordinary value. It carries zero public methods that you call directly. Instead, you use it in three ways:

1. **As a constraint in function signatures** — Declare `Allows[A, S]` as an implicit/using parameter to require that callers pass only types satisfying the grammar.
2. **To summon evidence** — Use `implicitly[Allows[A, S]]` (Scala 2) or `summon[Allows[A, S]]` (Scala 3) at a call site to check the constraint and get an error message if it fails.
3. **In type aliases** — Define type aliases like `type FlatRecord = Allows[_, Record[Primitive | Optional[Primitive]]]` to name constraints and reuse them across functions.

The macro that powers `Allows` checks the constraint **at compile time** and emits nothing but a reference to a single private singleton at runtime, so there is zero per-call-site overhead.

## Specific Primitives

The `Primitive` parent class is the catch-all: it accepts any of the 30 Schema 2 primitive types. For stricter control — such as when the target serialisation format only supports a subset — use the specific subtype nodes in `Allows.Primitive`:

```scala

// Only JSON-representable scalars (no UUID, Char, java.time.*)
type JsonPrimitive =
  Primitive.Boolean | Primitive.Int | Primitive.Long |
  Primitive.Double  | Primitive.String | Primitive.BigDecimal |
  Primitive.BigInt  | Primitive.Unit

def toJson[A](doc: A)(using Allows[A, Record[JsonPrimitive | Self]]): String = ???

// Only numeric types
type Numeric = Primitive.Int | Primitive.Long | Primitive.Double | Primitive.Float |
               Primitive.BigInt | Primitive.BigDecimal

def aggregate[A](data: A)(using Allows[A, Record[Numeric]]): Double = ???
```

A type annotated with `Primitive.Int` satisfies `Primitive` (the catch-all) because `Primitive.Int extends Primitive`:

```scala

val ev: Allows[Int, Primitive] = implicitly  // Primitive (catch-all) — ✓
val sp: Allows[Int, Primitive.Int] = implicitly  // Primitive.Int (specific) — ✓
```

### JSON Document Store Example

JSON's primitive value set is `null | boolean | number | string`. Types such as `UUID`, `Char`, and all `java.time.*` types have no native JSON representation and must be encoded as strings at the application layer. Using `JsonPrimitive` instead of the catch-all `Primitive` enforces this at compile time.

A JSON document grammar is straightforward: a JSON value is either a record (JSON object) or a sequence (JSON array), and `Self` handles all nesting:

```scala

type JsonPrimitive =
  Primitive.Boolean | Primitive.Int | Primitive.Long | Primitive.Double |
  Primitive.String  | Primitive.BigDecimal | Primitive.BigInt | Primitive.Unit

type Json = Record[JsonPrimitive | Self] | Sequence[JsonPrimitive | Self]

def toJson[A](doc: A)(using Allows[A, Json]): String = ???
```

`Self` recurses back to `Json` at every nested position, so `List[String]` satisfies `Sequence[JsonPrimitive | Self]` (String is JsonPrimitive), `List[Author]` satisfies it too (Author satisfies `Record[JsonPrimitive | Self]` via Self), and top-level arrays work directly.

A type with a UUID or Instant field fails at compile time with this error:

```
[error] Schema shape violation at WithUUID.id: found Primitive(java.util.UUID),
        required Primitive.Boolean | Primitive.Int | ... | Primitive.String | ...
        UUID is not a JSON-native type — encode it as Primitive.String.
```

## Union Syntax

Union types express "or" in the grammar.

<Tabs groupId="scala-version" defaultValue="scala2">
  <TabItem value="scala2" label="Scala 2">

Uses the infix operator `` Primitive `|` Optional[Primitive] `` from `Allows`:

```scala

def writeCsv[A](rows: Seq[A])(implicit
  ev: Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???
```

  </TabItem>
  <TabItem value="scala3" label="Scala 3">

Uses native union type syntax:

```scala

def writeCsv[A](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???
```

  </TabItem>
</Tabs>

Both spellings compile and produce the same semantic behavior. The grammar is identical — the only difference is how the union type is expressed.

## Use Cases

### Flat Record (CSV, RDBMS Row)

```scala

// Flat record: only primitives and optional primitives allowed
def writeCsv[A: Schema](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???

// RDBMS INSERT: primitives, optional primitives, or string-keyed maps (JSONB)
def insert[A: Schema](value: A)(using
  Allows[A, Record[Primitive | Optional[Primitive] | Allows.Map[Primitive, Primitive]]]
): String = ???
```

If a user passes a type with nested records, they get a precise compile-time error like this:

```
[error] Schema shape violation at UserWithAddress.address: found Record(Address),
        required Primitive | Optional[Primitive] | Map[Primitive, Primitive]
```

### Event Bus / Message Broker

Published events are typically sealed traits of flat record cases. No `Variant` node is needed — sealed traits are automatically unwrapped:

```scala

// DomainEvent is a sealed trait; its cases must each satisfy Record[Primitive | Sequence[Primitive]]
def publish[A: Schema](event: A)(using
  Allows[A, Record[Primitive | Optional[Primitive] | Sequence[Primitive]]]
): Unit = ???
```

If a case of the sealed trait has a nested record field, the error names that case and field like this:

```
[error] Schema shape violation at DomainEvent.OrderPlaced.items.<element>:
        found Record(OrderItem), required Primitive | Optional[Primitive] | Sequence[Primitive]
```

### JSON Document Store (Recursive)

A document store accepts arbitrarily nested records but not `DynamicValue` leaves. The `Self` node expresses the recursive grammar:

```scala

type JsonDocument =
  Record[Primitive | Self | Optional[Primitive | Self] | Sequence[Primitive | Self] | Allows.Map[Primitive, Primitive | Self]]

def toJson[A: Schema](doc: A)(using Allows[A, JsonDocument]): String = ???
```

This grammar allows:
- `case class Author(name: String, email: String)` — Record[Primitive] ✓
- `case class Book(title: String, author: Author, tags: List[String])` — Record with Self-nested record and Sequence[Primitive] ✓
- `case class Category(name: String, subcategories: List[Category])` — recursive ✓

But rejects:
- `case class Bad(name: String, payload: DynamicValue)` — DynamicValue is not in the grammar ✗

### GraphQL / Tree Structures (Self)

```scala

def graphqlType[A: Schema]()(using
  Allows[A, Record[Primitive | Optional[Self] | Sequence[Self]]]
): String = ???

// Works:
case class TreeNode(value: Int, children: List[TreeNode])
object TreeNode { implicit val schema: Schema[TreeNode] = Schema.derived }
// graphqlType[TreeNode]() — compiles fine
```

## Sequence Subtypes

The `Sequence[A]` node accepts any collection type. When a DSL needs to restrict to a specific kind of collection — for example, a DynamoDB `Set` operation that is only valid on sets, not lists — use the `Sequence` subtypes:

```scala

// Only an immutable List is accepted
val listOnly: Allows[List[Int], Sequence.List[Primitive]] = implicitly

// Only an immutable Set is accepted
val setOnly: Allows[Set[Int], Sequence.Set[Primitive]] = implicitly

// Only a Vector
val vecOnly: Allows[Vector[String], Sequence.Vector[Primitive]] = implicitly

// Only an Array
val arrOnly: Allows[Array[Int], Sequence.Array[Primitive]] = implicitly

// Only a Chunk

val chkOnly: Allows[Chunk[String], Sequence.Chunk[Primitive]] = implicitly
```

Each subtype extends `Sequence[A]`, so a grammar written with the parent `Sequence` still accepts all collection kinds. A grammar written with a subtype rejects other kinds at compile time:

```
// Set[Int] does NOT satisfy Sequence.List[Primitive] — compile error:
[error] Shape violation at Set: found Sequence[scala.Int], required Sequence.List[...]
val bad: Allows[Set[Int], Sequence.List[Primitive]] = implicitly
```

### DynamoDB-style set operations

A DynamoDB grammar can encode the distinction between set types and list types exactly, without any additional runtime proof. We use `Sequence.Set` to narrow the grammar to sets-only operations:

```scala

type N  = Primitive.Int | Primitive.Long | Primitive.Float | Primitive.Double | Primitive.Short
type S  = Primitive.String
type NS = Sequence.Set[N | Wrapped[N]]
type SS = Sequence.Set[S | Wrapped[S]]

// addSet is only callable with a Set — List[Int] or Vector[Int] would fail at compile time
def addSet[A](set: scala.collection.immutable.Set[A])(implicit
  ev: Allows[scala.collection.immutable.Set[A], NS | SS]
): String = "ok"
```

## `IsType[A]`

`IsType[A]` is a nominal type predicate. It is satisfied only when the checked Scala type is exactly `A` (i.e. `checked =:= A`). It is most useful as an element constraint inside `Sequence` subtypes, where it aligns the element type of a collection with a polymorphic method type parameter.

The primary motivation (GitHub issue #1172) is DSL methods that must constrain both the container kind and the element type in a single `Allows` expression. Without `IsType`, a separate type class (like `Containable`) is needed to connect `A` in `contains[A]` to the element type of the collection. With `IsType`, the connection is expressed directly in the grammar.

To use `IsType[A]` with a polymorphic `A`, require `IsNominalType[A]` from `zio-blocks-typeid` at the call site. This ensures the macro always sees a concrete type when it evaluates `IsType[A]`:

```scala

// `To` must be a Set whose element type is exactly `A`.
// IsNominalType[A] ensures A is concrete at the call site — an unresolved
// type parameter would fail to produce IsNominalType and the call site
// would not compile.
def contains[To, From, A: IsNominalType](a: A)(implicit
  ev: Allows[To, Sequence.Set[IsType[A]]]
): Boolean = ev.ne(null)

// Compiles: Set[Int] satisfies Sequence.Set[IsType[Int]]
val r1: Boolean = contains[Set[Int], Nothing, Int](42)

// Compiles: Set[String] satisfies Sequence.Set[IsType[String]]
val r2: Boolean = contains[Set[String], Nothing, String]("hello")
```

A mismatch between the element type and `A` is a compile-time error:

```
[error] Shape violation at ...<element>: found Primitive(java.lang.String),
        required IsType[Int]
```

`IsType[A]` can also appear as a standalone constraint, or anywhere a grammar node is accepted:

```scala

// Int satisfies IsType[Int] exactly
val ev: Allows[Int, Allows.IsType[Int]] = implicitly

// List[String] satisfies Sequence[IsType[String]]
val ev2: Allows[List[String], Allows.Sequence[Allows.IsType[String]]] = implicitly
```

## The `Self` Grammar Node

`Self` refers back to the entire enclosing `Allows[A, S]` grammar. It allows the grammar to describe recursive data structures.

**Non-recursive types** satisfy `Self`-containing grammars without issue: if no field ever recurses back to the root type, the `Self` position is never reached, and the constraint is vacuously satisfied.

**Mutual recursion** between two or more distinct types is a compile-time error reported as:

```
[error] Mutually recursive types are not supported by Allows.
        Cycle: Forest -> Tree -> Forest
```

## `Wrapped[A]` and Newtypes

The `Wrapped[A]` node matches ZIO Prelude `Newtype` and `Subtype` wrappers. The underlying type must satisfy `A`. Here's an example:

```scala

// ZIO Prelude Newtype pattern:
object ProductCode extends Newtype[String]
type ProductCode = ProductCode.Type

given Schema[ProductCode] =
  Schema[String].transform(_.asInstanceOf[ProductCode], _.asInstanceOf[String])

// ProductCode satisfies Wrapped[Primitive] — its underlying String is Primitive
val ev: Allows[ProductCode, Wrapped[Primitive]] = implicitly
```

**Scala 3 opaque types** are resolved to their underlying type by the macro (they are transparent), so an opaque alias like this satisfies `Primitive` directly:

```scala
opaque type UserId = java.util.UUID
```

## Sealed Traits and Enums (Auto-Unwrap)

Sealed traits and enums are **automatically unwrapped** by the macro. Whenever a sealed type is encountered at any grammar check position, the macro recursively checks every case against the same grammar. This makes a `Variant` grammar node unnecessary.

```scala

sealed trait Shape
case class Circle(radius: Double)                    extends Shape
case class Rectangle(width: Double, height: Double)  extends Shape
case object Point                                    extends Shape

// No Variant node — Shape is auto-unwrapped, all cases checked against Record[Primitive]
val ev: Allows[Shape, Record[Primitive]] = implicitly
```

Auto-unwrap is recursive: if a case is itself a sealed trait, its cases are unwrapped too, to any depth.

Union branches (`A | B`) work naturally with auto-unwrap: unused branches are fine under `Allows` upper-bound semantics.

## Error Messages

When a type does not satisfy the grammar, the macro reports:

1. **The path** to the violating field: `Order.items.<element>`
2. **What was found**: `Record(OrderItem)`
3. **What was required**: `Primitive | Sequence[Primitive]`
4. **A hint** where applicable

Multiple violations are reported in a single compilation pass — the user sees all problems at once, for example:

```
[error] Schema shape violation at UserWithAddress.address: found Record(Address),
        required Primitive | Optional[Primitive] | Map[Primitive, Primitive]
[error]   Hint: Type 'Address' does not match any allowed shape
```

## Singleton / Zero-Field Records

`Record[A]` is vacuously true for case objects and zero-field records, since there are no fields to violate the constraint:

```scala

case object EmptyEvent
implicit val schema: Schema[EmptyEvent.type] = Schema.derived

val ev: Allows[EmptyEvent.type, Record[Primitive]] = implicitly  // vacuously true
```

## Runtime Cost

`Allows[A, S]` carries **zero runtime overhead**. The macro emits a reference to a single private singleton `Allows.instance` cast to the required type. There is no per-call-site allocation.

## Scala 2 vs Scala 3

| Feature | Scala 2 | Scala 3 |
|---|---|---|
| Union syntax | `` A `\|` B `` infix | `A \| B` native union type |
| Summon syntax | `implicitly[Allows[A, S]]` | `summon[Allows[A, S]]` or `implicitly` |
| Evidence parameter | `(implicit ev: Allows[A, S])` | `(using Allows[A, S])` |
| Opaque type detection | ZIO Prelude only | Scala 3 opaque types + ZIO Prelude + neotype |
| Derivation keyword | `Schema.derived` implicit | `Schema.derived` or `derives Schema` |

Both Scala versions produce the same macro behavior and the same error messages.

## Integration with Schema

`Allows` and `Schema` are complementary but independent:

- **`Schema[A]`** describes what an `A` looks like at runtime — how to serialize, deserialize, introspect, or transform it. It requires explicit derivation and handles the full type signature.
- **`Allows[A, S]`** describes what an `A` *may* look like at compile time — a structural grammar that `A` must satisfy. It requires no schema and uses only the Scala type system.

You can use `Allows` **without** `Schema`:

```scala

// Pure shape constraint, no Schema required
def writeCsv[A](rows: Seq[A])(using Allows[A, Record[Primitive | Optional[Primitive]]]): Unit = ???
```

Or combine them when runtime encoding **and** shape validation are both needed:

```scala

// Shape constraint + runtime encoding
def writeCsv[A: Schema](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???
```

When combined, `Allows` enforces the structural guarantee that `Schema` can use — for example, a CSV serializer can assume that every field is a primitive or optional primitive and skip defensive type checks.

See [Schema](./schema.md) for more on runtime encoding and decoding with schemas.

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

**CSV serializer with flat record compile-time constraints**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsCsvExample.scala))

```bash
sbt "schema-examples/runMain comptime.AllowsCsvExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsCsvExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package comptime

// ---------------------------------------------------------------------------
// CSV serializer example using Allows[A, S] compile-time shape constraints
//
// A CSV row is a flat record: every field must be a primitive scalar or an
// optional primitive (for nullable columns). Nested records, sequences, and
// maps are all rejected at compile time.
// ---------------------------------------------------------------------------

// Compatible: flat record of primitives and optional primitives
case class Employee(name: String, department: String, salary: BigDecimal, active: Boolean)
object Employee { implicit val schema: Schema[Employee] = Schema.derived }

case class SensorReading(sensorId: String, timestamp: Long, value: Double, unit: Option[String])
object SensorReading { implicit val schema: Schema[SensorReading] = Schema.derived }

object CsvSerializer {

  type FlatRow = Primitive | AOptional[Primitive]

  /** Serialize a sequence of flat records to CSV format. */
  def toCsv[A](rows: Seq[A])(implicit schema: Schema[A], ev: Allows[A, Record[FlatRow]]): String = {
    val reflect = schema.reflect.asRecord.get
    val header  = reflect.fields.map(_.name).mkString(",")
    val lines   = rows.map { row =>
      val dv = schema.toDynamicValue(row)
      dv match {
        case DynamicValue.Record(fields) =>
          fields.map { case (_, v) => csvEscape(dvToString(v)) }.mkString(",")
        case _ => ""
      }
    }
    (header +: lines).mkString("\n")
  }

  private def dvToString(dv: DynamicValue): String = dv match {
    case DynamicValue.Primitive(PrimitiveValue.String(s))     => s
    case DynamicValue.Primitive(PrimitiveValue.Boolean(b))    => b.toString
    case DynamicValue.Primitive(PrimitiveValue.Int(n))        => n.toString
    case DynamicValue.Primitive(PrimitiveValue.Long(n))       => n.toString
    case DynamicValue.Primitive(PrimitiveValue.Double(n))     => n.toString
    case DynamicValue.Primitive(PrimitiveValue.Float(n))      => n.toString
    case DynamicValue.Primitive(PrimitiveValue.BigDecimal(n)) => n.toString
    case DynamicValue.Primitive(v)                            => v.toString
    case DynamicValue.Null                                    => ""
    case DynamicValue.Variant(tag, inner) if tag == "Some"    => dvToString(inner)
    case DynamicValue.Variant(tag, _) if tag == "None"        => ""
    case DynamicValue.Record(fields)                          =>
      fields.headOption.map { case (_, v) => dvToString(v) }.getOrElse("")
    case other => other.toString
  }

  private def csvEscape(s: String): String =
    if (s.contains(",") || s.contains("\"") || s.contains("\n"))
      "\"" + s.replace("\"", "\"\"") + "\""
    else s
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsCsvExample extends App {

  // Flat records of primitives — compiles fine
  val employees = Seq(
    Employee("Alice", "Engineering", BigDecimal("120000.00"), true),
    Employee("Bob", "Marketing", BigDecimal("95000.50"), true),
    Employee("Carol", "Engineering", BigDecimal("115000.00"), false)
  )

  // CSV output for a flat record of primitives
  show(CsvSerializer.toCsv(employees))

  // Flat record with optional fields — also compiles
  val readings = Seq(
    SensorReading("temp-01", 1709712000L, 23.5, Some("celsius")),
    SensorReading("temp-02", 1709712060L, 72.1, None)
  )

  // Optional fields become empty CSV cells when None
  show(CsvSerializer.toCsv(readings))

  // The following would NOT compile — uncomment to see the error:
  //
  // case class Nested(name: String, address: Address)
  // object Nested { implicit val schema: Schema[Nested] = Schema.derived }
  // CsvSerializer.toCsv(Seq(Nested("Alice", Address("1 Main St", "NY", "10001"))))
  //   [error] Schema shape violation at Nested.address: found Record(Address),
  //           required Primitive | Optional[Primitive]
}
```

**Event bus with sealed trait auto-unwrap and nested hierarchies**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsEventBusExample.scala))

```bash
sbt "schema-examples/runMain comptime.AllowsEventBusExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsEventBusExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package comptime

// ---------------------------------------------------------------------------
// Event bus / message broker example using Allows[A, S]
//
// Published events are typically sealed traits of flat record cases. Sealed
// traits are automatically unwrapped by the Allows macro — each case is
// checked individually against the grammar. No Variant node is needed.
//
// This example also shows nested sealed traits (auto-unwrap is recursive).
// ---------------------------------------------------------------------------

// Domain events — a sealed trait hierarchy
sealed trait AccountEvent
case class AccountOpened(accountId: String, owner: String, initialBalance: BigDecimal) extends AccountEvent
case class FundsDeposited(accountId: String, amount: BigDecimal)                       extends AccountEvent
case class FundsWithdrawn(accountId: String, amount: BigDecimal)                       extends AccountEvent
case class AccountClosed(accountId: String, reason: Option[String])                    extends AccountEvent
object AccountEvent { implicit val schema: Schema[AccountEvent] = Schema.derived }

// Nested sealed trait — InventoryEvent has a sub-hierarchy
sealed trait InventoryEvent
case class ItemAdded(sku: String, quantity: Int)   extends InventoryEvent
case class ItemRemoved(sku: String, quantity: Int) extends InventoryEvent

sealed trait InventoryAlert                      extends InventoryEvent
case class LowStock(sku: String, remaining: Int) extends InventoryAlert
case class OutOfStock(sku: String)               extends InventoryAlert

object InventoryEvent { implicit val schema: Schema[InventoryEvent] = Schema.derived }

// Event with sequence fields (e.g. tags or batch items)
sealed trait BatchEvent
case class BatchImport(batchId: String, itemIds: List[String]) extends BatchEvent
case class BatchComplete(batchId: String, count: Int)          extends BatchEvent
object BatchEvent { implicit val schema: Schema[BatchEvent] = Schema.derived }

object EventBus {

  type EventShape = Primitive | AOptional[Primitive]

  /**
   * Publish a domain event. All cases of the sealed trait must be flat records.
   */
  def publish[A](event: A)(implicit schema: Schema[A], ev: Allows[A, Record[EventShape]]): String = {
    val dv                  = schema.toDynamicValue(event)
    val (typeName, payload) = dv match {
      case DynamicValue.Variant(name, inner) => (name, inner.toJson.toString)
      case _                                 => (schema.reflect.typeId.name, dv.toJson.toString)
    }
    s"PUBLISH topic=${schema.reflect.typeId.name} type=$typeName payload=$payload"
  }

  /**
   * Publish events that may contain sequence fields (e.g. batch operations).
   */
  def publishBatch[A](event: A)(implicit
    schema: Schema[A],
    ev: Allows[A, Record[Primitive | Sequence[Primitive]]]
  ): String = {
    val dv                  = schema.toDynamicValue(event)
    val (typeName, payload) = dv match {
      case DynamicValue.Variant(name, inner) => (name, inner.toJson.toString)
      case _                                 => (schema.reflect.typeId.name, dv.toJson.toString)
    }
    s"PUBLISH topic=${schema.reflect.typeId.name} type=$typeName payload=$payload"
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsEventBusExample extends App {

  // Flat sealed trait — all cases are records of primitives/optionals
  show(EventBus.publish[AccountEvent](AccountOpened("acc-001", "Alice", BigDecimal("1000.00"))))
  show(EventBus.publish[AccountEvent](FundsDeposited("acc-001", BigDecimal("500.00"))))
  show(EventBus.publish[AccountEvent](AccountClosed("acc-001", Some("customer request"))))

  // Nested sealed trait — auto-unwrap is recursive
  // InventoryAlert extends InventoryEvent, both are unwrapped
  show(EventBus.publish[InventoryEvent](ItemAdded("SKU-100", 50)))
  show(EventBus.publish[InventoryEvent](LowStock("SKU-100", 3)))
  show(EventBus.publish[InventoryEvent](OutOfStock("SKU-100")))

  // Events with sequence fields use a wider grammar
  show(EventBus.publishBatch[BatchEvent](BatchImport("batch-42", List("item-1", "item-2", "item-3"))))
  show(EventBus.publishBatch[BatchEvent](BatchComplete("batch-42", 3)))
}
```

**GraphQL / tree structures using Self for recursive grammars**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsGraphQLTreeExample.scala))

```bash
sbt "schema-examples/runMain comptime.AllowsGraphQLTreeExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsGraphQLTreeExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package comptime

// ---------------------------------------------------------------------------
// GraphQL / tree structure example using Self for recursive grammars
//
// Self refers back to the entire enclosing Allows[A, S] grammar, allowing
// the constraint to describe recursive data structures like trees, linked
// lists, and nested menus.
//
// Non-recursive types also satisfy Self-containing grammars — the Self
// position is never reached, so the constraint is vacuously satisfied.
// ---------------------------------------------------------------------------

// Recursive tree: children reference the same type
case class TreeNode(value: Int, children: List[TreeNode])
object TreeNode { implicit val schema: Schema[TreeNode] = Schema.derived }

// Recursive category hierarchy (common in e-commerce, CMS, etc.)
case class NavCategory(name: String, slug: String, subcategories: List[NavCategory])
object NavCategory { implicit val schema: Schema[NavCategory] = Schema.derived }

// Linked list via Optional[Self]
case class Chain(label: String, next: Option[Chain])
object Chain { implicit val schema: Schema[Chain] = Schema.derived }

// Non-recursive type — satisfies Self-containing grammars vacuously
case class FlatNode(id: Int, label: String)
object FlatNode { implicit val schema: Schema[FlatNode] = Schema.derived }

object GraphQL {

  type TreeShape = Primitive | Sequence[ASelf] | AOptional[ASelf]

  /** Generate a simplified GraphQL type definition for a recursive type. */
  def graphqlType[A](implicit schema: Schema[A], ev: Allows[A, Record[TreeShape]]): String = {
    val reflect = schema.reflect.asRecord.get
    val fields  = reflect.fields.map { f =>
      s"  ${f.name}: ${gqlType(resolve(f.value), schema.reflect.typeId.name)}"
    }
    s"type ${schema.reflect.typeId.name} {\n${fields.mkString("\n")}\n}"
  }

  /** Unwrap Deferred to get the actual Reflect node. */
  private def resolve(r: Reflect.Bound[_]): Reflect.Bound[_] = r match {
    case d: Reflect.Deferred[_, _] => resolve(d.value.asInstanceOf[Reflect.Bound[_]])
    case other                     => other
  }

  private def gqlType(r: Reflect.Bound[_], selfName: String): String = r match {
    case _: Reflect.Sequence[_, _, _] => s"[$selfName]"
    case p: Reflect.Primitive[_, _]   =>
      p.primitiveType match {
        case PrimitiveType.Int(_)     => "Int"
        case PrimitiveType.Long(_)    => "Int"
        case PrimitiveType.Float(_)   => "Float"
        case PrimitiveType.Double(_)  => "Float"
        case PrimitiveType.String(_)  => "String"
        case PrimitiveType.Boolean(_) => "Boolean"
        case _                        => "String"
      }
    case _ => selfName
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsGraphQLTreeExample extends App {

  // Recursive tree with Sequence[Self]
  show(GraphQL.graphqlType[TreeNode])

  // Recursive categories — same grammar, different domain
  show(GraphQL.graphqlType[NavCategory])

  // Linked list via Optional[Self]
  show(GraphQL.graphqlType[Chain])

  // Non-recursive type also satisfies the grammar (vacuously — Self is never reached)
  show(GraphQL.graphqlType[FlatNode])

  // Show that recursive data actually works at runtime
  val tree = TreeNode(
    1,
    List(
      TreeNode(2, List(TreeNode(4, Nil), TreeNode(5, Nil))),
      TreeNode(3, Nil)
    )
  )
  show(Schema[TreeNode].toDynamicValue(tree).toJson.toString)

  val nav = NavCategory(
    "Electronics",
    "electronics",
    List(
      NavCategory("Phones", "phones", Nil),
      NavCategory(
        "Laptops",
        "laptops",
        List(
          NavCategory("Gaming", "gaming", Nil)
        )
      )
    )
  )
  show(Schema[NavCategory].toDynamicValue(nav).toJson.toString)
}
```

**Sealed trait auto-unwrap with nested hierarchies and case objects**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/AllowsSealedTraitExample.scala))

```bash
sbt "schema-examples/runMain comptime.AllowsSealedTraitExample"
```

```scala title="schema-examples/src/main/scala/comptime/AllowsSealedTraitExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package comptime

// ---------------------------------------------------------------------------
// Sealed trait auto-unwrap example
//
// Sealed traits and enums are automatically unwrapped by the Allows macro.
// Each case is checked individually against the grammar — no Variant node
// is needed.
//
// Auto-unwrap is recursive: if a case is itself a sealed trait, its cases
// are unwrapped too, to any depth.
//
// Zero-field records (case objects) are vacuously true for any Record[A]
// constraint.
// ---------------------------------------------------------------------------

// Simple sealed trait with case classes and a case object
sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape
case object Point                                   extends Shape
object Shape { implicit val schema: Schema[Shape] = Schema.derived }

// Nested sealed trait hierarchy — two levels deep
sealed trait Expr
sealed trait BinaryOp                            extends Expr
case class Add(left: Double, right: Double)      extends BinaryOp
case class Multiply(left: Double, right: Double) extends BinaryOp
case class Literal(value: Double)                extends Expr
case object Zero                                 extends Expr
object Expr { implicit val schema: Schema[Expr] = Schema.derived }

// All-singleton enum (all case objects)
sealed trait Color
case object Red   extends Color
case object Green extends Color
case object Blue  extends Color
object Color { implicit val schema: Schema[Color] = Schema.derived }

object SealedTraitValidator {

  /** Validate that a value's type has a flat record structure. */
  def validate[A](value: A)(implicit schema: Schema[A], ev: Allows[A, Record[Primitive]]): String = {
    val dv = schema.toDynamicValue(value)
    dv match {
      case DynamicValue.Variant(caseName, inner) =>
        s"Valid variant case '$caseName': ${inner.toJson}"
      case DynamicValue.Record(fields) =>
        s"Valid record with ${fields.size} field(s): ${fields.map(_._1).mkString(", ")}"
      case _ =>
        s"Valid: ${dv.toJson}"
    }
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object AllowsSealedTraitExample extends App {

  // Simple sealed trait — all cases checked against Record[Primitive]
  // Circle: Record(radius: Double) — satisfies Record[Primitive]
  // Rectangle: Record(width: Double, height: Double) — satisfies Record[Primitive]
  // Point: zero-field case object — vacuously true
  show(SealedTraitValidator.validate[Shape](Circle(3.14)))
  show(SealedTraitValidator.validate[Shape](Rectangle(4.0, 5.0)))
  show(SealedTraitValidator.validate[Shape](Point))

  // Nested sealed trait — auto-unwrap is recursive
  // BinaryOp is itself sealed with Add and Multiply
  // All leaf cases have only Double fields — satisfies Record[Primitive]
  show(SealedTraitValidator.validate[Expr](Add(1.0, 2.0)))
  show(SealedTraitValidator.validate[Expr](Multiply(3.0, 4.0)))
  show(SealedTraitValidator.validate[Expr](Literal(42.0)))
  show(SealedTraitValidator.validate[Expr](Zero))

  // All-singleton enum — every case is a zero-field record (vacuously true)
  show(SealedTraitValidator.validate[Color](Red))
  show(SealedTraitValidator.validate[Color](Green))
  show(SealedTraitValidator.validate[Color](Blue))
}
```

**RDBMS library with CREATE TABLE and INSERT using flat record constraints** (compile-only)
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/RdbmsExample.scala))

Demonstrates how Allows constraints are verified at compile time — the code below shows valid examples that compile successfully, and includes comments showing which patterns would be rejected:

```scala title="schema-examples/src/main/scala/comptime/RdbmsExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package comptime

// ---------------------------------------------------------------------------
// Realistic RDBMS example using Allows[A, S] compile-time shape constraints
//
// Demonstrates how a library can require that user-supplied types have a
// structure compatible with what a relational database can represent:
//   - Flat records of primitives, optional primitives, or primitive-valued maps
//   - Top-level variants (enum tables) whose cases are flat records
//
// Incompatible types (nested records, sequences of records, etc.) are
// rejected at the call site with a precise compile-time error message.
// ---------------------------------------------------------------------------

// ---------------------------------------------------------------------------
// Domain types — vary from compatible to incompatible
// ---------------------------------------------------------------------------

// Compatible: flat record of primitives and optional primitives
case class UserRow(
  id: java.util.UUID,
  name: String,
  email: Option[String],
  age: Int,
  active: Boolean
)
object UserRow {
  implicit val schema: Schema[UserRow] = Schema.derived
}

// Compatible: flat record with a string-keyed map column (stored as JSON/JSONB)
case class ProductRow(
  id: java.util.UUID,
  name: String,
  price: BigDecimal,
  attributes: scala.collection.immutable.Map[String, String]
)
object ProductRow {
  implicit val schema: Schema[ProductRow] = Schema.derived
}

// Compatible: event table — a variant (sealed trait) of flat record cases
sealed trait DomainEvent
case class UserCreated(id: java.util.UUID, name: String, email: String)               extends DomainEvent
case class UserDeleted(id: java.util.UUID)                                            extends DomainEvent
case class OrderPlaced(id: java.util.UUID, userId: java.util.UUID, total: BigDecimal) extends DomainEvent
object DomainEvent {
  implicit val schema: Schema[DomainEvent] = Schema.derived
}

// Incompatible: contains a nested record (Address is not a primitive)
case class Address(street: String, city: String, zip: String)
object Address { implicit val schema: Schema[Address] = Schema.derived }

case class UserWithAddress(
  id: java.util.UUID,
  name: String,
  address: Address // ← incompatible: nested record
)
object UserWithAddress {
  implicit val schema: Schema[UserWithAddress] = Schema.derived
}

// ---------------------------------------------------------------------------
// Simulated RDBMS library API
//
// The Allows constraint is checked at the CALL SITE — the library author
// writes these signatures once. Users get a compile-time error if their type
// doesn't match, with a message pointing to the exact violating field.
// ---------------------------------------------------------------------------

object Rdbms {

  // A flat record row: primitives, optional primitives, or string-keyed maps
  type FlatRow = Primitive | AOptional[Primitive] | AMap[Primitive, Primitive]

  /** Generate a CREATE TABLE DDL statement for a flat record type. */
  def createTable[A](implicit
    schema: Schema[A],
    ev: Allows[A, Record[FlatRow]]
  ): String = {
    val fields = schema.reflect.asRecord.get.fields
    val cols   = fields.map { f =>
      val tpe = sqlType(f.value)
      s"  ${f.name} $tpe"
    }
    s"CREATE TABLE ${tableName(schema)} (\n${cols.mkString(",\n")}\n)"
  }

  /** Generate an INSERT statement for a single flat record row. */
  def insert[A](value: A)(implicit
    schema: Schema[A],
    ev: Allows[A, Record[FlatRow]]
  ): String = {
    val dv = schema.toDynamicValue(value)
    dv match {
      case DynamicValue.Record(fields) =>
        val cols = fields.map(_._1).mkString(", ")
        val vals = fields.map { case (_, v) => sqlLiteralDv(v) }.mkString(", ")
        s"INSERT INTO ${tableName(schema)} ($cols) VALUES ($vals)"
      case _ => s"INSERT INTO ${tableName(schema)} VALUES (?)"
    }
  }

  /**
   * Insert an event into an event-sourcing table.
   *
   * The type must be a sealed trait / enum whose cases are flat records. No
   * explicit Variant node is needed — sealed traits are auto-unwrapped by the
   * macro.
   */
  def insertEvent[A](event: A)(implicit
    schema: Schema[A],
    ev: Allows[A, Record[FlatRow]]
  ): String = {
    val typeName = schema.toDynamicValue(event) match {
      case DynamicValue.Variant(caseName, _) => caseName
      case _                                 => schema.reflect.typeId.name
    }
    val payload = schema.toDynamicValue(event).toJson.toString
    s"INSERT INTO events (type, payload) VALUES ('$typeName', '$payload')"
  }

  // ---------------------------------------------------------------------------
  // Helpers
  // ---------------------------------------------------------------------------

  private def tableName[A](schema: Schema[A]): String =
    schema.reflect.modifiers.collectFirst {
      case Modifier.config(k, v) if k == "sql.table_name" => v
    }.getOrElse(schema.reflect.typeId.name.toLowerCase + "s")

  private def sqlLiteralDv(dv: DynamicValue): String = dv match {
    case DynamicValue.Primitive(PrimitiveValue.String(s))  => s"'${s.replace("'", "''")}'"
    case DynamicValue.Primitive(PrimitiveValue.Boolean(b)) => if (b) "TRUE" else "FALSE"
    case DynamicValue.Primitive(v)                         => v.toString
    case DynamicValue.Null                                 => "NULL"
    case other                                             => s"'${other.toString.replace("'", "''")}'"
  }

  private def sqlType(reflect: Reflect.Bound[_]): String = reflect match {
    case p: Reflect.Primitive[_, _] =>
      p.primitiveType match {
        case PrimitiveType.Int(_)           => "INTEGER"
        case PrimitiveType.Long(_)          => "BIGINT"
        case PrimitiveType.String(_)        => "TEXT"
        case PrimitiveType.Boolean(_)       => "BOOLEAN"
        case PrimitiveType.Double(_)        => "DOUBLE PRECISION"
        case PrimitiveType.Float(_)         => "REAL"
        case PrimitiveType.BigDecimal(_)    => "NUMERIC"
        case PrimitiveType.UUID(_)          => "UUID"
        case PrimitiveType.Instant(_)       => "TIMESTAMPTZ"
        case PrimitiveType.LocalDate(_)     => "DATE"
        case PrimitiveType.LocalDateTime(_) => "TIMESTAMP"
        case _                              => "TEXT"
      }
    case _: Reflect.Map[_, _, _, _]   => "JSONB"
    case _: Reflect.Sequence[_, _, _] => "TEXT[]"
    case _                            => "TEXT"
  }
}

// ---------------------------------------------------------------------------
// Demonstration — these all compile
// ---------------------------------------------------------------------------

object RdbmsDemo {

  // Flat rows compile fine
  val createUser: String    = Rdbms.createTable[UserRow]
  val createProduct: String = Rdbms.createTable[ProductRow]
  val insertUser: String    = Rdbms.insert(UserRow(new java.util.UUID(0, 0), "Alice", Some("a@b.com"), 30, true))
  val insertEvent: String   = Rdbms.insertEvent[DomainEvent](UserCreated(new java.util.UUID(0, 0), "Alice", "a@b.com"))

  // The following would NOT compile — uncomment to see the error:
  //
  // val bad = Rdbms.createTable[UserWithAddress]
  //   [error] Schema shape violation at UserWithAddress.address: found Record(Address), required
  //           Primitive | Optional[Primitive] | Map[Primitive, Primitive]
}
```

**JSON document store with specific primitives and recursive Self grammar** (compile-only)
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/comptime/DocumentStoreExample.scala))

Demonstrates how Allows enforces recursive schema constraints at compile time:

```scala title="schema-examples/src/main/scala/comptime/DocumentStoreExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package comptime

// ---------------------------------------------------------------------------
// Realistic JSON document-store example using Allows[A, S]
//
// JSON has a limited primitive value set:
//   - null    → Unit / Option
//   - boolean → Boolean
//   - number  → Int, Long, Float, Double, BigDecimal, BigInt
//   - string  → String
//
// Notably absent from JSON: Char, Byte, Short, UUID, Currency,
// and ALL java.time.* types. A JSON library should use SPECIFIC primitive
// nodes to reject non-JSON scalars at the call site.
//
// The grammar is simply:
//   type Json = Record[JsonPrimitive | Self] | Sequence[JsonPrimitive | Self]
//
// Self handles all nesting: a field may be a primitive or another Json value.
// Sequences of primitives (List[String]) and sequences of records
// (List[Author]) both satisfy Sequence[JsonPrimitive | Self].
// Optional fields (Option[X]) satisfy Record[...] because Option is recognised
// by the Optional grammar node which falls through to the field constraint.
// ---------------------------------------------------------------------------

// ---------------------------------------------------------------------------
// Domain types
// ---------------------------------------------------------------------------

// Compatible: flat document
case class Author(name: String, email: String)
object Author { implicit val schema: Schema[Author] = Schema.derived }

// Compatible: document with nested documents and sequences
case class BookChapter(title: String, wordCount: Int)
object BookChapter { implicit val schema: Schema[BookChapter] = Schema.derived }

case class Book(
  title: String,
  author: Author,              // nested record — satisfied via Self
  chapters: List[BookChapter], // sequence of records — satisfied via Sequence[Self]
  tags: List[String],          // sequence of primitives — satisfied via Sequence[JsonPrimitive]
  rating: Option[Double]       // optional primitive — satisfied via Optional[JsonPrimitive | Self]
)
object Book { implicit val schema: Schema[Book] = Schema.derived }

// Compatible: recursive document
case class Category(name: String, subcategories: List[Category])
object Category { implicit val schema: Schema[Category] = Schema.derived }

// Compatible: variant of search results (for indexing)
sealed trait SearchResult
case class BookResult(title: String, score: Double)   extends SearchResult
case class AuthorResult(name: String, bookCount: Int) extends SearchResult
object SearchResult { implicit val schema: Schema[SearchResult] = Schema.derived }

// INCOMPATIBLE: UUID is not a JSON-native scalar
case class WithUUID(id: java.util.UUID, name: String)
object WithUUID { implicit val schema: Schema[WithUUID] = Schema.derived }

// INCOMPATIBLE: Instant is not a JSON-native scalar
case class WithTimestamp(name: String, createdAt: java.time.Instant)
object WithTimestamp { implicit val schema: Schema[WithTimestamp] = Schema.derived }

// ---------------------------------------------------------------------------
// Document store library API
// ---------------------------------------------------------------------------

object DocumentStore {

  /**
   * JSON-representable scalar types.
   *
   * Excludes Char, Byte, Short, UUID, Currency, and all java.time.* types —
   * none of these have a native JSON encoding. Authors who need java.time
   * values in JSON should store them as Primitive.String (ISO-8601 etc.).
   */
  type JsonPrimitive =
    Allows.Primitive.Boolean | Allows.Primitive.Int | Allows.Primitive.Long | Allows.Primitive.Float |
      Allows.Primitive.Double | Allows.Primitive.String | Allows.Primitive.BigDecimal | Allows.Primitive.BigInt |
      Allows.Primitive.Unit

  /**
   * A JSON value is either a JSON object (Record) or a JSON array (Sequence).
   * Self recurses back to this same grammar, so nesting works at any depth.
   * Optional covers nullable fields (JSON null / absent key).
   */
  type Json = Record[JsonPrimitive | AOptional[JsonPrimitive | ASelf] | ASelf] | Sequence[JsonPrimitive | ASelf]

  /**
   * Encode a value to its JSON string representation.
   *
   * Accepts both JSON objects (records) and JSON arrays (sequences) at the top
   * level. Fields may be primitives, nested documents, sequences, or optionals
   * — all handled via Self and the JsonPrimitive constraint. Types containing
   * UUID, Instant, Char etc. are rejected at compile time.
   */
  def toJson[A: Schema](doc: A)(implicit ev: Allows[A, Json]): String =
    Schema[A].toDynamicValue(doc).toJson.toString

  /** Serialize to DynamicValue for further processing. */
  def serialize[A: Schema](doc: A)(implicit ev: Allows[A, Json]): DynamicValue =
    Schema[A].toDynamicValue(doc)

  /**
   * Index a search result. The type must be a sealed trait of flat JSON
   * records. No explicit Variant node needed — sealed traits are
   * auto-unwrapped.
   */
  def index[A: Schema](result: A)(implicit
    ev: Allows[A, Record[JsonPrimitive | AOptional[JsonPrimitive | ASelf] | Sequence[JsonPrimitive | ASelf]]]
  ): String = {
    val typeName = Schema[A].toDynamicValue(result) match {
      case DynamicValue.Variant(name, _) => name
      case _                             => Schema[A].reflect.typeId.name
    }
    val payload = Schema[A].toDynamicValue(result).toJson.toString
    s"""{"_type":"$typeName","_doc":$payload}"""
  }
}

// ---------------------------------------------------------------------------
// Demonstration
// ---------------------------------------------------------------------------

object DocumentStoreDemo {

  // JSON objects compile fine
  val bookJson: String = DocumentStore.toJson(
    Book(
      "ZIO Blocks",
      Author("John", "john@example.com"),
      List(BookChapter("Intro", 1000)),
      List("scala", "zio"),
      Some(4.9)
    )
  )

  val categoryJson: String = DocumentStore.toJson(
    Category("Programming", List(Category("Scala", Nil)))
  )

  // JSON arrays also satisfy Json (top-level Sequence)
  val tagListJson: String    = DocumentStore.toJson(List("scala", "zio", "functional"))
  val authorListJson: String = DocumentStore.toJson(List(Author("Alice", "a@b.com"), Author("Bob", "b@b.com")))

  val indexed: String = DocumentStore.index[SearchResult](BookResult("ZIO Blocks", 0.99))

  // The following would NOT compile — uncomment to see the errors:
  //
  // DocumentStore.toJson(WithUUID(new java.util.UUID(0, 0), "Alice"))
  //   [error] Schema shape violation at WithUUID.id: found Primitive(java.util.UUID),
  //           required JsonPrimitive (Boolean | Int | Long | Float | Double | String | ...)
  //   UUID is not a JSON-native type — encode it as Primitive.String.
  //
  // DocumentStore.toJson(WithTimestamp("Alice", java.time.Instant.EPOCH))
  //   [error] Schema shape violation at WithTimestamp.createdAt: found Primitive(java.time.Instant)
  //   Instant is not a JSON-native type — encode it as Primitive.String (ISO-8601).
}
```

---

## BindingResolver

`BindingResolver` is the **read-only interface for looking up bindings by type identity** during schema rebinding. Given a type `A`, a resolver searches its internal storage and returns either the matching `Binding` or `None`.

```scala
trait BindingResolver {
  def resolveRecord[A](implicit typeId: TypeId[A]): Option[Binding.Record[A]]
  def resolveVariant[A](implicit typeId: TypeId[A]): Option[Binding.Variant[A]]
  def resolvePrimitive[A](implicit typeId: TypeId[A]): Option[Binding.Primitive[A]]
  def resolveWrapper[A](implicit typeId: TypeId[A]): Option[Binding.Wrapper[A, _]]
  def resolveDynamic(implicit typeId: TypeId[DynamicValue]): Option[Binding.Dynamic]
  def resolveSeq[X](implicit typeId: TypeId[X], u: UnapplySeq[X]): Option[Binding.Seq[u.C, u.A]]
  def resolveSeqFor[C[_], A](typeId: TypeId[C[A]]): Option[Binding.Seq[C, A]]
  def resolveMap[X](implicit typeId: TypeId[X], u: UnapplyMap[X]): Option[Binding.Map[u.M, u.K, u.V]]
  def resolveMapFor[M[_, _], K, V](typeId: TypeId[M[K, V]]): Option[Binding.Map[M, K, V]]
  final def ++(that: BindingResolver): BindingResolver
}
```

`BindingResolver`:

- Uses `TypeId` as the lookup key for all resolution methods.
- Stores sequence and map bindings by their *unapplied type constructor*, so one `List` binding covers `List[Int]`, `List[String]`, and any other element type.
- Composes via `BindingResolver#++` with left-biased precedence: the left resolver is tried first, the right serves as fallback.

## Motivation

ZIO Blocks separates schema **structure** from schema **behavior**:

- A `Reflect.Unbound[A]` carries only structural metadata—field names, type names, documentation. It contains no Scala functions and is fully serializable.
- A `Reflect.Bound[A]` pairs each structural node with its `Binding`, enabling actual construction and deconstruction of values.

This separation powers a key workflow: serialize a schema as a `DynamicSchema` (unbound), transmit or store it, then **rebind** it on the other side using a `BindingResolver` to recover a fully operational `Schema[A]`.

```
  Schema[Person]
       │  toDynamicSchema
       ▼
  DynamicSchema          ◄── serializable, no Scala functions
  (Reflect.Unbound[_])
       │  rebind[Person](resolver)
       ▼
  Schema[Person]         ◄── operational, can encode and decode
  (Reflect.Bound[Person])
```

The `BindingResolver` is what bridges the final step: it supplies the `Binding` for each node in the unbound reflect tree.

A minimal end-to-end example:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val dynamic: DynamicSchema = Schema[Person].toDynamicSchema

val resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[Person]) ++ BindingResolver.defaults

val rebound: Schema[Person] = dynamic.rebind[Person](resolver)
```

## Predefined Resolvers

ZIO Blocks ships three ready-made resolvers. We almost always compose them with `BindingResolver#++` rather than using them in isolation.

### `BindingResolver.empty`

`BindingResolver.empty` is an empty `Registry` with no bindings. It is the starting point for building a custom registry with `Registry#bind`:

```scala

val empty: BindingResolver.Registry = BindingResolver.empty
```

### `BindingResolver.defaults`

`BindingResolver.defaults` is a pre-populated `Registry` covering all primitive types, `java.time` types, `java.util.UUID`, `java.util.Currency`, `DynamicValue`, common sequence types (`List`, `Vector`, `Set`, `IndexedSeq`, `Seq`, `Chunk`), and `Map`. In practice we place it at the right end of a `BindingResolver#++` chain so custom bindings can override it when needed.

The types covered by `defaults` include:

| Category    | Types                                                                                                                                                                                                       |
|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Primitives  | `Unit`, `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`, `String`, `BigInt`, `BigDecimal`                                                                                             |
| `java.time` | `DayOfWeek`, `Duration`, `Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `Month`, `MonthDay`, `OffsetDateTime`, `OffsetTime`, `Period`, `Year`, `YearMonth`, `ZoneId`, `ZoneOffset`, `ZonedDateTime` |
| `java.util` | `Currency`, `UUID`                                                                                                                                                                                          |
| Dynamic     | `DynamicValue`                                                                                                                                                                                              |
| Sequences   | `List`, `Vector`, `Set`, `IndexedSeq`, `Seq`, `Chunk`                                                                                                                                                      |
| Maps        | `Map`                                                                                                                                                                                                       |

```scala

val defaults: BindingResolver.Registry = BindingResolver.defaults

defaults.resolvePrimitive[Int]               // Some(...)
defaults.resolvePrimitive[java.time.Instant] // Some(...)
defaults.resolveSeq[List[Int]]               // Some(...)
defaults.resolveMap[Map[String, Int]]        // Some(...)
```

### `BindingResolver.reflection` (JVM only)

`BindingResolver.reflection` derives `Binding.Record` instances at runtime using Java reflection for case classes. Derived bindings are cached per `TypeId` in a `ConcurrentHashMap`, so the reflection cost is paid only once per type.

On Scala.js, `BindingResolver.reflection` is a no-op resolver that returns `None` for every query.

```scala

case class Order(id: Long, item: String, quantity: Int)

object Order {
  implicit val schema: Schema[Order] = Schema.derived[Order]
}

// reflection handles the record; defaults handles Long, String, Int
val resolver: BindingResolver = BindingResolver.reflection ++ BindingResolver.defaults

val rebound: Schema[Order] = Schema[Order].toDynamicSchema.rebind[Order](resolver)
```

:::warning
`BindingResolver.reflection` only derives `Binding.Record` for case classes. It returns `None` for primitives, variants, wrappers, sequences, and maps. Always compose it with `BindingResolver.defaults` to cover those types.
:::

## Building a Registry

`BindingResolver.Registry` is an immutable, map-backed resolver. Every `Registry#bind` call returns a **new** `Registry` with the binding added; the original registry is unchanged.

### `Registry#bind` for proper types

The unified `Registry#bind` method accepts any proper binding—`Record`, `Variant`, `Primitive`, `Wrapper`, or `Dynamic`—and dispatches to the correct internal storage slot automatically. We typically call `Binding.of[A]` to derive the right binding at compile time:

```scala

sealed trait Color
case object Red  extends Color
case object Blue extends Color

object Color {
  implicit val schema: Schema[Color] = Schema.derived[Color]
}

case class Palette(primary: Color, name: String)

object Palette {
  implicit val schema: Schema[Palette] = Schema.derived[Palette]
}

val registry: BindingResolver.Registry =
  BindingResolver.empty
    .bind(Binding.of[Color])    // derives Binding.Variant[Color]
    .bind(Binding.of[Palette])  // derives Binding.Record[Palette]
```

`Binding.of[A]` is a compile-time macro that selects the appropriate binding kind based on the type of `A`:

- Case class → `Binding.Record`
- Sealed trait or enum → `Binding.Variant`
- Scalar type (`Int`, `String`, …) → `Binding.Primitive`
- Single-field wrapper with a smart constructor → `Binding.Wrapper`

:::warning
Passing a `Binding.Seq` or `Binding.Map` to the unified `Registry#bind` method throws `IllegalArgumentException` at runtime. Use the specialized overloads shown below for collection types.
:::

### `Registry#bind` for sequence types

Sequence bindings are keyed by their unapplied type constructor. We supply `[C[_]]` as the explicit type parameter so the compiler uses the constructor—not a specific applied type—as the lookup key:

```scala

val registry: BindingResolver.Registry =
  BindingResolver.empty.bind[List](Binding.Seq.list[Nothing])

// One binding resolves any element type
registry.resolveSeq[List[Int]]    // Some(...)
registry.resolveSeq[List[String]] // Some(...)
```

### `Registry#bind` for map types

Map bindings follow the same pattern with `[M[_, _]]` as the type parameter:

```scala

val registry: BindingResolver.Registry =
  BindingResolver.empty.bind[Map](Binding.Map.map[Nothing, Nothing])

registry.resolveMap[Map[String, Int]] // Some(...)
registry.resolveMap[Map[Int, String]] // Some(...)
```

## Combining Resolvers

The `BindingResolver#++` operator composes two resolvers into a left-biased fallback chain. The left resolver is consulted first; the right is used only when the left returns `None`.

```scala
trait BindingResolver {
  final def ++(that: BindingResolver): BindingResolver
}
```

This makes it straightforward to layer custom bindings over the built-in defaults:

```scala

case class UserId(value: Long)

object UserId {
  implicit val schema: Schema[UserId] =
    Schema[Long].transform(UserId(_), _.value)
}

// Custom bindings shadow matching entries in defaults
val resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[UserId]) ++ BindingResolver.defaults
```

`BindingResolver#++` is associative for resolution outcomes: `(a ++ b) ++ c` and `a ++ (b ++ c)` always resolve to the same binding for any type.

## Resolution Methods

All `resolve*` methods return an `Option` and never throw. They return `None` when no binding is registered for the requested type.

### `BindingResolver#resolveRecord`

`BindingResolver#resolveRecord` returns the `Binding.Record` for a product type (case class, tuple, module object):

```scala
trait BindingResolver {
  def resolveRecord[A](implicit typeId: TypeId[A]): Option[Binding.Record[A]]
}
```

The `TypeId[A]` witness is satisfied implicitly; we only need to supply the type parameter:

```scala

case class Point(x: Double, y: Double)

object Point {
  implicit val schema: Schema[Point] = Schema.derived[Point]
}

val registry = BindingResolver.empty.bind(Binding.of[Point])
val binding: Option[Binding.Record[Point]] = registry.resolveRecord[Point]
```

### `BindingResolver#resolveVariant`

`BindingResolver#resolveVariant` returns the `Binding.Variant` for a sum type (sealed trait, Scala 3 enum):

```scala
trait BindingResolver {
  def resolveVariant[A](implicit typeId: TypeId[A]): Option[Binding.Variant[A]]
}
```

```scala

sealed trait Shape
case class Circle(radius: Double)          extends Shape
case class Rectangle(w: Double, h: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
}

val registry = BindingResolver.empty.bind(Binding.of[Shape])
val binding: Option[Binding.Variant[Shape]] = registry.resolveVariant[Shape]
```

### `BindingResolver#resolvePrimitive`

`BindingResolver#resolvePrimitive` returns the `Binding.Primitive` for scalar types such as `Int`, `String`, `java.time.Instant`, and `java.util.UUID`:

```scala
trait BindingResolver {
  def resolvePrimitive[A](implicit typeId: TypeId[A]): Option[Binding.Primitive[A]]
}
```

```scala

val binding: Option[Binding.Primitive[Int]] = BindingResolver.defaults.resolvePrimitive[Int]
```

### `BindingResolver#resolveWrapper`

`BindingResolver#resolveWrapper` returns the `Binding.Wrapper` for newtype patterns—single-field case classes and smart-constructor wrappers. The binding holds a `wrap: B => A` and an `unwrap: A => B` function, converting between the inner type `B` and the outer type `A`:

```scala
trait BindingResolver {
  def resolveWrapper[A](implicit typeId: TypeId[A]): Option[Binding.Wrapper[A, _]]
}
```

```scala

case class Email(value: String)

object Email {
  implicit val schema: Schema[Email] =
    Schema[String].transform(Email(_), _.value)
}

val registry = BindingResolver.empty.bind(Binding.of[Email])
val binding: Option[Binding.Wrapper[Email, _]] = registry.resolveWrapper[Email]
```

### `BindingResolver#resolveDynamic`

`BindingResolver#resolveDynamic` returns the `Binding.Dynamic` singleton, used for `DynamicValue` nodes in the reflect tree. `BindingResolver.defaults` already includes it, so manual registration is rarely needed:

```scala
trait BindingResolver {
  def resolveDynamic(implicit typeId: TypeId[DynamicValue]): Option[Binding.Dynamic]
}
```

```scala

val binding: Option[Binding.Dynamic] = BindingResolver.defaults.resolveDynamic
```

### `BindingResolver#resolveSeq` and `BindingResolver#resolveSeqFor`

`BindingResolver#resolveSeq` uses `UnapplySeq` evidence to decompose an applied type like `List[Int]` into its constructor `List` and element type `Int`, then looks up the binding by constructor. `BindingResolver#resolveSeqFor` is the explicit variant when the constructor and element types are already known as separate parameters:

```scala
trait BindingResolver {
  def resolveSeq[X](implicit typeId: TypeId[X], u: UnapplySeq[X]): Option[Binding.Seq[u.C, u.A]]
  def resolveSeqFor[C[_], A](typeId: TypeId[C[A]]): Option[Binding.Seq[C, A]]
}
```

Because the binding is stored by type constructor, a single registered `List` binding handles any element type:

```scala

val defaults = BindingResolver.defaults

val listIntBinding: Option[Binding.Seq[List, Int]]    = defaults.resolveSeq[List[Int]]
val listStrBinding: Option[Binding.Seq[List, String]] = defaults.resolveSeq[List[String]]

// Explicit form using resolveSeqFor
val explicit: Option[Binding.Seq[List, Int]] =
  defaults.resolveSeqFor[List, Int](TypeId.of[List[Int]])
```

### `BindingResolver#resolveMap` and `BindingResolver#resolveMapFor`

`BindingResolver#resolveMap` and `BindingResolver#resolveMapFor` follow the same pattern as their sequence counterparts, applied to key-value collection types:

```scala
trait BindingResolver {
  def resolveMap[X](implicit typeId: TypeId[X], u: UnapplyMap[X]): Option[Binding.Map[u.M, u.K, u.V]]
  def resolveMapFor[M[_, _], K, V](typeId: TypeId[M[K, V]]): Option[Binding.Map[M, K, V]]
}
```

```scala

val binding: Option[Binding.Map[Map, String, Int]] =
  BindingResolver.defaults.resolveMap[Map[String, Int]]
```

## Registry Inspection

`Registry` exposes several methods to interrogate its state without performing a full resolution. `Registry#contains` checks whether a proper binding (Record, Variant, Primitive, Wrapper, or Dynamic) is registered for type `A`. `Registry#containsSeq` and `Registry#containsMap` check for sequence and map type constructors respectively:

```scala
final class Registry {
  def contains[A](implicit typeId: TypeId[A]): Boolean
  def containsSeq[X](implicit typeId: TypeId[X], u: UnapplySeq[X]): Boolean
  def containsMap[X](implicit typeId: TypeId[X], u: UnapplyMap[X]): Boolean
  def size: Int
  def isEmpty: Boolean
  def nonEmpty: Boolean
}
```

```scala

val registry = BindingResolver.defaults

registry.contains[Int]              // true — primitive binding exists
registry.containsSeq[List[String]]  // true — List constructor is bound
registry.containsMap[Map[Int, Int]] // true — Map constructor is bound
registry.size                       // total number of registered bindings
```

## Integration with `DynamicSchema`

The primary consumer of `BindingResolver` is `DynamicSchema#rebind`. Given an unbound `DynamicSchema`, `DynamicSchema#rebind` walks the `Reflect` tree and queries the resolver for each node's binding, then returns a fully operational `Schema[A]`:

```scala

case class Product(sku: String, price: Double, tags: List[String])

object Product {
  implicit val schema: Schema[Product] = Schema.derived[Product]
}

val dynamic: DynamicSchema = Schema[Product].toDynamicSchema

// The resolver must cover every concrete type that appears in the schema tree
val resolver: BindingResolver =
  BindingResolver.empty.bind(Binding.of[Product]) ++ BindingResolver.defaults

val rebound: Schema[Product] = dynamic.rebind[Product](resolver)
```

When using `BindingResolver.reflection` on the JVM, individual record types do not need to be registered explicitly—the resolver derives their bindings on demand from the class structure:

```scala

case class Address(street: String, city: String, zip: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}

// No explicit bind(Binding.of[Address]) needed on JVM
val resolver: BindingResolver = BindingResolver.reflection ++ BindingResolver.defaults

val rebound: Schema[Address] = Schema[Address].toDynamicSchema.rebind[Address](resolver)
```

:::warning
If `DynamicSchema#rebind` cannot find a binding for any type present in the unbound schema tree, it throws at runtime. Make sure the resolver covers every concrete type—records, variants, wrappers, primitives, and collections—that appears in the schema.
:::

See [Binding](./binding.md) for details on each binding kind, and [Schema](./schema.md) for the overall structure of the schema system.

---

## The Binding Data Type

`Binding` is a sealed trait in ZIO Blocks that provides the operational machinery for constructing and deconstructing values of schema-described types. While `Reflect` describes the **structure** of data types, `Binding` provides the **behavior** needed to work with those types at runtime.

In other words, a binding is used to attach non-serializable Scala functions, such as **constructors**, **deconstructors**, and **matchers**, to a reflection type.

The combination of `Reflect` and `Binding` forms the foundation of `Schema`: a bound reflect (one where `F[_, _] = Binding`) that can both describe and manipulate data.

```scala
sealed trait Binding[T <: BindingType, A]
```

The `Binding` trait is parameterized by:

- **`T <: BindingType`**: A phantom type that identifies the kind of binding (Record, Variant, Seq, Map, etc.)
- **`A`**: The Scala type that this binding operates on

## The Role of Binding in the Schema Architecture

ZIO Blocks separates operational behavior from structural metadata using two key abstractions:

1. **Reflect** (`Reflect[F, A]`): A generic data structure describing the shape of type `A`. It's parameterized by `F[_, _]`, which can be plugged with different "binding strategies".
2. **Binding**: The concrete binding strategy that embeds construction/deconstruction capabilities within a `Reflect` structure. The binding is the operational behavior attached to the reflect.

When `F[_, _] = Binding`, you get a **bound reflect** that can actually construct and deconstruct values. When `F[_, _] = NoBinding`, you get an **unbound reflect** that contains only structural metadata and is fully **serializable**.

```scala
// Type aliases for clarity
type Reflect.Bound[A] = Reflect[Binding, A]     // Can construct/deconstruct A
type Reflect.Unbound[A] = Reflect[NoBinding, A] // Pure metadata, serializable
```

A `Schema[A]` is simply a wrapper around `Reflect.Bound[A]`:

```scala
final class Schema[A](val reflect: Reflect.Bound[A])
```

## Binding Variants

The `Binding` sealed trait has several case class variants, each corresponding to a different kind of `Reflect` node:

### 1. `Binding.Record`

Provides construction and deconstruction capabilities for product types—case classes, tuples, and any type composed of multiple named or positional fields.

```scala
final case class Record[A](
  constructor: Constructor[A],
  deconstructor: Deconstructor[A],
  defaultValue: Option[() => A] = None,
  examples: collection.immutable.Seq[A] = Nil
) extends Binding[BindingType.Record, A]
```

**Components:**

- `constructor`: Builds an `A` from primitive components stored in `Registers`
- `deconstructor`: Breaks down an `A` into primitive components in `Registers`
- `defaultValue`: An optional thunk `() => A` that produces a default instance. This is useful for formats like Protocol Buffers where missing fields assume default values.
- `examples`: Sample values for documentation or testing, or OpenAPI schema generation.

Here is an example of a `Binding.Record` for a simple `Person` case class:

```scala

case class Person(name: String, age: Int)

val initialRegisters = RegisterOffset(objects = 1, ints = 1)

val personRecord: Binding.Record[Person] =
  Binding.Record[Person](
    constructor = new Constructor[Person] {
      def usedRegisters = initialRegisters

      def construct(in: Registers, offset: RegisterOffset): Person =
        Person(
          in.getObject(offset).asInstanceOf[String],
          in.getInt(offset + RegisterOffset(objects = 1))
        )
    },
    deconstructor = new Deconstructor[Person] {

      def usedRegisters = initialRegisters

      def deconstruct(out: Registers, offset: RegisterOffset, p: Person): Unit = {
        out.setObject(offset, p.name)
        out.setInt(offset + RegisterOffset(objects = 1), p.age)
      }
    }
  )
```

### 2. `Binding.Variant`

The `Binding.Variant` data type provides discrimination capabilities and matchers for sum types—sealed traits, Scala 3 enums, and any type that represents one of several possible alternatives:

```scala
final case class Variant[A](
  discriminator: Discriminator[A],
  matchers: Matchers[A],
  defaultValue: Option[() => A] = None,
  examples: collection.immutable.Seq[A] = Nil
) extends Binding[BindingType.Variant, A]
```

**Discriminator** and **Matchers** are complementary components in ZIO Blocks that work together to handle sum types (ADTs/sealed traits/enums):

- `Discriminator`: Given a value of a sum type, determine which case it belongs to by returning a numerical index.
- `Matchers`: Given a value and a case index, safely downcast the value to the specific case type, or return null if it doesn't match.

```scala

sealed trait Shape extends Product with Serializable
case class Circle(radius: Double) extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

val shapeBinding = Binding.Variant[Shape](
  discriminator = new Discriminator[Shape] {
    override def discriminate(a: Shape): Int = a match {
      case Circle(_) => 0
      case Rectangle(_, _) => 1
    }
  },
  matchers = Matchers(
    new Matcher[Circle] {
      override def downcastOrNull(any: Any): Circle =
        any match {
          case c: Circle => c
          case _ => null.asInstanceOf[Circle]
        }
    },
    new Matcher[Rectangle] {
      override def downcastOrNull(any: Any): Rectangle =
        any match {
          case r: Rectangle => r
          case _ => null.asInstanceOf[Rectangle]
        }
    }
  )
)
```

Please note that the `Variant` is only responsible for discrimination and downcasting. The actual construction and deconstruction of each case is handled by the individual case's `Binding.Record` (or other appropriate binding).

### 3. Binding.Seq

The `Binding.Seq` data type provides efficient construction and deconstruction capabilities for sequence/collection types—`List`, `Vector`, `Array`, `Set`, `Chunk`, and any ordered collection:

```scala
final case class Seq[C[_], A](
  constructor: SeqConstructor[C],
  deconstructor: SeqDeconstructor[C],
  defaultValue: Option[() => C[A]] = None,
  examples: collection.immutable.Seq[C[A]] = Nil
) extends Binding[BindingType.Seq[C], C[A]]
```

The type parameter is `C[_]` (the collection type constructor) rather than `C[A]` (a specific element type). This allows a single `Binding.Seq[List]` to work for `List[Int]`, `List[String]`, `List[Person]`, etc. Inside the companion object of `Binding.Seq`, there are implementations for standard Scala collections like `List`, `Vector`, `Seq`, `IndexedSeq`, `Set`, and `ArraySeq`.

Sequences are ubiquitous in data modeling, and their efficient handling is critical for performance. The `Seq` binding abstracts over the specific collection type using the **builder pattern**, enabling both stacked and heap-allocated construction strategies.

- `seqConstructor` is a sequence constructor which provides how to build the sequence of type `C[_]`. It provides collection-specific builder operations. Different collections have different constructor strategies (e.g., `Array` needs size upfront and is stack-allocated, while `List` can be built incrementally on the heap).
- `seqDeconstructor` is a sequence deconstructor that provides iteration and size information for the collection type `C[_]`, which enables us to tear down the collection into its elements efficiently. The size hints enable codecs to pre-allocate output buffers and builders.

Let's take a deep dive into these two main components.

#### SeqConstructor

The `SeqConstructor` has two main variants:

1. **Heap-allocated Constructors** which are for collections like `List`, `Vector`, `Set`, etc. These collections typically use mutable builders under the hood to efficiently accumulate elements before producing an immutable collection. The `Boxed` abstract class is a convenient base class for heap-allocated constructors that uses a single builder type for all primitive types. This makes it easy to implement sequence constructors for collections of heap-allocated objects:

```scala
abstract class Boxed[C[_]] extends SeqConstructor[C] {
  override type BooleanBuilder = ObjectBuilder[Boolean]
  override type ByteBuilder = ObjectBuilder[Byte]
  // ... all primitive builders are aliases to ObjectBuilder

  def addBoolean(builder: BooleanBuilder, a: Boolean): Unit = addObject(builder, a)
  // primitives get boxed when passed to addObject

  // ... similarly for other primitive adders
}
```

To implement a boxed sequence, we have to define just four things: the builder type, how to create a new builder, how to add elements to it, and how to produce the final collection.

ZIO Blocks provides `SeqConstructor.Boxed` implementations for standard Scala collections like `List`, `Vector`, `Set`, `IndexedSeq` and `Seq`. Here is an example of a `SeqConstructor.Boxed` for `List`:

```scala
val listConstructor: SeqConstructor[List] = new Boxed[List] {
  type ObjectBuilder[A] = scala.collection.mutable.ListBuffer[A]

  def newObjectBuilder[A](sizeHint: Int): ObjectBuilder[A] = new ListBuffer[A]

  def addObject[A](builder: ObjectBuilder[A], a: A): Unit = builder.addOne(a)

  def resultObject[A](builder: ObjectBuilder[A]): List[A] = builder.toList
}
```

2. **Stack-allocated Constructors** which are for collections like `Array`, `ArraySeq`, and `IArray`.

All the implementations are found in the companion object of the `SeqConstructor` trait.

#### SeqDeconstructor

The `SeqDeconstructor` has a simple interface with two methods:

```scala
trait SeqDeconstructor[C[_]] {
  def deconstruct[A](c: C[A]): Iterator[A]

  def size[A](c: C[A]): Int
}
```

The `deconstruct` method provides an iterator over the elements of the collection, while the `size` method returns the number of elements.

Inside the companion object of the `SeqDeconstructor` there are implementations for standard Scala collections like `List`, `Vector`, `Seq`, and `IndexedSeq`.

For efficiency, there are specialized implementations for array-based sequences like `Array`, `ArraySeq`, and `IArray` which provide direct access to elements, besides the iterator-based approach. As a result, they are much more performant than the generic iterator-based deconstruction. When an `Optic` needs to access a specific element by index, it can use these specialized methods to avoid the overhead of iterator traversal.

This is done through the `SpecializedIndexed` trait:

```scala
sealed trait SpecializedIndexed[C[_]] extends SeqDeconstructor[C] {
  def elementType[A](c: C[A]): RegisterType[A]
  def objectAt[A](c: C[A], index: Int): A
  def booleanAt(c: C[Boolean], index: Int): Boolean
  def byteAt(c: C[Byte], index: Int): Byte
  def shortAt(c: C[Short], index: Int): Short
  def intAt(c: C[Int], index: Int): Int
  def longAt(c: C[Long], index: Int): Long
  def floatAt(c: C[Float], index: Int): Float
  def doubleAt(c: C[Double], index: Int): Double
  def charAt(c: C[Char], index: Int): Char
}
```

You can find implementations of `SpecializedIndexed` for `Array`, `ArraySeq`, and `IArray` in the companion object of `SeqDeconstructor`.

### 4. Binding.Map

To describe the construction and deconstruction of key-value collections, use `Binding.Map`:

```scala
final case class Map[M[_, _], K, V](
  constructor: MapConstructor[M],
  deconstructor: MapDeconstructor[M],
  defaultValue: Option[() => M[K, V]] = None,
  examples: collection.immutable.Seq[M[K, V]] = Nil
) extends Binding[BindingType.Map[M], M[K, V]]
```

It has two main components for building maps and tearing down maps to their key-value pairs:

- `mapConstructor`: Builds maps from key-value pairs using builders
- `mapDeconstructor`: Provides key-value iteration for maps

Their interfaces are similar to `SeqConstructor` and `SeqDeconstructor`, but specialized for key-value pairs.

ZIO Blocks has an implementation of `Binding.Map` for standard Scala `Map`s in the companion object of `Binding.Map` called `Binding.Map.map`.

### 5. Binding.Primitive

The `Binding.Primitive` provides metadata for primitive/scalar types—`Int`, `Long`, `Double`, `String`, `Boolean`, `Byte`, `Short`, `Char`, `Float`, `BigInt`, `BigDecimal`, `UUID`, and temporal types.

```scala
final case class Primitive[A](
  defaultValue: Option[() => A] = None,
  examples: collection.immutable.Seq[A] = Nil
) extends Binding[BindingType.Primitive, A]
```

Primitives require minimal binding logic since they are atomic values that don't need construction from parts. You can access all built-in primitive bindings in the companion object of `Binding.Primitive`, such as `Primitive.int`, `Primitive.string`, etc.

### 6. Binding.Wrapper

The `Binding.Wrapper` provides wrap/unwrap capabilities for newtype patterns—opaque types, value classes, validated wrappers, and single-field case class wrappers.

`Newtypes` are extremely common in well-designed codebases. Instead of passing raw String values for emails, you define `Email` as a distinct type. Instead of `Int` for user IDs, you define `UserId`. This provides type safety without runtime overhead.

The Wrapper binding handles the bidirectional transformation between the wrapper type and its underlying representation, with optional validation:

```scala
final case class Wrapper[A, B](
  wrap: B => Either[SchemaError, A],
  unwrap: A => Either[SchemaError, B]
) extends Binding[BindingType.Wrapper[A, B], A]
```

**Components:**

- `wrap`: Converts from the underlying type `B` to the wrapper type `A`
- `unwrap`: Extracts the underlying `B` from an `A`

Here is an example of a `Binding.Wrapper` for an `Email` newtype:

```scala

case class Email(value: String)

object Email {
  new Binding.Wrapper[Email, String](
    wrap = value => new Email(value),
    unwrap = email => email.value
  )
}
```

### 7. Binding.Dynamic

To bind untyped dynamic values, use `Binding.Dynamic`. It provides a binding for dynamically typed values whose structure is not known at compile time:

```scala
final case class Dynamic(
  defaultValue: Option[() => DynamicValue] = None,
  examples: collection.immutable.Seq[DynamicValue] = Nil
) extends Binding[BindingType.Dynamic, DynamicValue]
```

Used when the schema of the data is not known at compile time, such as JSON payloads with arbitrary structure.

## Binding and Schema Serialization

When `F[_, _] = NoBinding` in `Reflect[F[_, _], A]` the `Reflect` structure contains only pure data (no functions) and becomes fully serializable. This enables:

1. **Schema serialization**: Convert schemas to JSON Schema or other formats, making them portable
2. **Schema rebinding**: Deserialize a schema and rebind it using a `TypeRegistry`, so it becomes type-safe and operational again

We will cover schema serialization and rebinding in more detail in the `Reflect` data type documentation page. For the full API of the binding lookup mechanism used during rebinding, see [BindingResolver](./binding-resolver.md).

## Summary

The `Binding` data type is the operational heart of ZIO Blocks' schema system:

| Reflect Node        | Binding Variant     | Construction and Deconstruction of |
|---------------------|---------------------|------------------------------------|
| `Reflect.Record`    | `Binding.Record`    | Product Types                      |
| `Reflect.Variant`   | `Binding.Variant`   | Sum Types                          |
| `Reflect.Sequence`  | `Binding.Seq`       | Sequence Collection Types          |
| `Reflect.Map`       | `Binding.Map`       | Key-value Collection Types         |
| `Reflect.Primitive` | `Binding.Primitive` | Primitive Types                    |
| `Reflect.Wrapper`   | `Binding.Wrapper`   | Wrap/unwrap newtypes               |
| `Reflect.Dynamic`   | `Binding.Dynamic`   | Untyped Dynamic Values             |

A key innovation is the clean separation between structure (`Reflect`) and behavior (`Binding`), enabling us to achieve serializable schemas with unbound reflects, while the `Reflect` remains pluggable with the `Binding` for runtime operations.

---

## Chunk(Reference)

`Chunk[A]` is an immutable, indexed sequence optimized for high-performance operations. Unlike Scala's built-in collections, Chunk is designed for zero-allocation access patterns, efficient concatenation, and unboxed primitive storage.

## Why Chunk?

Chunk addresses several limitations of standard Scala collections:

| Feature | Chunk | Vector | Array |
|---------|-------|--------|-------|
| Immutable | ✓ | ✓ | ✗ |
| O(1) indexed access | ✓ | ~O(1) | ✓ |
| Unboxed primitives | ✓ | ✗ | ✓ |
| Efficient concatenation | ✓ | ✓ | ✗ |
| Safe functional interface | ✓ | ✓ | ✗ |
| Lazy slicing | ✓ | ✗ | ✗ |

Key advantages:

- **Zero-boxing for primitives**: `Chunk[Int]`, `Chunk[Double]`, etc. store values unboxed in specialized arrays
- **Lazy concatenation**: Uses balanced tree structures (based on Conc-Trees) for O(log n) concatenation
- **Efficient slicing**: `drop`, `take`, and `slice` create views without copying
- **Automatic materialization**: Deep operation chains are materialized when depth exceeds thresholds
- **Scala collections integration**: Implements `IndexedSeq` for seamless interoperability

## Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-chunk" % "<version>"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-chunk" % "<version>"
```

Supported Scala versions: 2.13.x and 3.x

## Creating Chunks

### From Varargs

```scala

val numbers = Chunk(1, 2, 3, 4, 5)
val strings = Chunk("hello", "world")
val empty   = Chunk.empty[Int]
```

### From a Single Element

```scala

val single = Chunk.single(42)
val unit   = Chunk.unit // Chunk(())
```

### From Arrays

When you have an existing array, use `fromArray`. Note that the array should not be mutated after wrapping:

```scala

val arr   = Array(1, 2, 3)
val chunk = Chunk.fromArray(arr)
```

### From Iterables and Iterators

```scala

val fromList   = Chunk.fromIterable(List(1, 2, 3))
val fromVector = Chunk.fromIterable(Vector("a", "b"))
val fromIter   = Chunk.fromIterator(Iterator.range(0, 10))
```

### From Java Collections

```scala

val javaList = new util.ArrayList[String]()
javaList.add("one")
javaList.add("two")

val chunk = Chunk.fromJavaIterable(javaList)
```

### From NIO Buffers

Chunk provides direct integration with Java NIO buffers:

```scala

val buffer = ByteBuffer.wrap(Array[Byte](1, 2, 3, 4))
val bytes  = Chunk.fromByteBuffer(buffer)
```

Available buffer constructors:
- `Chunk.fromByteBuffer(ByteBuffer): Chunk[Byte]`
- `Chunk.fromCharBuffer(CharBuffer): Chunk[Char]`
- `Chunk.fromIntBuffer(IntBuffer): Chunk[Int]`
- `Chunk.fromLongBuffer(LongBuffer): Chunk[Long]`
- `Chunk.fromShortBuffer(ShortBuffer): Chunk[Short]`
- `Chunk.fromFloatBuffer(FloatBuffer): Chunk[Float]`
- `Chunk.fromDoubleBuffer(DoubleBuffer): Chunk[Double]`

### Generator Functions

```scala

val filled   = Chunk.fill(5)("x")         // Chunk("x", "x", "x", "x", "x")
val iterated = Chunk.iterate(1, 5)(_ * 2) // Chunk(1, 2, 4, 8, 16)
val unfolded = Chunk.unfold(0)(n => if (n < 5) Some((n, n + 1)) else None)
```

## Core Operations

### Element Access

```scala

val chunk = Chunk(10, 20, 30, 40, 50)

val first  = chunk(0)          // 10
val second = chunk(1)          // 20
val head   = chunk.head        // 10
val last   = chunk.last        // 50
val len    = chunk.length      // 5

val maybeHead = chunk.headOption // Some(10)
val maybeLast = chunk.lastOption // Some(50)
```

For primitive chunks, specialized accessors avoid boxing:

```scala

val ints = Chunk(1, 2, 3)
val i: Int = ints.int(0)     // unboxed access

val bytes = Chunk[Byte](1, 2, 3)
val b: Byte = bytes.byte(0)  // unboxed access

val doubles = Chunk(1.0, 2.0, 3.0)
val d: Double = doubles.double(0)  // unboxed access
```

### Transformations

```scala

val chunk = Chunk(1, 2, 3, 4, 5)

val doubled  = chunk.map(_ * 2)           // Chunk(2, 4, 6, 8, 10)
val filtered = chunk.filter(_ > 2)        // Chunk(3, 4, 5)
val flatted  = chunk.flatMap(n => Chunk(n, n)) // Chunk(1, 1, 2, 2, ...)
val collected = chunk.collect { case n if n % 2 == 0 => n * 10 } // Chunk(20, 40)
```

### Concatenation

Concatenation is efficient—Chunk uses balanced tree structures to avoid copying:

```scala

val a = Chunk(1, 2, 3)
val b = Chunk(4, 5, 6)

val combined = a ++ b           // Chunk(1, 2, 3, 4, 5, 6)
val appended = a :+ 4           // Chunk(1, 2, 3, 4)
val prepended = 0 +: a          // Chunk(0, 1, 2, 3)
```

### Slicing

Slicing operations create views and don't copy data:

```scala

val chunk = Chunk(1, 2, 3, 4, 5, 6, 7, 8)

val firstThree = chunk.take(3)        // Chunk(1, 2, 3)
val lastThree  = chunk.takeRight(3)   // Chunk(6, 7, 8)
val dropped    = chunk.drop(2)        // Chunk(3, 4, 5, 6, 7, 8)
val sliced     = chunk.slice(2, 5)    // Chunk(3, 4, 5)

val (left, right) = chunk.splitAt(4)  // (Chunk(1,2,3,4), Chunk(5,6,7,8))
```

### Conditional Operations

```scala

val chunk = Chunk(1, 2, 3, 4, 5, 6)

val takeWhileSmall = chunk.takeWhile(_ < 4)    // Chunk(1, 2, 3)
val dropWhileSmall = chunk.dropWhile(_ < 4)    // Chunk(4, 5, 6)
val takeUntilBig   = chunk.takeWhile(_ <= 3)   // Chunk(1, 2, 3)
val dropUntilBig   = chunk.dropUntil(_ > 3)    // Chunk(5, 6)
```

### Folding and Reduction

```scala

val chunk = Chunk(1, 2, 3, 4, 5)

val sum     = chunk.foldLeft(0)(_ + _)    // 15
val product = chunk.foldRight(1)(_ * _)   // 120
val summed  = chunk.reduce(_ + _)         // 15

val runningSum = chunk.foldWhile(0)(_ < 10)(_ + _) // 10 (1+2+3+4)
```

### Searching and Predicates

```scala

val chunk = Chunk(1, 2, 3, 4, 5)

val hasEven  = chunk.exists(_ % 2 == 0)  // true
val allSmall = chunk.forall(_ < 10)      // true
val found    = chunk.find(_ > 3)         // Some(4)
val index    = chunk.indexWhere(_ > 3)   // 3
```

### Zipping

```scala

val as = Chunk("a", "b", "c")
val bs = Chunk(1, 2, 3)

val zipped      = as.zip(bs)              // Chunk(("a",1), ("b",2), ("c",3))
val withIndex   = as.zipWithIndex         // Chunk(("a",0), ("b",1), ("c",2))
val zipWith     = as.zipWith(bs)(_ + _)   // Chunk("a1", "b2", "c3")
val zipAll      = as.zipAll(Chunk(1, 2))  // handles different lengths
```

### Updating Elements

Updates are immutable and use efficient buffering:

```scala

val chunk   = Chunk(1, 2, 3, 4, 5)
val updated = chunk.updated(2, 100) // Chunk(1, 2, 100, 4, 5)
```

### Deduplication and Sorting

```scala

val withDupes = Chunk(1, 1, 2, 2, 2, 3, 3)
val deduped   = withDupes.dedupe  // Chunk(1, 2, 3) - removes adjacent duplicates

val unsorted = Chunk(3, 1, 4, 1, 5)
val sorted   = unsorted.sorted    // Chunk(1, 1, 3, 4, 5)
```

### Splitting

```scala

val chunk = Chunk(1, 2, 3, 4, 5, 6)

val parts = chunk.split(3) // Chunk(Chunk(1,2), Chunk(3,4), Chunk(5,6))
val (before, after) = chunk.splitWhere(_ > 3) // splits at first element > 3
```

### String Conversion

```scala

val bytes = Chunk[Byte](72, 101, 108, 108, 111)
val str   = bytes.asString  // "Hello"

val chars = Chunk('H', 'e', 'l', 'l', 'o')
val str2  = chars.asString  // "Hello"

val withCharset = bytes.asString(StandardCharsets.UTF_8) // "Hello"
val base64      = bytes.asBase64String // base64-encoded string
```

### Materialization

For complex operation chains, you can force materialization to an array-backed chunk:

```scala

val complex = Chunk(1, 2, 3) ++ Chunk(4, 5) ++ Chunk(6, 7)
val materialized = complex.materialize // backed by a single array
```

## NonEmptyChunk

`NonEmptyChunk[A]` is a chunk guaranteed to contain at least one element. This enables safe use of operations like `head` and `reduce`:

```scala

val nec = NonEmptyChunk(1, 2, 3)

val first: Int = nec.head      // always safe
val sum: Int   = nec.reduce(_ + _)  // always safe

val mapped: NonEmptyChunk[Int] = nec.map(_ * 2)
val flatMapped: NonEmptyChunk[Int] = nec.flatMap(n => NonEmptyChunk(n, n + 1))
```

### Creating NonEmptyChunk

```scala

val fromValues = NonEmptyChunk(1, 2, 3)
val single     = NonEmptyChunk.single(42)
val fromCons   = NonEmptyChunk.fromCons(::(1, List(2, 3)))
val fromIterable = NonEmptyChunk.fromIterable(1, List(2, 3))

val maybeNec: Option[NonEmptyChunk[Int]] = NonEmptyChunk.fromChunk(Chunk(1, 2))
val empty: Option[NonEmptyChunk[Int]] = NonEmptyChunk.fromChunk(Chunk.empty) // None
```

### Converting Between Chunk and NonEmptyChunk

```scala

val nec = NonEmptyChunk(1, 2, 3)
val chunk: Chunk[Int] = nec.toChunk

val chunk2 = Chunk(1, 2, 3)
chunk2.nonEmptyOrElse(0)(_.reduce(_ + _)) // 6 if non-empty, 0 if empty
```

### Operations That Preserve NonEmptiness

These operations return `NonEmptyChunk`:
- `map`, `flatMap`, `flatten`
- `append`, `prepend`, `++`
- `zip`, `zipWith`, `zipWithIndex`
- `sorted`, `sortBy`, `reverse`
- `distinct`, `materialize`

Operations that might produce empty results return `Chunk`:
- `filter`, `filterNot`
- `collect`
- `tail`, `init`

## ChunkBuilder

`ChunkBuilder` is a mutable builder for creating chunks efficiently. It's specialized for primitives to avoid boxing:

```scala

val builder = ChunkBuilder.make[Int]()
builder.addOne(1)
builder.addOne(2)
builder.addAll(List(3, 4, 5))
val result: Chunk[Int] = builder.result() // Chunk(1, 2, 3, 4, 5)
```

### Specialized Builders

For primitives, use specialized builders for best performance:

```scala

val intBuilder = new ChunkBuilder.Int
intBuilder.addOne(1)
intBuilder.addOne(2)
val ints: Chunk[Int] = intBuilder.result()

val byteBuilder = new ChunkBuilder.Byte
val longBuilder = new ChunkBuilder.Long
val doubleBuilder = new ChunkBuilder.Double
val boolBuilder = new ChunkBuilder.Boolean
```

Available specialized builders: `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, `Double`

## Bit Operations

Chunk provides efficient bit-level operations for working with binary data:

### Converting to Bits

```scala

val bytes = Chunk[Byte](0x0F, 0xF0.toByte)
val bits  = bytes.asBitsByte  // Chunk of 16 booleans

val ints = Chunk(0x12345678)
val intBits = ints.asBitsInt(Chunk.BitChunk.Endianness.BigEndian)

val longs = Chunk(0x123456789ABCDEF0L)
val longBits = longs.asBitsLong(Chunk.BitChunk.Endianness.BigEndian)
```

### Bitwise Operations

```scala

val a = Chunk(true, false, true, false)
val b = Chunk(true, true, false, false)

val andResult = a & b  // Chunk(true, false, false, false)
val orResult  = a | b  // Chunk(true, true, true, false)
val xorResult = a ^ b  // Chunk(false, true, true, false)
val negated   = a.negate // Chunk(false, true, false, true)
```

### Packing Booleans

```scala

val bits = Chunk(true, false, true, false, true, true, true, true)
val packedBytes: Chunk[Byte] = bits.toPackedByte  // Efficient byte representation

val packedInts: Chunk[Int] = bits.toPackedInt(Chunk.BitChunk.Endianness.BigEndian)
val packedLongs: Chunk[Long] = bits.toPackedLong(Chunk.BitChunk.Endianness.BigEndian)
```

### Binary String

```scala

val bits = Chunk(true, false, true, true)
val binary: String = bits.toBinaryString // "1011"
```

## ChunkMap

`ChunkMap[K, V]` is an order-preserving immutable map backed by parallel chunks. It maintains insertion order during iteration:

```scala

val map = ChunkMap("a" -> 1, "b" -> 2, "c" -> 3)

val value = map.get("b")      // Some(2)
val updated = map.updated("d", 4)
val removed = map.removed("b")
```

### Creating ChunkMap

```scala

val empty = ChunkMap.empty[String, Int]
val fromPairs = ChunkMap("x" -> 1, "y" -> 2)
val fromChunk = ChunkMap.fromChunk(Chunk(("a", 1), ("b", 2)))
val fromChunks = ChunkMap.fromChunks(Chunk("a", "b"), Chunk(1, 2))
```

### Indexed Access

ChunkMap provides O(1) positional access:

```scala

val map = ChunkMap("z" -> 1, "a" -> 2, "m" -> 3)

val first = map.atIndex(0)      // ("z", 1)
val key   = map.keyAtIndex(1)   // "a"
val value = map.valueAtIndex(2) // 3

val keys: Chunk[String] = map.keysChunk
val values: Chunk[Int] = map.valuesChunk
```

### Optimized Lookup

For frequent lookups, create an indexed version with O(1) key access:

```scala

val map = ChunkMap("a" -> 1, "b" -> 2, "c" -> 3)
val indexed = map.indexed  // O(1) lookups, extra memory for index

val value = indexed.get("b")  // O(1) instead of O(n)
```

## Scala Collections Integration

Chunk implements `IndexedSeq` and integrates seamlessly with Scala collections:

```scala

val chunk = Chunk(1, 2, 3, 4, 5)

val list: List[Int] = chunk.toList
val vector: Vector[Int] = chunk.toVector
val array: Array[Int] = chunk.toArray

val fromSeq: Chunk[Int] = Chunk.from(Vector(1, 2, 3))
```

Standard collection operations work as expected:

```scala

val chunk = Chunk(1, 2, 3)
val result = chunk
  .filter(_ > 1)
  .map(_ * 2)
  .flatMap(n => Chunk(n, n + 1))
```

## Performance Characteristics

| Operation | Time Complexity | Notes |
|-----------|-----------------|-------|
| `apply(i)` | O(1) | Direct array access for materialized chunks |
| `length` | O(1) | Cached |
| `head`, `last` | O(1) | |
| `++` | O(log n) | Balanced tree concatenation |
| `:+`, `+:` | O(1) amortized | Buffered appends |
| `take`, `drop`, `slice` | O(1) | Creates view |
| `map`, `filter`, `flatMap` | O(n) | |
| `updated` | O(1) amortized | Buffered updates |
| `materialize` | O(n) | Copies to array |

### When to Materialize

Chunk automatically materializes when:
- Tree depth exceeds internal thresholds
- You call `materialize` explicitly
- Converting to array with `toArray`

Consider explicit materialization when:
- Performing many random accesses on a deeply nested chunk
- Passing data to APIs that need arrays
- Optimizing a hot loop

---

## Codec

`Codec[DecodeInput, EncodeOutput, Value]` is the base abstraction for encoding and decoding values between a specific input representation and a specific output representation. It forms the foundation of ZIO Blocks' multi-format serialization system, enabling a single `Schema[A]` to derive codecs for JSON, Avro, TOON, MessagePack, Thrift, and other formats that are integrated via the `Codec`/`Format` system. BSON support is provided separately via `BsonSchemaCodec`, which is not a subtype of `codec.Codec` and is not derived via `Schema.derive(format)`.

## Overview

`Codec` defines two abstract methods that every concrete codec must implement:

```scala
abstract class Codec[DecodeInput, EncodeOutput, Value] {
  def encode(value: Value, output: EncodeOutput): Unit
  def decode(input: DecodeInput): Either[SchemaError, Value]
}
```

- **`encode`** writes the encoded form of `value` into `output`. The output parameter is typically a mutable buffer (`ByteBuffer`, `CharBuffer`) that the caller provides.
- **`decode`** reads from `input` and returns either a `SchemaError` describing the failure or the decoded value.

End users rarely interact with `Codec` directly. Instead, they work with format-specific subclasses like `JsonCodec[A]` or `ToonCodec[A]`, which add convenience methods for common input/output types.

Given a `Schema[A]`, you can derive a codec for any supported format by calling `Schema[A].derive(format)`, which uses the `Deriver` associated with that format to generate the appropriate codec instance. For example, to derive a JSON codec:

```scala

case class Person(name: String, age: Int)

object Person {
  // Derive a schema for Person (required for codec derivation)
  implicit val schema: Schema[Person] = Schema.derived
   // Derive a JSON codec from the schema 
  val codec: JsonCodec[Person] = schema.derive(JsonFormat)
}

// Encode
val bytes: Array[Byte] = Person.codec.encode(Person("Alice", 30))

// Decode
val result: Either[SchemaError, Person] = Person.codec.decode(bytes)
```

## Installation

To include the base schema module with JSON support, add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

Additional format modules are separate artifacts:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro"        % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson"        % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-csv"         % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift"      % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon"        % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-xml"         % "0.0.33"
libraryDependencies += "dev.zio" %% "zio-blocks-schema-yaml"        % "0.0.33"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## BinaryCodec and TextCodec

The codec system in ZIO Blocks is organized as a layered hierarchy:

```
Codec[DecodeInput, EncodeOutput, Value]        
├── BinaryCodec[A] =  Codec[ByteBuffer, ByteBuffer, A]   (ByteBuffer ↔ A)
│   ├── AvroCodec[A]                   
│   ├── JsonCodec[A]                    
│   ├── MessagePackCodec[A]         
│   ├── ThriftCodec[A]               
│   ├── ToonCodec[A]                  
│   ├── XmlCodec[A]         
│   └── YamlCodec[A]         
└── TextCodec[A] = Codec[CharBuffer, CharBuffer, A]      (CharBuffer ↔ A)
    └── CsvCodec[A]                    
```

1. **`BinaryCodec[A]`** fixes both the input and output to `ByteBuffer` and is the base class for all codecs that operate on binary data:

```scala
abstract class BinaryCodec[A] extends Codec[ByteBuffer, ByteBuffer, A]
```

2. **`TextCodec[A]`** fixes both the input and output to `CharBuffer`:

```scala
abstract class TextCodec[A] extends Codec[CharBuffer, CharBuffer, A]
```

All built-in serialization formats (JSON, Avro, TOON, MessagePack, Thrift) extend `BinaryCodec`. Despite JSON being a text format, the JSON codec operates on UTF-8 encoded bytes for performance.

`TextCodec` exists for formats that operate on character data rather than raw bytes. No built-in formats currently use `TextCodec`, but it is available for custom text-based formats.

## Deriving Codecs

### Using Schema.derive

The primary way to obtain a codec is through `Schema[A].derive`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Pass a Format object to get a codec for that format
val jsonCodec: JsonCodec[Person] = Schema[Person].derive(JsonFormat)
```

This works with any format:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val jsonCodec = Schema[Person].derive(JsonFormat)
val toonCodec = Schema[Person].derive(ToonFormat)
```

### Using Schema.deriving for Customization

For more control over the derived codec, use `deriving` to get a `DerivationBuilder`. This lets you override instances for specific substructures or inject modifiers before finalizing:

```scala

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name = $(_.name)
  val age  = $(_.age)
}

// Override the codec for the "name" field
val customNameCodec = new JsonCodec[String] {
  def decodeValue(in: JsonReader): String = in.readString()
  def encodeValue(x: String, out: JsonWriter): Unit = out.writeVal(x.toUpperCase)
}

val codec: JsonCodec[Person] = Schema[Person]
  .deriving(JsonFormat.deriver)
  .instance(Person.name, customNameCodec)
  .derive
```

### Using Schema#decode and Schema#encode

`Schema` also provides `decode` and `encode` methods that internally call `derive` (with caching) and then delegate to the codec:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Encode directly from Schema
val buffer = ByteBuffer.allocate(1024)
Schema[Person].encode(JsonFormat)(buffer)(Person("Alice", 30))

// Decode directly from Schema
buffer.flip()
val result: Either[SchemaError, Person] = Schema[Person].decode(JsonFormat)(buffer)
```

### Using a Deriver Directly

Each `Format` object contains a `Deriver[TC]` that can also be passed to `derive`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// These are equivalent:
val codec1 = Schema[Person].derive(JsonFormat)
val codec2 = Schema[Person].derive(JsonFormat.deriver)
```

Passing a `Deriver` directly is useful when working with custom or configured derivers (see [Configuring Codecs](#configuring-codecs)).

## Convenience Methods on Format-Specific Codecs

While the base `Codec` class defines only `encode(value, output)` and `decode(input)`, format-specific subclasses like `JsonCodec` and `ToonCodec` add convenience overloads for common I/O types.

### JsonCodec Convenience Methods

`JsonCodec[A]` provides the following overloads beyond the base `ByteBuffer` API:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(JsonFormat)
val person = Person("Alice", 30)

// Array[Byte]
val bytes: Array[Byte] = codec.encode(person)
val fromBytes1: Either[SchemaError, Person] = codec.decode(bytes)
val fromBytes2: Either[SchemaError, Person] = codec.decode(bytes, 0, bytes.length)

// String
val jsonStr: String = codec.encodeToString(person)
val fromStr: Either[SchemaError, Person] = codec.decode("""{"name":"Alice","age":30}""")

// InputStream / OutputStream

val os = new ByteArrayOutputStream()
codec.encode(person, os)

val is = new ByteArrayInputStream(os.toByteArray)
val fromStream: Either[SchemaError, Person] = codec.decode(is)
```

### ToonCodec Convenience Methods

`ToonCodec[A]` provides the same set of overloads:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(ToonFormat)
val person = Person("Alice", 30)

// Array[Byte]
val bytes: Array[Byte] = codec.encode(person)
val fromBytes: Either[SchemaError, Person] = codec.decode(bytes)

// String
val toonStr: String = codec.encodeToString(person)
val fromStr: Either[SchemaError, Person] = codec.decode("name: Alice\nage: 30")
```

### Summary of Convenience Methods

`BinaryCodec` subclasses (JSON, TOON, MessagePack, Avro, Thrift) expose the following convenience overloads (availability may vary by format):

| Method                                                                   | Description                                      |
|--------------------------------------------------------------------------|--------------------------------------------------|
| `encode(value): Array[Byte]`                                             | Encode to a byte array                           |
| `decode(input: Array[Byte]): Either[SchemaError, A]`                     | Decode from a byte array                         |
| `decode(input: Array[Byte], from: Int, to: Int): Either[SchemaError, A]` | Decode from a byte array slice                   |
| `encode(value, output: ByteBuffer): Unit`                                | Encode into a `ByteBuffer`                       |
| `decode(input: ByteBuffer): Either[SchemaError, A]`                      | Decode from a `ByteBuffer`                       |
| `encode(value, output: OutputStream): Unit`                              | Encode into an `OutputStream` (JSON, TOON, Avro) |
| `decode(input: InputStream): Either[SchemaError, A]`                     | Decode from an `InputStream` (JSON, TOON, Avro)  |
| `encodeToString(value): String`                                          | Encode to a `String` (JSON, TOON)                |
| `decode(input: String): Either[SchemaError, A]`                          | Decode from a `String` (JSON, TOON)              |

The `String`-based methods are available on text-oriented binary codecs (JSON, TOON) but not on purely binary formats like Avro or Thrift.

## Configuring Codecs

Format-specific derivers support configuration options that control encoding behavior. Instead of passing a `Format` object to `derive`, you pass a configured `Deriver`:

### JSON Configuration

```scala

case class Person(
  firstName: String,
  lastName: String,
  middleName: Option[String] = None
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val customDeriver = JsonCodecDeriver
  .withFieldNameMapper(NameMapper.SnakeCase)
  .withTransientNone(true)
  .withRejectExtraFields(true)

val codec = Schema[Person].derive(customDeriver)

// Encodes as: {"first_name":"Alice","last_name":"Smith"}
// (middleName omitted because it is None and transientNone is true)
val json = codec.encodeToString(Person("Alice", "Smith"))
```

| Option                          | Description                                            | Default    |
|---------------------------------|--------------------------------------------------------|------------|
| `withFieldNameMapper`           | Transform field names (Identity, SnakeCase, KebabCase) | `Identity` |
| `withCaseNameMapper`            | Transform variant/case names                           | `Identity` |
| `withDiscriminatorKind`         | ADT discriminator style (Key, Field, None)             | `Key`      |
| `withRejectExtraFields`         | Error on unknown fields during decoding                | `false`    |
| `withEnumValuesAsStrings`       | Encode enum values as strings                          | `true`     |
| `withTransientNone`             | Omit `None` values from output                         | `true`     |
| `withTransientEmptyCollection`  | Omit empty collections from output                     | `true`     |
| `withTransientDefaultValue`     | Omit fields with default values                        | `true`     |
| `withRequireOptionFields`       | Require optional fields in input                       | `false`    |
| `withRequireCollectionFields`   | Require collection fields in input                     | `false`    |
| `withRequireDefaultValueFields` | Require fields with defaults in input                  | `false`    |

### TOON Configuration

```scala

case class Person(
  firstName: String,
  lastName: String
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val customDeriver = ToonCodecDeriver
  .withFieldNameMapper(NameMapper.SnakeCase)
  .withArrayFormat(ArrayFormat.Tabular)
  .withDiscriminatorKind(DiscriminatorKind.Field("type"))

val codec = Schema[Person].derive(customDeriver)
```

## Error Handling

All `decode` operations return `Either[SchemaError, A]`. `SchemaError` includes path information that pinpoints where in the data structure decoding failed:

```scala

case class Address(street: String, city: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(JsonFormat)

// Missing required field
val result = codec.decode("""{"name":"Alice","address":{}}""")

result match {
  case Right(person) => println(person)
  case Left(error)   => error.errors.foreach(e => println(s"Error: ${e.message}"))
}
```

---

## Combinators

The `combinators` module provides compile-time typeclasses for composing and decomposing values in type-safe ways. Each module focuses on a specific domain: tuples, Either types, and union types.

## Overview

The combinators module consists of three core modules:

- **Tuples** - Tuple composition with automatic flattening and separation
- **Eithers** - Either canonicalization to left-nested form
- **Unions** - Union type operations (Scala 3 only)

Each module provides:
- A unified typeclass (e.g., `Tuples.Tuples[L, R]`) that provides both `combine` and `separate` operations
- A convenience method `combine` (the `separate` operation is available on the typeclass instance)

All typeclasses are derived automatically via compile-time resolution and provide zero-cost abstractions.

## Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-combinators" % "<version>"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-combinators" % "<version>"
```

Supported platforms:
- **Tuples, Eithers**: JVM, Scala.js (Scala 2.13 and 3.x)
- **Unions**: JVM, Scala.js (Scala 3 only)

## Tuples

The `Tuples` module combines values into flat tuples and separates them back.

### combine

`Tuples.Tuples[L, R]` combines two values into a flattened tuple.

```scala

// Basic combination
val result1: (Int, String) = Tuples.combine(1, "hello")

// Tuple flattening
val result2: (Int, String, Boolean) = Tuples.combine((1, "hello"), true)

// Deep flattening (Scala 3)
val result3: (Int, String, Boolean, Double) = Tuples.combine((1, "hello"), (true, 3.14))
```

#### Identity Handling

Unit and EmptyTuple values are automatically eliminated:

```scala

// Unit on left - returns right value
val result1: Int = Tuples.combine((), 42)

// Unit on right - returns left value
val result2: String = Tuples.combine("hello", ())

// EmptyTuple identity (Scala 3)
val result3: String = Tuples.combine(EmptyTuple, "world")
```

#### Tuple Flattening

Nested tuples are automatically flattened:

```scala

// Tuple + value flattens to larger tuple
val result1: (Int, String, Boolean) = Tuples.combine((1, "a"), true)

// Tuple + tuple concatenates (Scala 3 - recursive flattening)
val result2: (Int, String, Boolean, Double) = Tuples.combine((1, "a"), (true, 3.14))

// Scala 2 - flattens left tuple only
val result3: (Int, String, (Boolean, Double)) = Tuples.combine((1, "a"), (true, 3.14))
```

### separate

`separate` is accessed via the unified typeclass instance and splits a tuple into its init (all but last) and last element.

```scala

// 2-tuple separation
val t2 = summon[Tuples.Tuples[Int, String]]  // Scala 3
// or: implicitly[Tuples.Tuples[Int, String]]  // Scala 2
val (left1, right1): (Int, String) = t2.separate((1, "hello"))
// left1 = 1, right1 = "hello"

// 3-tuple separation
val t3 = summon[Tuples.Tuples[(Int, String), Boolean]]
val (left2, right2): ((Int, String), Boolean) = t3.separate((1, "hello", true))
// left2 = (1, "hello"), right2 = true

// 4-tuple separation
val t4 = summon[Tuples.Tuples[(Int, String, Boolean), Double]]
val (left3, right3): ((Int, String, Boolean), Double) = t4.separate((1, "hello", true, 3.14))
// left3 = (1, "hello", true), right3 = 3.14

```
### Type-Level Operations

The output type is computed at compile time via the `Out` type member:

```scala

// Access the combiner with explicit output type
val combiner: Tuples.Tuples.WithOut[Int, String, (Int, String)] = 
  summon[Tuples.Tuples[Int, String]]

// Access with explicit types
val instance: Tuples.Tuples.WithOut[Int, String, (Int, String)] = 
  summon[Tuples.Tuples[Int, String]]
```

### Scala 2 vs Scala 3 Differences

| Feature | Scala 2.13 | Scala 3.x |
|---------|------------|-----------|
| Maximum tuple arity | 22 | Unlimited |
| Tuple flattening | Left tuple only | Recursive both sides |
| EmptyTuple identity | Not available | Supported |

## Eithers

The `Eithers` module canonicalizes Either types to left-nested form and separates them.

### combine

`Eithers.Eithers[L, R]` transforms an `Either[L, R]` into its left-nested canonical form.

```scala

// Atomic Either - unchanged
val result1: Either[Int, String] = Eithers.combine(Left(42): Either[Int, String])

// Right-nested Either - reassociates to left-nested
val input: Either[Int, Either[String, Boolean]] = Right(Right(true))
val result2: Either[Either[Int, String], Boolean] = Eithers.combine(input)
// Right(Right(true)) becomes Right(true)

// Left(42) becomes Left(Left(42))
val input2: Either[Int, Either[String, Boolean]] = Left(42)
val result3: Either[Either[Int, String], Boolean] = Eithers.combine(input2)
```

#### Canonical Form

The canonical form is always left-nested:

```
Right-nested input:          Left-nested output:
Either[A, Either[B, C]]  =>  Either[Either[A, B], C]
Either[A, Either[B, Either[C, D]]]  =>  Either[Either[Either[A, B], C], D]
```

This transformation preserves values while reassociating the structure:
- `Left(a)` → `Left(Left(a))`
- `Right(Left(b))` → `Left(Right(b))`
- `Right(Right(c))` → `Right(c)`

### separate

`separate` is accessed via the unified typeclass instance and peels the rightmost alternative from a canonical Either:

```scala

val e = summon[Eithers.Eithers[Int, String]]
val input: Either[Int, String] = Left(42)
val result: Either[Int, String] = e.separate(e.combine(input))
```

### Use Cases

Eithers canonicalization is useful for:
- **Schema sum type encoding** - Uniform representation of sealed traits
- **Error handling composition** - Combining error types systematically
- **Cross-version compatibility** - Works identically on Scala 2 and 3

## Unions (Scala 3 Only)

The `Unions` module converts between Either types and Scala 3 union types.

### combine

`Unions.Unions[L, R]` converts an `Either[L, R]` to a union type `L | R`:

```scala

val either: Either[Int, String] = Left(42)
val union: Int | String = Unions.combine(either)
// Result: 42 (typed as Int | String)

val either2: Either[Int, String] = Right("hello")
val union2: Int | String = Unions.combine(either2)
// Result: "hello" (typed as Int | String)
```

### separate

`separate` is accessed via the unified typeclass instance and discriminates a union type back to Either:

```scala

val u = summon[Unions.Unions.WithOut[Int, String, Int | String]]
val result: Either[Int, String] = u.separate(42: Int | String)
// Result: Left(42)

val result2: Either[Int, String] = u.separate("hello": Int | String)
// Result: Right("hello")
```
### Same-Type Rejection

Union types collapse same types (`A | A` = `A`), making them ambiguous. The separator rejects overlapping types at compile time:

```scala

// Compile error: Union types must contain unique types
// val u = summon[Unions.Unions.WithOut[Int, Int, Int | Int]]

// Use Either for same-type alternation instead:

val either: Either[Int, Int] = Left(1)  // Distinguishable via Left/Right
```

### Type Erasure Caveat

Union discrimination relies on runtime type tests, which are fragile for erased types:

```scala
// Problematic: List[Int] and List[String] erase to List
val value: List[Int] | List[String] = List(1, 2, 3)
// Runtime cannot distinguish List[Int] from List[String]

// Safe: Use distinct concrete types
val value: Int | String = 42  // Works reliably
```

## Generic Usage Patterns

### With Implicit Parameters (Scala 2)

```scala

def combineAll[A, B, C](a: A, b: B, c: C)(
  implicit ab: Tuples.Tuples[A, B],
           abc: Tuples.Tuples[ab.Out, C]
): abc.Out = {
  val step1 = ab.combine(a, b)
  abc.combine(step1, c)
}

val result = combineAll(1, "hello", true)
// result: (Int, String, Boolean)
```

### With Context Parameters (Scala 3)

```scala

def combineAll[A, B, C](a: A, b: B, c: C)(using
  ab: Tuples.Tuples[A, B],
  abc: Tuples.Tuples[ab.Out, C]
): abc.Out =
  val step1 = ab.combine(a, b)
  abc.combine(step1, c)

val result = combineAll(1, "hello", true)
// result: (Int, String, Boolean)
```

### Path-Dependent Types

The `Out`, `Left`, and `Right` type members are path-dependent:

```scala

def process[L, R](l: L, r: R)(using t: Tuples.Tuples[L, R]): (L, R) =
  t.separate(t.combine(l, r))

val result: (Int, String) = process(1, "hello")
```

### Type Aliases for Clarity

```scala

// Typeclass with known output type
type IntStringTuples = Tuples.Tuples.WithOut[Int, String, (Int, String)]

// Typeclass with known left/right types
type TripleTuples = Tuples.Tuples.WithOut[(Int, String), Boolean, (Int, String, Boolean)]
```

## Performance Characteristics

| Module | Time Complexity | Notes |
|--------|-----------------|-------|
| Tuples.combine | O(1) to O(n) | O(1) for small tuples; O(n) for flattening nested tuples |
| Tuples.separate | O(n) | Splits tuple at size-1 position |
| Eithers.combine | O(d) | d = nesting depth of right-nested Either |
| Eithers.separate | O(d) | Same as combine (delegates to combiner) |
| Unions.combine | O(1) | Direct Either fold |
| Unions.separate | O(1) | Single type test |

All operations are pure and allocation-minimal.

## Cross-Version Summary

| Feature | Scala 2.13 | Scala 3.x |
|---------|------------|-----------|
| Tuples.Tuples | Yes (max 22) | Yes (unlimited) |
| Eithers.Eithers | Yes | Yes |
| Unions.Unions | No | Yes |
| Recursive tuple flattening | No | Yes |
| EmptyTuple handling | No | Yes |

---

## Context

`Context[+R]` is a type-indexed heterogeneous collection that stores values of different types, indexed by their types, with compile-time type safety for lookups. It provides an immutable, cache-aware dependency container where the phantom type `R` (using intersection types) tracks which types are present.

The core type looks like this:

```scala
// Signature (showing public API structure, not actual implementation)
final class Context[+R] {
  def size: Int
  def isEmpty: Boolean
  def nonEmpty: Boolean
  def get[A >: R](implicit ev: IsNominalType[A]): A
  def getOption[A](implicit ev: IsNominalType[A]): Option[A]
  def add[A](a: A)(implicit ev: IsNominalType[A]): Context[R & A]
  def update[A >: R](f: A => A)(implicit ev: IsNominalType[A]): Context[R]
  def ++[R1](that: Context[R1]): Context[R & R1]
  def prune[A >: R](implicit ev: IsNominalIntersection[A]): Context[A]
  override def toString: String
}
```

Key properties: covariant (`+R`), immutable, cached for repeated lookups, supports only nominal types.

## Overview

Context serves as a type-safe registry for heterogeneous dependencies. Here's a quick example:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)
```

We can create and retrieve values by type:

```scala
val ctx: Context[Config & Logger & Metrics] = Context(
  Config(debug = true),
  Logger("app"),
  Metrics(count = 42)
)
// ctx: Context[Config & Logger & Metrics] = Context(repl.MdocSession.MdocApp.Metrics -> Metrics(42), repl.MdocSession.MdocApp.Logger -> Logger(app), repl.MdocSession.MdocApp.Config -> Config(true))

val config: Config = ctx.get[Config]
// config: Config = Config(true)
val logger: Logger = ctx.get[Logger]
// logger: Logger = Logger("app")
```

This ASCII diagram shows how Context maps types to values:

```
┌─────────────────────────────────┐
│         Context[R]              │
│  (Type-Indexed Store)           │
├─────────────────────────────────┤
│  Config         → Config(true)  │
│  Logger         → Logger("app") │
│  Metrics        → Metrics(42)   │
├─────────────────────────────────┤
│ ✓ Type-safe retrieval           │
│ ✓ No casting                    │
│ ✓ Cache-aware (O(1) repeats)    │
└─────────────────────────────────┘
```

## Motivation

When building modular applications, we often need to pass multiple dependencies around—a database connection, a config object, a logger, and so on. Existing approaches each have limitations:

**`Map[Class[_], Any]`** — no compile-time safety. You must cast the result and remember which keys you registered:

```scala
// Unsafe approach — easy to make mistakes
val deps = scala.collection.mutable.Map[Class[_], Any]()
deps(classOf[Config]) = Config(debug = true)
deps(classOf[Logger]) = Logger("app")

val config = deps(classOf[Config]).asInstanceOf[Config]  // Manual cast
val db = deps(classOf[Database]) // Runtime error if missing
```

**ZIO's `ZEnvironment`** — type-safe but requires the full ZIO effect system:

```scala
// Requires ZIO context

val makeEnv = for {
  config <- ZIO.service[Config]
  logger <- ZIO.service[Logger]
} yield (config, logger)
```

**Context** — combines compile-time type safety with synchronous, pure code:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)

// Type-safe, no effects needed
val ctx = Context(Config(true), Logger("app"))
val config = ctx.get[Config]  // Compile-time proof it exists
```

## Installation

Add the ZIO Blocks Context module to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-context" % "0.0.33"
```

## Construction

Context provides several ways to create instances. Choose the approach that best fits your use case: start empty and add values incrementally, or construct a fully-populated context directly with `apply`.

### Creating Empty Contexts

Use `Context.empty` to create an empty context with no entries:

```scala

val emptyCtx: Context[Any] = Context.empty
// emptyCtx: Context[Any] = Context()
val isEmpty = emptyCtx.isEmpty
// isEmpty: Boolean = true
```

An empty context has type `Context[Any]` and represents no stored dependencies. This is a useful starting point for incremental construction.

### Creating Multi-Value Contexts with `Context.apply`

`Context.apply` is overloaded to accept 1–10 values and returns a context with type `Context[A1 & A2 & ...]`, reflecting all stored types.

#### Single Value

Create a context with one value:

```scala
case class Config(debug: Boolean)
```

With `Config` defined, we can create a single-value context:

```scala
val single: Context[Config] = Context(Config(debug = true))
// single: Context[Config] = Context(repl.MdocSession.MdocApp1.Config -> Config(true))
```

#### Multiple Values

Create a context with multiple values—the type parameter automatically becomes an intersection of all stored types:

```scala
case class Logger(name: String)
```

With `Config` and `Logger` in scope, we can create a multi-value context:

```scala
val multi: Context[Config & Logger] = Context(
  Config(debug = true),
  Logger("myapp")
)
// multi: Context[Config & Logger] = Context(repl.MdocSession.MdocApp1.Logger -> Logger(myapp), repl.MdocSession.MdocApp1.Config -> Config(true))
```

### Building Incrementally with `Context#add`

For contexts that grow over time, use `Context#add` to build incrementally from an empty context. This is useful when dependencies become available at different points in your initialization:

```scala
val ctx = Context.empty
  .add(Config(debug = false))
  .add(Logger("init"))
// ctx: Context[Config & Logger] = Context(repl.MdocSession.MdocApp1.Logger -> Logger(init), repl.MdocSession.MdocApp1.Config -> Config(false))
```

The context accumulates all added entries:

```scala
val size1 = ctx.size
// size1: Int = 2
```

**When to use `Context#add` vs. `Context.apply`:**
- Use `Context.apply` when you know all dependencies upfront and can construct them together
- Use `Context#add` when dependencies are added incrementally or conditionally

## Core Operations

Context supports inspection, retrieval, and modification operations. All methods are type-safe and leverage the phantom type `R` to track what's stored.

### Inspection

The following methods let you check the contents of a context without retrieving specific values:

#### `Context#size`

Returns the number of entries in the context:

```scala
case class Config(debug: Boolean)
case class Logger(name: String)

val ctx = Context(Config(true), Logger("app"))
```

Get the number of entries in the context:

```scala
val sz = ctx.size
// sz: Int = 2
```

#### `Context#isEmpty`

Returns `true` if the context contains no entries:

```scala
case class Config(debug: Boolean)

val empty = Context.empty
val notEmpty = Context(Config(true))
```

Now check the `isEmpty` status of both contexts:

```scala
val e1 = empty.isEmpty
// e1: Boolean = true
val e2 = notEmpty.isEmpty
// e2: Boolean = false
```

#### `Context#nonEmpty`

Returns `true` if the context contains at least one entry (opposite of `isEmpty`):

```scala
case class Config(debug: Boolean)

val empty = Context.empty
val notEmpty = Context(Config(true))
```

Check the `nonEmpty` status of both contexts:

```scala
val e1 = empty.nonEmpty
// e1: Boolean = false
val e2 = notEmpty.nonEmpty
// e2: Boolean = true
```

### Retrieval

The following methods let you retrieve values from a context by type:

#### `Context#get`

Retrieves a value by type. The type bound `A >: R` ensures that a value of type `A` (or a subtype of `A`) is present at compile time:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)

val ctx = Context(Config(debug = true), Logger("app"), Metrics(100))

// Retrieve by exact type
val config = ctx.get[Config]
```

Or by supertype (subtype matching):

```scala

trait Animal { def sound: String }
case class Dog(name: String) extends Animal {
  def sound = "Woof"
}

val ctxDog = Context(Dog("Buddy"))
val animal = ctxDog.get[Animal]
```

If you attempt to retrieve a type that is not in the context, the code will not compile because the type bound `A >: R` requires it to be present:

```scala
// This is a compile-time error, not a runtime error:
// val metrics: String = ctx.get[String]  // Error: String is not in context type
```

#### `Context#getOption`

Retrieves a value if present, returning `Option[A]`. Unlike `get`, this method does not require the type to be in the context's type parameter. Use it for optional lookups:

```scala
case class Config(debug: Boolean)

val ctx = Context(Config(debug = true))
```

Try to retrieve both an existing type and a missing type:

```scala
val found: Option[Config] = ctx.getOption[Config]
// found: Option[Config] = Some(Config(true))
val missing: Option[String] = ctx.getOption[String]
// missing: Option[String] = None
```

### Modification

All modification methods return a new `Context`—the original remains immutable:

#### `Context#add`

Adds a value to the context, expanding the phantom type by `& A`:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)

val ctx1 = Context(Config(true))
val ctx2 = ctx1.add(Logger("new"))
```

If a value of the same type already exists, it is replaced:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)

val ctx1 = Context(Config(true))
val ctx2 = ctx1.add(Logger("new"))
val ctx3 = ctx2.add(Config(debug = false))
val replaced = ctx3.get[Config]
```

#### `Context#update`

Transforms an existing value if it is present. If the type is not found, the context is returned unchanged:

```scala

case class Metrics(count: Int)

val ctx = Context(Metrics(count = 10))
val updated = ctx.update[Metrics](m => m.copy(count = m.count + 5))
val newCount = updated.get[Metrics].count
```

#### `Context#++ (Union)`

Combines two contexts into a new context containing all entries. When both contexts contain the same type, the value from the right side (second argument) wins:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)

val left = Context(Config(debug = false), Logger("left"))
val right = Context(Config(debug = true), Metrics(99))
val merged = left ++ right
```

#### `Context#prune`

Narrows a context to contain only specified types. All other entries are discarded:

```scala

case class Config(debug: Boolean)
case class Logger(name: String)
case class Metrics(count: Int)

val full = Context(Config(true), Logger("app"), Metrics(100))
val justConfig = full.prune[Config]
val configSize = justConfig.size
```

#### `Context#toString`

Returns a human-readable representation of the context showing all type-value pairs:

```scala
case class Config(debug: Boolean)
case class Logger(name: String)

val ctx = Context(Config(debug = true), Logger("app"))
```

Convert the context to a human-readable string representation:

```scala
val str = ctx.toString
// str: String = "Context(repl.MdocSession.MdocApp1.Logger -> Logger(app), repl.MdocSession.MdocApp1.Config -> Config(true))"
```

## Covariance

`Context` is covariant in its type parameter, meaning `Context[Dog] <: Context[Animal]` when `Dog <: Animal`. This allows passing a more-specific context to code expecting a more-general one:

```scala

trait Animal { def sound: String }
case class Dog(name: String) extends Animal {
  def sound = "Woof"
}

def processAnimal(ctx: Context[Animal]): String = ctx.get[Animal].sound

val dogCtx = Context(Dog("Buddy"))
val sound = processAnimal(dogCtx)
```

Covariance also applies during retrieval—if you request a supertype, the stored subtype is returned.

## Type Safety: IsNominalType

Context only accepts **nominal types**—concrete classes, case classes, traits, and objects. The compiler automatically derives `IsNominalType[A]` for allowed types and rejects unsupported kinds:

**Supported:**
- Classes: `case class Config(...)`
- Traits: `trait Logger`
- Objects: `object Registry`
- Applied types: `List[Int]`, `Map[String, Int]`
- Enums (Scala 3): `enum Color { case Red, Green, Blue }`

**Not supported (compile error):**
- Intersection types: `A & B` (use the context type parameter instead)
- Union types: `A | B`
- Structural types: `{ def foo: Int }`

Attempting to store an unsupported type:

```scala
// This fails at compile time:
// Context.empty.add(null: (String & Int))  // Error: unsupported type
```

## Performance

**Caching**: When a value is retrieved via `get` or `getOption`, the result is cached. Repeated lookups for the same type return the cached value in O(1) time without traversing the entries again.

**Subtype matching**: Supertype lookups scan the entry list linearly to find a compatible subtype. After the first lookup, the result is cached, so subsequent requests for that supertype are O(1).

**Platform optimizations**: The JVM implementation uses `ConcurrentHashMap` for thread-safe caching. The JavaScript platform uses a simpler in-memory mutable hash map for efficient lookups.

## Comparing Approaches

Here is a comparison of Context with related alternatives:

| Feature             | `Map[Class[_], Any]` | `ZEnvironment`   | `Context` |
|---------------------|----------------------|------------------|-----------|
| Type-safe retrieval | ✗ (cast required)    | ✓                | ✓         |
| Compile-time proof  | ✗                    | ✓                | ✓         |
| Effect-free         | ✓                    | ✗ (requires ZIO) | ✓         |
| Immutable           | ✓                    | ✓                | ✓         |
| Cached lookups      | ✗                    | ✓                | ✓         |
| Supertype matching  | ✗                    | ✓                | ✓         |

## Integration with Wire and Scope

`Context` is the dependency carrier in ZIO Blocks' Wire-based dependency injection system. A `Wire[-In, +Out]` describes how to build an output given input dependencies, and contexts supply those dependencies. Wire and Scope work together to provide type-safe, compile-checked dependency injection:

```scala
// Pseudocode illustrating how Context integrates with Wire and Scope

case class Config(debug: Boolean)
case class Logger(name: String)
case class Service(config: Config, logger: Logger)

// Define a wire that requires Config and Logger to build Service
val buildService = Wire.make[Config & Logger, Service]

// Create a context with the required dependencies
val deps = Context(Config(debug = true), Logger("app"))

// Create a scope and instantiate the service
Scope.global.scoped { scope =>
  val service: Service = buildService.make(scope, deps)
}
```

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

**Context construction: creating contexts with apply, empty.add, and inspecting size/isEmpty/nonEmpty**

```scala title="schema-examples/src/main/scala/context/ContextConstructionExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package context

// Context.empty creates an empty, type-safe dependency container.
// Use Context.apply(...) to construct contexts with 1–10 values.
// The phantom type parameter tracks which types are present.
object ContextConstructionExample extends App {

  case class Config(debug: Boolean)
  case class Logger(name: String)
  case class Metrics(count: Int)

  // Start with an empty context.
  show(Context.empty.size)
  show(Context.empty.isEmpty)

  // Create a context with one value.
  val ctx1 = Context(Config(debug = true))
  show(ctx1.size)
  show(ctx1.nonEmpty)

  // Create a context with multiple values (up to 10 supported).
  val ctx2 = Context(
    Config(debug = false),
    Logger("myapp")
  )
  show(ctx2.size)
  show(ctx2.isEmpty)

  // Create a larger context.
  val ctx3 = Context(
    Config(debug = true),
    Logger("prod"),
    Metrics(count = 100)
  )
  show(ctx3.size)

  // Build incrementally from empty using add.
  val ctxBuilt = Context.empty
    .add(Config(debug = false))
    .add(Logger("init"))
    .add(Metrics(count = 0))
  show(ctxBuilt.size)
  show(ctxBuilt.nonEmpty)

  // toString shows the context contents in a readable format.
  show(ctx3.toString)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/context/ContextConstructionExample.scala))

```bash
sbt "schema-examples/runMain context.ContextConstructionExample"
```

**Context retrieval: using get, supertype lookups, and getOption for safe access**

```scala title="schema-examples/src/main/scala/context/ContextRetrievalExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package context

// Context#get[A] retrieves a value by type with compile-time proof of existence.
// Context#getOption[A] retrieves a value if present, returning None if missing.
// Supertype matching allows retrieving a subtype using a more general supertype.
object ContextRetrievalExample extends App {

  case class Config(debug: Boolean)
  case class Logger(name: String)
  case class Metrics(count: Int)

  trait Animal { def sound: String }
  case class Dog(name: String) extends Animal {
    def sound = "Woof"
  }

  // Create a context with multiple values.
  val ctx = Context(
    Config(debug = true),
    Logger("app"),
    Metrics(count = 42)
  )

  // Retrieve by exact type.
  val config: Config = ctx.get[Config]
  show(config)

  val logger: Logger = ctx.get[Logger]
  show(logger)

  val metrics: Metrics = ctx.get[Metrics]
  show(metrics)

  // getOption returns Some if the type is present.
  val configOpt: Option[Config] = ctx.getOption[Config]
  show(configOpt)

  // getOption returns None if the type is missing (safe for optional lookups).
  val missingOpt: Option[String] = ctx.getOption[String]
  show(missingOpt)

  // Supertype matching: retrieve a subtype using a supertype.
  val dogCtx         = Context(Dog("Buddy"))
  val animal: Animal = dogCtx.get[Animal]
  show(animal.sound)

  // getOption also supports supertype matching.
  val animalOpt: Option[Animal] = dogCtx.getOption[Animal]
  show(animalOpt.map(_.sound))

  // Cache efficiency: repeated lookups return the cached instance.
  val first  = ctx.get[Logger]
  val second = ctx.get[Logger]
  show(first eq second)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/context/ContextRetrievalExample.scala))

```bash
sbt "schema-examples/runMain context.ContextRetrievalExample"
```

**Context modification: adding values, updating existing ones, merging contexts, and pruning types**

```scala title="schema-examples/src/main/scala/context/ContextModificationExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package context

// Context is immutable; modification methods return new contexts.
// Context#add expands the context with a new value (or replaces if type exists).
// Context#update transforms a value if present.
// Context#++ merges two contexts (right side wins on conflict).
// Context#prune narrows to specific types.
object ContextModificationExample extends App {

  case class Config(debug: Boolean)
  case class Logger(name: String)
  case class Metrics(count: Int)

  // Start with a simple context.
  val ctx1 = Context(Config(debug = false))
  show(ctx1.size)

  // add expands the context with a new value.
  val ctx2 = ctx1.add(Logger("app"))
  show(ctx2.size)
  show(ctx2.get[Config])
  show(ctx2.get[Logger])

  // add replaces if the type already exists.
  val ctx3 = ctx2.add(Config(debug = true))
  show(ctx3.size)
  show(ctx3.get[Config].debug)

  // update transforms a value if present.
  val ctx4    = ctx3.add(Metrics(count = 100))
  val updated = ctx4.update[Metrics](m => m.copy(count = m.count + 50))
  show(updated.get[Metrics].count)

  // update also works on existing types.
  val configUpdated = updated.update[Config](c => c.copy(debug = false))
  show(configUpdated.get[Config])

  // ++ merges two contexts; right side wins on conflict.
  val left   = Context(Config(debug = false), Logger("left"))
  val right  = Context(Config(debug = true), Metrics(count = 99))
  val merged = left ++ right
  show(merged.size)
  show(merged.get[Config].debug)
  show(merged.get[Logger].name)
  show(merged.get[Metrics].count)

  // prune narrows a context to specific types.
  val full = Context(Config(true), Logger("app"), Metrics(100))
  show(full.size)
  val justConfig = full.prune[Config]
  show(justConfig.size)
  show(justConfig.getOption[Logger])

  // Chaining modifications.
  val chain = Context.empty
    .add(Config(debug = true))
    .add(Logger("chain"))
    .add(Metrics(count = 0))
    .update[Metrics](m => m.copy(count = 42))
  show(chain.size)
  show(chain.get[Metrics].count)
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/context/ContextModificationExample.scala))

```bash
sbt "schema-examples/runMain context.ContextModificationExample"
```

---

## Docs Reference

# Docs Module Reference

Complete API reference for the zio-blocks-docs module - a zero-dependency GitHub Flavored Markdown library.

## Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-docs" % "0.0.33"
```

## Core Types

### Doc

The top-level document container. A `Doc` wraps a `Chunk[Block]` representing the document's block-level elements, plus optional metadata.

```scala
final case class Doc(blocks: Chunk[Block], metadata: Map[String, String] = Map.empty)
```

**Key methods:**
- `++`: Concatenate two documents (merges blocks and metadata, right wins on conflicts)
- `normalize`: Merge adjacent Text nodes and remove empty blocks
- `toHtml`: Render to full HTML5 document (with DOCTYPE, html, head, body tags)
- `toHtmlFragment`: Render to HTML content only (no html/head/body wrapper)
- `toTerminal`: Render with ANSI escape codes for terminal display
- `toString`: Render back to GFM Markdown

**Equality:** Two documents are equal if their normalized forms are equal.

**Example:**
```scala

val doc = Parser.parse("# Hello World").toOption.get
val markdown = doc.toString           // "# Hello World\n"
val html = doc.toHtml                 // Full HTML5 document
val fragment = doc.toHtmlFragment     // Just the content
val terminal = doc.toTerminal         // ANSI colored output
```

### Block

Block-level elements that make up a document:

| Variant | Description |
|---------|-------------|
| `Paragraph(content: Chunk[Inline])` | A paragraph of inline content |
| `Heading(level: HeadingLevel, content: Chunk[Inline])` | ATX heading (H1-H6) |
| `CodeBlock(info: Option[String], code: String)` | Fenced code block with optional language |
| `ThematicBreak` | Horizontal rule (`---`, `***`, `___`) |
| `BlockQuote(content: Chunk[Block])` | Quoted block content |
| `BulletList(items: Chunk[ListItem], tight: Boolean)` | Unordered list |
| `OrderedList(start: Int, items: Chunk[ListItem], tight: Boolean)` | Ordered list with start number |
| `ListItem(content: Chunk[Block], checked: Option[Boolean])` | List item, optionally a task item |
| `HtmlBlock(content: String)` | Raw HTML block |
| `Table(header: TableRow, alignments: Chunk[Alignment], rows: Chunk[TableRow])` | GFM table |

**Note on Lists:** The `tight` parameter indicates whether the list should be rendered without blank lines between items (tight) or with blank lines (loose).

### Inline

Inline elements within blocks:

| Variant | Description |
|---------|-------------|
| `Text(value: String)` | Plain text |
| `Code(value: String)` | Inline code (backticks) |
| `Emphasis(content: Chunk[Inline])` | Italic text (`*text*` or `_text_`) |
| `Strong(content: Chunk[Inline])` | Bold text (`**text**` or `__text__`) |
| `Strikethrough(content: Chunk[Inline])` | Strikethrough (`~~text~~`) |
| `Link(text: Chunk[Inline], url: String, title: Option[String])` | Hyperlink |
| `Image(alt: String, url: String, title: Option[String])` | Image |
| `HtmlInline(content: String)` | Raw inline HTML |
| `SoftBreak` | Soft line break (rendered as space in HTML) |
| `HardBreak` | Hard line break (two spaces or backslash before newline) |
| `Autolink(url: String, isEmail: Boolean)` | Auto-detected URL or email |

**Note:** Both top-level case classes and `Inline.X` nested variants exist for compatibility. They are treated identically.

### HeadingLevel

Heading levels H1 through H6:

```scala
sealed abstract class HeadingLevel(val value: Int)
object HeadingLevel {
  case object H1 extends HeadingLevel(1)
  case object H2 extends HeadingLevel(2)
  case object H3 extends HeadingLevel(3)
  case object H4 extends HeadingLevel(4)
  case object H5 extends HeadingLevel(5)
  case object H6 extends HeadingLevel(6)
  
  def fromInt(n: Int): Option[HeadingLevel]
  def unsafeFromInt(n: Int): HeadingLevel  // Throws on invalid input
}
```

**Example:**
```scala
HeadingLevel.fromInt(2)        // Some(H2)
HeadingLevel.fromInt(7)        // None
HeadingLevel.unsafeFromInt(3)  // H3
HeadingLevel.H1.value          // 1
```

### Alignment

Table column alignment:

```scala
sealed trait Alignment
object Alignment {
  case object None extends Alignment    // Default alignment (---)
  case object Left extends Alignment    // Left aligned (:---)
  case object Center extends Alignment  // Center aligned (:---:)
  case object Right extends Alignment   // Right aligned (---:)
}
```

### TableRow

A row in a table:

```scala
final case class TableRow(cells: Chunk[Chunk[Inline]])
```

Each cell contains a chunk of inline elements, allowing rich formatting within table cells.

## Parsing

### Parser.parse

Parse a Markdown string into a `Doc`:

```scala
object Parser {
  def parse(input: String): Either[ParseError, Doc]
}
```

**Example:**
```scala

val result = Parser.parse("# Hello\n\nThis is **bold**.")
// Right(Doc(Chunk(
//   Heading(H1, Chunk(Text("Hello"))),
//   Paragraph(Chunk(Text("This is "), Strong(Chunk(Text("bold"))), Text(".")))
// )))
```

### Supported Features

The parser supports all GitHub Flavored Markdown features:

- **ATX headings** (# to ######)
- **Fenced code blocks** (``` or ~~~)
- **Thematic breaks** (---, ***, ___)
- **Block quotes** (> prefix)
- **Bullet and ordered lists**
- **Task lists** (- [ ] and - [x])
- **Tables with alignment**
- **Inline formatting** (emphasis, strong, strikethrough, code)
- **Links and images**
- **Autolinks** (`<url>` or plain URLs)
- **HTML blocks and inline HTML**

### Not Supported

- **YAML frontmatter** (causes parse error)
- **Setext headings** (use ATX style with #)
- **Indented code blocks** (use fenced code blocks)
- **Link reference definitions**

### ParseError

Parsing error with location information:

```scala
final case class ParseError(
  message: String,
  line: Int,        // 1-based line number
  column: Int,      // 1-based column number
  input: String     // The line that caused the error
)
```

**Example:**
```scala
Parser.parse("---\ntitle: Test\n---") match {
  case Left(err) => 
    println(s"Error at line ${err.line}: ${err.message}")
    // "Error at line 1: Frontmatter is not supported"
  case Right(doc) => // Process doc
}
```

## Rendering

### Markdown Rendering

Render a `Doc` back to GFM Markdown:

```scala
object Renderer {
  def render(doc: Doc): String
  def renderBlock(block: Block): String
  def renderInlines(inlines: Chunk[Inline]): String
  def renderInline(inline: Inline): String
}
```

**Example:**
```scala
val doc = Parser.parse("# Title\n\nParagraph.").toOption.get
val markdown = Renderer.render(doc)
// "# Title\n\nParagraph.\n\n"
```

The rendered output is GFM-compliant and can be re-parsed to produce an equivalent AST.

### HTML Rendering

Render to HTML5-compliant HTML:

```scala
object HtmlRenderer {
  def render(doc: Doc): String         // Full HTML5 document
  def renderFragment(doc: Doc): String // Content only, no wrapper
  def renderBlock(block: Block): String
  def renderInlines(inlines: Chunk[Inline]): String
  def renderInline(inline: Inline): String
  def escape(s: String): String        // HTML entity escaping
}
```

**Example:**
```scala
val doc = Parser.parse("# Hello\n\n**Bold**").toOption.get

// Full document with <!DOCTYPE html>, <html>, <head>, <body>
val fullHtml = HtmlRenderer.render(doc)

// Just the content: HelloBold
val fragment = HtmlRenderer.renderFragment(doc)
```

**HTML Features:**
- Code blocks with language classes (`language-scala`, etc.)
- Tables with proper alignment styles
- Task list items with disabled checkboxes
- Proper HTML entity escaping for safety

### Terminal Rendering

Render with ANSI escape codes for colorful terminal display:

```scala
object TerminalRenderer {
  def render(doc: Doc): String
  def renderBlock(block: Block): String
  def renderInlines(inlines: Chunk[Inline]): String
  def renderInline(inline: Inline): String
}
```

**Example:**
```scala
val doc = Parser.parse("# Hello\n\nThis is **bold** and *italic*.").toOption.get
val terminal = TerminalRenderer.render(doc)
println(terminal)  // Displays with colors and formatting
```

**ANSI Styling:**
- **Headings:** Bold + colored (H1=red, H2=yellow, H3=green, H4=cyan, H5=blue, H6=magenta)
- **Code blocks:** Gray background
- **Inline code:** Gray background
- **Emphasis:** Italic
- **Strong:** Bold
- **Strikethrough:** Strike-through style
- **Links:** Blue + underlined
- **Block quotes:** Prefixed with │

## String Interpolator

### The md"..." Interpolator

Build documents with compile-time validated Markdown syntax:

```scala

val name = "World"
val greeting = md"# Hello $name"
// Doc(Chunk(Heading(H1, Chunk(Text("Hello World")))))

val items = List("one", "two", "three")
val list = md"""
# My List

${items.map(i => s"- $i").mkString("\n")}
"""
```

The interpolator:
- **Validates syntax at compile time** - invalid markdown causes compilation error
- **Requires ToMarkdown instances** for interpolated values
- **Supports multi-line markdown** with triple quotes

**Example with validation:**
```scala
// This won't compile - invalid heading level
val bad = md"####### Too many hashes"
// Error: Invalid markdown: Invalid heading level: 7 (max is 6)
```

### ToMarkdown Typeclass

Make custom types interpolatable:

```scala
trait ToMarkdown[-A] {
  def toMarkdown(a: A): Inline
}
```

**Built-in instances:**
- `String`, `Int`, `Long`, `Double`, `Boolean` → `Text`
- `Inline` → identity
- `Block` → rendered to markdown then wrapped as `Text`
- `List[A]`, `Vector[A]`, `Seq[A]`, `Chunk[A]` → comma-separated (where `A: ToMarkdown`)

**Custom instance example:**
```scala
case class User(name: String, email: String)

implicit val userToMarkdown: ToMarkdown[User] = user =>
  Text(s"${user.name} <${user.email}>")

val user = User("Alice", "alice@example.com")
val doc = md"Contact: $user"
// Doc(Chunk(Paragraph(Chunk(Text("Contact: Alice <alice@example.com>")))))
```

**Advanced example - custom formatting:**
```scala
case class CodeSnippet(lang: String, code: String)

implicit val codeSnippetToMarkdown: ToMarkdown[CodeSnippet] = snippet =>
  Text(s"```${snippet.lang}\n${snippet.code}\n```")

val snippet = CodeSnippet("scala", "val x = 42")
val doc = md"Here's an example:\n\n$snippet"
```

## Working with the AST

### Building Documents Programmatically

```scala

val doc = Doc(Chunk(
  Heading(HeadingLevel.H1, Chunk(Text("Title"))),
  Paragraph(Chunk(
    Text("This is "),
    Strong(Chunk(Text("important"))),
    Text(".")
  )),
  CodeBlock(Some("scala"), "val x = 42"),
  BulletList(Chunk(
    ListItem(Chunk(Paragraph(Chunk(Text("Item 1")))), None),
    ListItem(Chunk(Paragraph(Chunk(Text("Done")))), Some(true)),
    ListItem(Chunk(Paragraph(Chunk(Text("Todo")))), Some(false))
  ), tight = true)
))
```

### Concatenation

Combine documents with `++`:

```scala
val header = md"# Document Title"
val body = md"Some content here."
val footer = md"---\n*Footer*"

val full = header ++ body ++ footer
```

**Metadata merging:**
```scala
val doc1 = Doc(Chunk(Paragraph(Chunk(Text("A")))), Map("author" -> "Alice"))
val doc2 = Doc(Chunk(Paragraph(Chunk(Text("B")))), Map("version" -> "1.0"))
val combined = doc1 ++ doc2
// combined.metadata == Map("author" -> "Alice", "version" -> "1.0")
```

### Normalization

`normalize` cleans up the AST:
- Merges adjacent `Text` nodes
- Removes empty paragraphs and other empty blocks
- Recursively normalizes nested structures (lists, block quotes, tables)

```scala
val messy = Doc(Chunk(
  Paragraph(Chunk(
    Text("Hello "),
    Text("World")  // Adjacent Text nodes
  )),
  Paragraph(Chunk.empty)  // Empty paragraph
))

val clean = messy.normalize
// Doc(Chunk(Paragraph(Chunk(Text("Hello World")))))
```

**When to normalize:**
- Before comparing documents for equality (equality uses normalized form)
- After programmatic AST construction with potential duplicates
- When cleaning up parsed or generated content

**Note:** `Doc.equals` automatically normalizes both sides, so explicit normalization isn't needed for equality checks.

## Advanced Usage

### Custom Renderers

You can traverse the AST to create custom renderers:

```scala
def customRender(doc: Doc): String = {
  doc.blocks.map {
    case Heading(level, content) => 
      s"${"=" * level.value} ${renderInlines(content)}\n"
    case Paragraph(content) => 
      renderInlines(content) + "\n\n"
    case _ => 
      Renderer.renderBlock(_)
  }.mkString
}
```

### Extracting Information

Pattern match on the AST to extract structured data:

```scala
def extractHeadings(doc: Doc): List[(Int, String)] = {
  doc.blocks.collect {
    case Heading(level, content) =>
      (level.value, Renderer.renderInlines(content))
  }.toList
}

def extractLinks(doc: Doc): List[String] = {
  def findLinksInInlines(inlines: Chunk[Inline]): List[String] = {
    inlines.toList.flatMap {
      case Link(_, url, _) => List(url)
      case Strong(content) => findLinksInInlines(content)
      case Emphasis(content) => findLinksInInlines(content)
      case _ => Nil
    }
  }
  
  doc.blocks.flatMap {
    case Paragraph(content) => findLinksInInlines(content)
    case Heading(_, content) => findLinksInInlines(content)
    case _ => Nil
  }.toList
}
```

### Transforming Documents

Apply transformations to the AST:

```scala
def uppercaseHeadings(doc: Doc): Doc = {
  val transformedBlocks = doc.blocks.map {
    case Heading(level, content) =>
      val upperContent = content.map {
        case Text(value) => Text(value.toUpperCase)
        case other => other
      }
      Heading(level, upperContent)
    case other => other
  }
  Doc(transformedBlocks, doc.metadata)
}
```

## Best Practices

### Parsing
- Always handle `Either[ParseError, Doc]` - don't assume parsing succeeds
- For user input, display parse errors with line/column information
- Use the interpolator for static markdown (compile-time validation)

### Building
- Prefer the `md"..."` interpolator for compile-time safety
- Use programmatic construction for dynamic content
- Call `normalize` after complex programmatic construction

### Rendering
- Use `toHtmlFragment` when embedding in existing HTML pages
- Use `render` (full HTML) for standalone documents
- Use `toTerminal` for CLI tools and REPLs
- Use `toString` when you need markdown output

### Performance
- Parse once, render multiple times if possible
- Normalization is not free - don't call it unnecessarily
- The AST is immutable - transformations create new instances

---

## DynamicOptic

`DynamicOptic` is a runtime path through nested data structures in ZIO Blocks. It is the untyped,
dynamically-constructed counterpart to the typed [`Optic[S, A]`](./optics.md). Where typed optics are bound to specific
Scala types at compile time, a `DynamicOptic` is a sequence of **navigation steps** that can be built, composed, and
applied at runtime:

```scala

// Build a path: .users[0].name
val path = DynamicOptic.root.field("users").at(0).field("name")

// Navigate a DynamicValue
val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice"))
  )
)

val result = data.get(path).one
// Right(DynamicValue.Primitive(PrimitiveValue.String("Alice")))
```

:::tip
For a practical example of extracting column names from `DynamicOptic` paths to generate SQL, see the [Query DSL Part 2: SQL Generation](../guides/query-dsl-sql.md) guide.
:::

## Motivation

Most of the time you work with typed `Optic[S, A]` values — they are statically verified and provide type-safe
get/set/modify operations. `DynamicOptic` is useful when:

- **The path is determined at runtime** — user input, configuration, query parameters
- **You are working with `DynamicValue`** — navigating, modifying, or querying schema-less data
- **You need schema introspection** — walking a `Schema` or `Reflect` structure programmatically
- **Serializing optic paths** — `DynamicOptic` has its own `Schema[DynamicOptic]` and can be persisted

The relationship between typed and dynamic optics:

```
   Typed World                        Dynamic World
┌──────────────────┐             ┌──────────────────┐
│  Optic[S, A]     │─────────────│  DynamicOptic    │
│                  │  toDynamic  │                  │
│  Lens, Prism,    │             │  Sequence of     │
│  Optional,       │             │  Node steps      │
│  Traversal       │             │                  │
├──────────────────┤             ├──────────────────┤
│  Operates on     │             │  Operates on     │
│  typed values    │             │  DynamicValue,   │
│  (case classes,  │             │  Schema,         │
│  sealed traits)  │             │  Reflect         │
└──────────────────┘             └──────────────────┘
```

The `Optic[S, A]` and `DynamicOptic` types serve complementary roles in ZIO Blocks' optics system. `Optic[S, A]` provides compile-time type safety through macros or manual construction, operates on typed Scala values, and can be converted to a `DynamicOptic` via the `optic.toDynamic` method. In contrast, `DynamicOptic` performs runtime type checking and is constructed through a builder API or [path interpolator](../path-interpolator.md), operating directly on [DynamicValue](./dynamic-value.md), [Schema](./schema.md), and [Reflect](./reflect.md) representations. 

## Design & Structure

`DynamicOptic` is modeled as a case class wrapping an `IndexedSeq[Node]`, where each `Node` represents one step in a navigation path:

```
DynamicOptic(nodes: IndexedSeq[Node])

sealed trait DynamicOptic.Node
 ├── Field(name: String)            — named field in a record
 ├── Case(name: String)             — specific case in a variant
 ├── AtIndex(index: Int)            — element at index in a sequence
 ├── AtIndices(index: Seq[Int])     — elements at multiple indices
 ├── AtMapKey(key: DynamicValue)    — value at a specific map key
 ├── AtMapKeys(keys: Seq[DynamicValue]) — values at multiple map keys
 ├── Elements                       — all elements in a sequence (wildcard)
 ├── MapKeys                        — all keys in a map (wildcard)
 ├── MapValues                      — all values in a map (wildcard)
 └── Wrapped                        — inner value of a wrapper/newtype
```

Each of the `DynamicOptic` nodes represents a different way to navigate through a data structure. Here's a reference table for the nodes, their navigation semantics, and string representations:

| Node                  | Navigates             | `toString`   | `toScalaString`     |
|-----------------------|-----------------------|--------------|---------------------|
| `Field("name")`       | Record field          | `.name`      | `.name`             |
| `Case("Email")`       | Variant case          | `<Email>`    | `.when[Email]`      |
| `AtIndex(0)`          | Sequence element      | `[0]`        | `.at(0)`            |
| `AtIndices(Seq(0,2))` | Multiple elements     | `[0,2]`      | `.atIndices(0, 2)`  |
| `AtMapKey(k)`         | Map entry by key      | `{"host"}`   | `.atKey("host")`    |
| `AtMapKeys(ks)`       | Multiple entries      | `{"a", "b"}` | `.atKeys("a", "b")` |
| `Elements`            | All sequence elements | `[*]`        | `.each`             |
| `MapKeys`             | All map keys          | `{*:}`       | `.eachKey`          |
| `MapValues`           | All map values        | `{*}`        | `.eachValue`        |
| `Wrapped`             | Wrapper inner value   | `.~`         | `.wrapped`          |

Key design decisions:

- **Map keys as `DynamicValue`** — Map keys can be any type (`String`, `Int`, `Boolean`, etc.), so `AtMapKey` stores the key as a `DynamicValue` to remain type-agnostic.
- **Dual rendering** — `toString` produces a compact interpolator syntax (`.field[0]{key}`), while `toScalaString` produces Scala method-call syntax (`.field.at(0).atKey(key)`). The compact format is designed to be copy-pasteable into the `p"..."` string interpolator, while the Scala format is used in error messages.
- **Every typed `Optic` has a `toDynamic` method** — This bridges the typed and untyped worlds, allowing any `Lens`, `Prism`, `Optional`, or `Traversal` to produce its `DynamicOptic` representation.

## Construction

### Starting Points

Every `DynamicOptic` starts from `root` (the identity/empty path) or one of the pre-built singletons:

```scala

val root = DynamicOptic.root // empty path: "."
val elems = DynamicOptic.elements // "[*]"
val keys = DynamicOptic.mapKeys // "{*:}"
val values = DynamicOptic.mapValues // "{*}"
val inner = DynamicOptic.wrapped // ".~"
```

### Builder Methods

Chain builder methods on `DynamicOptic.root` (or any existing optic) to construct paths fluently:

```scala

// Navigate into a record field, then a sequence index, then another field
val path = DynamicOptic.root.field("users").at(0).field("name")
// toString: .users[0].name

// Navigate into a map with a typed key
val configPath = DynamicOptic.root.field("config").atKey("host")
// toString: .config{"host"}

// Navigate into a variant case, then a field
val resultPath = DynamicOptic.root.field("result").caseOf("Success").field("value")
// toString: .result<Success>.value

// Select all elements, then a field on each
val allEmails = DynamicOptic.root.field("users").elements.field("email")
// toString: .users[*].email
```

| Method           | Node Produced        | Example               |
|------------------|----------------------|-----------------------|
| `.field(name)`   | `Field(name)`        | `.field("street")`    |
| `.caseOf(name)`  | `Case(name)`         | `.caseOf("Email")`    |
| `.at(index)`     | `AtIndex(index)`     | `.at(0)`              |
| `.atIndices(i*)` | `AtIndices(indices)` | `.atIndices(0, 2, 5)` |
| `.atKey[K](key)` | `AtMapKey(dv)`       | `.atKey("host")`      |
| `.atKeys[K](k*)` | `AtMapKeys(dvs)`     | `.atKeys("a", "b")`   |
| `.elements`      | `Elements`           | `.elements`           |
| `.mapKeys`       | `MapKeys`            | `.mapKeys`            |
| `.mapValues`     | `MapValues`          | `.mapValues`          |
| `.wrapped`       | `Wrapped`            | `.wrapped`            |

Note: `.atKey` and `.atKeys` require an implicit `Schema[K]` to convert the typed key to a `DynamicValue`.

### Path Interpolator

The [`p"..."` path interpolator](../path-interpolator.md) provides a concise compile-time syntax for building
`DynamicOptic` values:

```scala

// Equivalent builder vs interpolator
val builderPath     : DynamicOptic = DynamicOptic.root.field("users").at(0).field("name")
val interpolatorPath: DynamicOptic = p".users[0].name"
// Both produce the same DynamicOptic

// Wildcards
val allEmails = p".users[*].email"

// Map access
val host = p""".config{"host"}"""

// Variant cases
val success = p".result<Success>.value"

// Complex path
val complex = p""".groups[*].members[0].contacts{"email"}"""
```

See [Path Interpolator](../path-interpolator.md) for the full syntax reference.

### From Typed Optics

Every typed `Optic[S, A]` can be converted to a `DynamicOptic` via `toDynamic`:

```scala

case class Address(street: String, city: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, address: Address)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name: Lens[Person, String] = optic(_.name)
  val street: Lens[Person, String] = optic(_.address.street)
}

val dynamicName: DynamicOptic = Person.name.toDynamic
// toString: .name

val dynamicStreet: DynamicOptic = Person.street.toDynamic
// toString: .address.street
```

See [Optics](./optics.md) for more on typed optics.

## Operations

### Composition

`DynamicOptic` values compose via the `apply` method, which concatenates their node sequences:

```scala

val users = DynamicOptic.root.field("users")
val first = DynamicOptic.root.at(0)
val name = DynamicOptic.root.field("name")

// Compose three paths into one
val fullPath = users(first)(name)
// toString: .users[0].name

// Compose with pre-built singletons
val allUserNames = users(DynamicOptic.elements)(name)
// toString: .users[*].name

// Compose with interpolator-built paths
val emails = users(p"[*].email")
// toString: .users[*].email
```

### DynamicValue Operations

`DynamicOptic` is the path argument for all `DynamicValue` operations: `get(path)` for retrieval (with `.one` or `.toChunk`), `modify(path)(f)` for transformation, `set(path, value)` for replacement, `delete(path)` for removal, and `insert(path, value)` for addition. By default, the mutating operations (`modify`, `set`, `delete`, `insert`) are lenient and return the original value unchanged if the path doesn't resolve, whereas `get(path)` yields a failing `DynamicValueSelection` on a missing path (though calling `.toChunk` on it will produce an empty chunk). Each operation also has a strict `*OrFail` variant returning `Either[SchemaError, DynamicValue]` with error details on failure.

For example, the `DynamicValue#get` method uses `DynamicOptic` to navigate and extract values:

```scala

val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice"), "age" -> DynamicValue.int(30)),
    DynamicValue.Record("name" -> DynamicValue.string("Bob"), "age" -> DynamicValue.int(25))
  )
)

// Get a single value
val firstName = data.get(p".users[0].name").one
// Right(DynamicValue.Primitive(PrimitiveValue.String("Alice")))

// Get all matching values (wildcard)
val allNames = data.get(p".users[*].name").toChunk
// Chunk(DynamicValue.string("Alice"), DynamicValue.string("Bob"))
```

### Schema & Reflect Operations

`DynamicOptic` can navigate schema structures, not just values. This is useful for schema introspection and
metaprogramming.

1. The `Schema#get` method takes a `DynamicOptic` path and returns the `Reflect` for the nested component at that path, if it exists. This allows you to programmatically explore the structure of a schema:

```scala

case class Address(street: String, city: String)

object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

case class Person(name: String, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Get the Reflect for the "street" field inside Person
val streetReflect: Option[Reflect.Bound[?]] =
  Schema[Person].get(p".address.street")
```

2. The `DynamicSchema#get` method works similarly, allowing you to navigate a `DynamicSchema` structure:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynSchema = Schema[Person].toDynamicSchema

val nameReflect: Option[Reflect.Unbound[_]] =
  dynSchema.get(p".name")
```

3. By applying a `DynamicOptic` directly to a `Reflect` value, you can navigate the reflected structure of a type:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val reflect = Schema[Person].reflect
val path = p".name"

// Using the DynamicOptic's apply method
val result: Option[Reflect[?, ?]] = path(reflect)
```

## Failure Path in OpticCheck

Typed optics use `DynamicOptic` internally for error reporting. When a typed optic operation fails (e.g., a `Prism` encounters the wrong case), the error includes the `DynamicOptic` path to pinpoint exactly where the failure occurred.

```
OpticCheck(errors: ::[Single])
 └── Single (sealed trait)
      ├── Error (sealed trait)
      │    ├── UnexpectedCase    — prism matched wrong variant case
      │    └── WrappingError     — wrapper conversion failed
      └── Warning (sealed trait)
           ├── EmptySequence     — traversal over empty sequence
           ├── SequenceIndexOutOfBounds — index beyond sequence length
           ├── MissingKey        — map key not found
           └── EmptyMap          — traversal over empty map
```

Every `OpticCheck.Single` carries two `DynamicOptic` paths:

- **`full`** — The complete optic path that was being evaluated
- **`prefix`** — The path up to the point where the error occurred

Error messages use `toScalaString` for human-readable output:

```
During attempted access at .when[Email].subject,
encountered an unexpected case at .when[Email]:
expected Email, but got Push
```

## Path String Syntax

`DynamicOptic` provides two string formats for different contexts:

- **`toString`** — Compact path syntax matching the [`p"..."` interpolator](../path-interpolator.md) format
- **`toScalaString`** — Scala method call syntax used in error messages

| Node                      | `toString`   | `toScalaString`     |
|---------------------------|--------------|---------------------|
| `Field("name")`           | `.name`      | `.name`             |
| `Case("Email")`           | `<Email>`    | `.when[Email]`      |
| `AtIndex(0)`              | `[0]`        | `.at(0)`            |
| `AtIndices(Seq(0, 2))`    | `[0,2]`      | `.atIndices(0, 2)`  |
| `AtMapKey(string "host")` | `{"host"}`   | `.atKey("host")`    |
| `AtMapKey(int 42)`        | `{42}`       | `.atKey(42)`        |
| `AtMapKey(bool true)`     | `{true}`     | `.atKey(true)`      |
| `AtMapKey(char 'a')`      | `{'a'}`      | `.atKey('a')`       |
| `AtMapKeys(strings)`      | `{"a", "b"}` | `.atKeys("a", "b")` |
| `Elements`                | `[*]`        | `.each`             |
| `MapKeys`                 | `{*:}`       | `.eachKey`          |
| `MapValues`               | `{*}`        | `.eachValue`        |
| `Wrapped`                 | `.~`         | `.wrapped`          |
| root                      | `.`          | `.`                 |

## Serialization

`DynamicOptic` has an implicit `Schema[DynamicOptic]` defined in its companion object, which means it can be serialized and deserialized just like any other schema-equipped type. This enables storing optic paths in databases, sending them over the wire, or including them in configuration files:

```scala

// Schema[DynamicOptic] is available implicitly
val opticSchema: Schema[DynamicOptic] = Schema[DynamicOptic]

// Convert to/from DynamicValue for serialization
val path = p".users[0].name"
val serialized: DynamicValue = opticSchema.toDynamicValue(path)
```

## See Also

- [DynamicSchema](./dynamic-schema.md) — use `DynamicSchema#get` with a `DynamicOptic` to navigate schema trees and inspect nested type structures.

---

## DynamicSchema

`DynamicSchema` is a **type-erased schema container** that wraps a `Reflect.Unbound[_]` tree — all the structural information from a `Schema[A]` (field names, case names, type identities, validations, annotations) with no Scala functions attached. The two fundamental uses are validating `DynamicValue` instances at runtime, and transporting schemas across process boundaries.

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_])
```

`DynamicSchema`:

- Holds the full structural shape of a type without capturing constructors or deconstructors
- Can validate any `DynamicValue` against its structure using `DynamicSchema#check` and `DynamicSchema#conforms`
- Can be serialized to a `DynamicValue` and deserialized back, making it storable in databases or transmittable over the network
- Can be rehydrated into a fully operational `Schema[A]` by supplying runtime bindings via `DynamicSchema#rebind`

## Motivation

A `Schema[A]` carries both *structure* (field names, type identities, validations) and *behaviour* (constructors and deconstructors as Scala closures). Closures cannot cross process boundaries. `DynamicSchema` is the structural half — a schema stripped of all Scala functions:

```
Schema[OrderPlaced]   ←── compile-time type, closures attached
       │
       │  Schema[A]#toDynamicSchema
       ▼
DynamicSchema         ←── serializable, no closures
(Reflect.Unbound[_])       stored in registry or sent over the wire
       │
       │  DynamicSchema.toDynamicValue / DynamicSchema.fromDynamicValue
       ▼
DynamicValue          ←── uniform, format-neutral blob
       │
       │  DynamicSchema#fromDynamicValue (on the consumer side)
       ▼
DynamicSchema         ←── reconstructed from storage
       │
       │  DynamicSchema#rebind[OrderPlaced](resolver)
       ▼
Schema[OrderPlaced]   ←── operational again, can encode and decode
```

This pattern enables schema registries: the Checkout Service registers its event schema on startup; the Fulfillment Service fetches it and rebinds it against its own type definitions, guaranteeing it uses the exact schema that was in effect when the event was encoded. See [BindingResolver](./binding-resolver.md) for the complete rebinding API.

## Creating a DynamicSchema

There are two ways to obtain a `DynamicSchema`: strip bindings from an existing typed schema, or reconstruct one from a serialized blob.

### `Schema[A]#toDynamicSchema`

`Schema[A]#toDynamicSchema` strips all runtime bindings from a typed schema, returning the structural skeleton as a `DynamicSchema`. The method is defined on `Schema`:

```scala
trait Schema[A] {
  def toDynamicSchema: DynamicSchema
}
```

This is the standard entry point. All structural information — field names, case names, type IDs, `Validation` constraints, `Modifier` annotations, docs, default values, and examples — is preserved:

```scala

case class Address(street: String, city: String)
case class Person(name: String, age: Int, address: Address)

object Address { implicit val schema: Schema[Address] = Schema.derived[Address] }
object Person  { implicit val schema: Schema[Person]  = Schema.derived[Person]  }

val dynamic: DynamicSchema = Schema[Person].toDynamicSchema
```

We can inspect the schema's structure by printing it:

```scala
dynamic
// res1: DynamicSchema = DynamicSchema(
//   Record(
//     fields = IndexedSeq(
//       Term(
//         name = "name",
//         value = Primitive(
//           primitiveType = String(None),
//           typeId = String,
//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5ca5d848,
//           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//           modifiers = List(),
//           storedDefaultValue = None,
//           storedExamples = List()
//         ),
//         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//         modifiers = List()
//       ),
//       Term(
//         name = "age",
//         value = Primitive(
//           primitiveType = Int(None),
//           typeId = Int,
//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5ca5d848,
//           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//           modifiers = List(),
//           storedDefaultValue = None,
//           storedExamples = List()
//         ),
//         doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//         modifiers = List()
//       ),
//       Term(
//         name = "address",
//         value = Record(
//           fields = IndexedSeq(
//             Term(
//               name = "street",
//               value = Primitive(
//                 primitiveType = String(None),
//                 typeId = String,
//                 primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5ca5d848,
//                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
//                 modifiers = List(),
//                 storedDefaultValue = None,
//                 storedExamples = List()
//               ),
// ...
```

### `DynamicSchema.fromDynamicValue`

`DynamicSchema.fromDynamicValue` reconstructs a `DynamicSchema` from a previously serialized `DynamicValue` blob. This is the consumer-side entry point when schemas are loaded from a registry or database:

```scala
object DynamicSchema {
  def fromDynamicValue(dv: DynamicValue): DynamicSchema
}
```

Example showing the full store/retrieve round-trip:

```scala

case class Product(id: Long, name: String, price: Double)
object Product { implicit val schema: Schema[Product] = Schema.derived[Product] }

val original: DynamicSchema = Schema[Product].toDynamicSchema
val blob:     DynamicValue  = DynamicSchema.toDynamicValue(original)
val restored: DynamicSchema = DynamicSchema.fromDynamicValue(blob)
```

We inspect the restored schema's type name to confirm the round-trip preserved it:

```scala
restored.typeId.name
// res3: String = "Product"
```

## Serializing a DynamicSchema

`DynamicSchema.toDynamicValue` converts a `DynamicSchema` to a `DynamicValue` for storage or transmission. The result contains only field names, type names, validations, and annotations — no Scala closures:

```scala
object DynamicSchema {
  def toDynamicValue(ds: DynamicSchema): DynamicValue
}
```

The following example converts a simple case class schema and stores the blob:

```scala

case class Tag(name: String)
object Tag { implicit val schema: Schema[Tag] = Schema.derived[Tag] }

val ds:   DynamicSchema = Schema[Tag].toDynamicSchema
val blob: DynamicValue  = DynamicSchema.toDynamicValue(ds)
```

We inspect the blob's value type to confirm the record structure was preserved:

```scala
blob.valueType
// res5: DynamicValueType = <function1>
```

## Validating DynamicValues

`DynamicSchema` can check whether a `DynamicValue` conforms to its structure. Validation is recursive: field counts and names, variant case names, collection element types, and primitive type + validation constraints are all checked.

### `DynamicSchema#check`

`DynamicSchema#check` returns `None` if the value is valid, or `Some(SchemaError)` describing the first validation failure:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def check(value: DynamicValue): Option[SchemaError]
}
```

The following example shows all three outcomes — a valid value, a missing field, and a type mismatch:

```scala

case class Point(x: Int, y: Int)
object Point { implicit val schema: Schema[Point] = Schema.derived[Point] }

val dynSchema = Schema[Point].toDynamicSchema

val valid = DynamicValue.Record(Chunk(
  "x" -> DynamicValue.int(3),
  "y" -> DynamicValue.int(7)
))

val missing = DynamicValue.Record(Chunk(
  "x" -> DynamicValue.int(3)
  // "y" missing
))

val wrongType = DynamicValue.Record(Chunk(
  "x" -> DynamicValue.int(3),
  "y" -> DynamicValue.string("not an int")
))
```

We evaluate all three cases to observe the outcomes:

```scala
dynSchema.check(valid)
// res7: Option[SchemaError] = None
dynSchema.check(missing)
// res8: Option[SchemaError] = Some(
//   SchemaError(
//     List(MissingField(source = DynamicOptic(IndexedSeq()), fieldName = "y"))
//   )
// )
dynSchema.check(wrongType)
// res9: Option[SchemaError] = Some(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(IndexedSeq(Field("y"))),
//         expectation = "Expected Int, got String"
//       )
//     )
//   )
// )
```

Validation rules:

- **Records**: every field in the schema must be present; extra fields in the value are rejected.
- **Variants**: the case name must be valid; the case payload is validated recursively.
- **Sequences**: every element is validated against the element schema.
- **Maps**: every key and every value is validated against their respective schemas.
- **Primitives**: the `PrimitiveType` must match, and any `Validation` constraints (range, pattern, etc.) must pass.

### `DynamicSchema#conforms`

`DynamicSchema#conforms` is a convenience method that returns `true` when the value is valid:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def conforms(value: DynamicValue): Boolean
}
```

We pass a well-formed record to confirm the return value:

```scala

case class Tag(name: String)
object Tag { implicit val schema: Schema[Tag] = Schema.derived[Tag] }

val dynSchema = Schema[Tag].toDynamicSchema
val value     = DynamicValue.Record(Chunk("name" -> DynamicValue.string("scala")))
```

```scala
dynSchema.conforms(value)
// res11: Boolean = true
```

## Rebinding to a Typed Schema

`DynamicSchema#rebind` converts a structural `DynamicSchema` back into a fully operational `Schema[A]` by walking the `Reflect.Unbound` tree and attaching runtime bindings from a `BindingResolver`:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def rebind[A](resolver: BindingResolver): Schema[A]
}
```

The resolver must provide a binding for every concrete type referenced in the schema tree: record types, variant types, and wrapper types must be covered explicitly; primitives, sequences, and maps are covered by `BindingResolver.defaults`:

```scala

case class OrderId(value: String)
case class LineItem(sku: String, quantity: Int)
case class Order(id: OrderId, items: List[LineItem])

object OrderId  { implicit val schema: Schema[OrderId]  = Schema.derived[OrderId]  }
object LineItem { implicit val schema: Schema[LineItem] = Schema.derived[LineItem] }
object Order    { implicit val schema: Schema[Order]    = Schema.derived[Order]    }

val blob: DynamicValue     = DynamicSchema.toDynamicValue(Schema[Order].toDynamicSchema)
val dynamic: DynamicSchema = DynamicSchema.fromDynamicValue(blob)

val resolver: BindingResolver =
  BindingResolver.empty
    .bind(Binding.of[OrderId])
    .bind(Binding.of[LineItem])
    .bind(Binding.of[Order])
    ++ BindingResolver.defaults

val rebound: Schema[Order] = dynamic.rebind[Order](resolver)
val order                  = Order(OrderId("ORD-1"), List(LineItem("SKU-A", 2)))
val encoded                = rebound.toDynamicValue(order)
```

We decode the encoded value to confirm the round-trip is lossless:

```scala
rebound.fromDynamicValue(encoded)
// res13: Either[SchemaError, Order] = Right(
//   Order(
//     id = OrderId("ORD-1"),
//     items = List(LineItem(sku = "SKU-A", quantity = 2))
//   )
// )
```

:::warning
If `DynamicSchema#rebind` cannot find a binding for any type in the unbound schema tree, it throws a `RebindException` at runtime. Ensure the resolver covers every concrete type — records, variants, wrappers, primitives, and standard collections — that appears in the schema. `BindingResolver.defaults` covers all standard primitives, `java.time` types, `List`, `Map`, `Option`, and other standard collections.
:::

See [BindingResolver](./binding-resolver.md) for the full resolver API including `BindingResolver.reflection` for automatic binding via reflection.

## Structural Navigation

`DynamicSchema#get` navigates into the schema tree using a `DynamicOptic` path, returning the nested `Reflect.Unbound[_]` at that location:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def get(optic: DynamicOptic): Option[Reflect.Unbound[_]]
}
```

The following example navigates to a field nested two levels deep:

```scala

case class Address(street: String, city: String)
case class Person(name: String, age: Int, address: Address)

object Address { implicit val schema: Schema[Address] = Schema.derived[Address] }
object Person  { implicit val schema: Schema[Person]  = Schema.derived[Person]  }

val dynSchema = Schema[Person].toDynamicSchema
val streetSchema: Option[Reflect.Unbound[_]] =
  dynSchema.get(DynamicOptic.root.field("address").field("street"))
```

We map over the result to read the type name at that path:

```scala
streetSchema.map(_.typeId.name)
// res15: Option[String] = Some("String")
```

See [DynamicOptic](./dynamic-optic.md) for the full path DSL.

## Metadata Access and Updates

`DynamicSchema` provides read and write access to the metadata stored in the `Reflect` tree.

### `DynamicSchema#typeId`

Returns the `TypeId` of the root type:

```scala

case class Event(id: Long, kind: String)
object Event { implicit val schema: Schema[Event] = Schema.derived[Event] }

val dynSchema = Schema[Event].toDynamicSchema
```

```scala
dynSchema.typeId.name
// res17: String = "Event"
dynSchema.typeId.fullName
// res18: String = "repl.MdocSession.MdocApp16.Event"
```

### `DynamicSchema#doc`

Reads and writes the documentation annotation on the schema. The zero-argument form reads the current `Doc`; the single-argument form returns a copy with updated documentation:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def doc: Doc
  def doc(value: String): DynamicSchema
}
```

We attach a description and read it back to confirm it was applied:

```scala

case class Event(id: Long, kind: String)
object Event { implicit val schema: Schema[Event] = Schema.derived[Event] }

val updated: DynamicSchema = Schema[Event].toDynamicSchema.doc("An event in the event log")
```

```scala
updated.doc
// res20: Doc = Doc(
//   blocks = IndexedSeq(Paragraph(IndexedSeq(Text("An event in the event log")))),
//   metadata = Map()
// )
```

### `DynamicSchema#modifiers` and `DynamicSchema#modifier`

`DynamicSchema#modifiers` returns the `Modifier.Reflect` annotations attached to the root node. `DynamicSchema#modifier` returns a copy with an additional modifier appended:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def modifiers: Seq[Modifier.Reflect]
  def modifier(m: Modifier.Reflect): DynamicSchema
}
```

### `DynamicSchema#getDefaultValue` and `DynamicSchema#defaultValue`

`DynamicSchema#getDefaultValue` returns the stored default `DynamicValue`, if one is set. `DynamicSchema#defaultValue` returns a copy with the given default:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def getDefaultValue: Option[DynamicValue]
  def defaultValue(value: DynamicValue): DynamicSchema
}
```

We attach a default value and then retrieve it:

```scala

case class Config(retries: Int)
object Config { implicit val schema: Schema[Config] = Schema.derived[Config] }

val withDefault: DynamicSchema =
  Schema[Config].toDynamicSchema
    .defaultValue(DynamicValue.Record(
      zio.blocks.chunk.Chunk("retries" -> DynamicValue.int(3))
    ))
```

```scala
withDefault.getDefaultValue
// res22: Option[DynamicValue] = Some(
//   Record(IndexedSeq(("retries", Primitive(Int(3)))))
// )
```

### `DynamicSchema#examples`

The zero-argument form returns stored examples as a `Seq[DynamicValue]`. The multi-argument form returns a copy with the given examples set:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def examples: Seq[DynamicValue]
  def examples(value: DynamicValue, values: DynamicValue*): DynamicSchema
}
```

## Converting to a Typed Schema

`DynamicSchema#toSchema` returns a `Schema[DynamicValue]` — it stays fully in the dynamic world and requires no bindings. Use it when you have received a `DynamicSchema` over the wire and need a codec-compatible schema that enforces structural conformance without binding any Scala types.

After transporting a `DynamicSchema`, you may not have the Scala types that `DynamicSchema#rebind` requires. For example, when the consumer is a validation gateway or format converter that handles arbitrary event shapes, any codec pipeline that accepts `Schema[DynamicValue]` can use the result directly. `DynamicSchema#toSchema` is the right choice for schema validation middleware, event-store gateways, and format converters that must enforce structure without knowing the concrete type.

:::warning
Use `DynamicSchema#rebind` instead when you have a `BindingResolver` and need a fully operational `Schema[A]` for typed encoding and decoding.
:::
`DynamicSchema#toSchema` is defined as:

```scala
final case class DynamicSchema(reflect: Reflect.Unbound[_]) {
  def toSchema: Schema[DynamicValue]
}
```

The following example shows a schema-validation gateway: we receive a `DynamicSchema` from a registry, convert it to a `Schema[DynamicValue]`, and use the result to validate an incoming payload, rejecting it on structural mismatch:

```scala

case class OrderEvent(orderId: String, amount: Double)
object OrderEvent { implicit val schema: Schema[OrderEvent] = Schema.derived[OrderEvent] }

val blob: DynamicValue          = DynamicSchema.toDynamicValue(Schema[OrderEvent].toDynamicSchema)
val received: DynamicSchema     = DynamicSchema.fromDynamicValue(blob)
val gatewaySchema: Schema[DynamicValue] = received.toSchema

val incoming: DynamicValue = DynamicValue.Record(
  zio.blocks.chunk.Chunk(
    "orderId" -> DynamicValue.string("ORD-42"),
    "amount"  -> DynamicValue.double(99.95)
  )
)
```

We validate the payload and observe the result:

```scala
gatewaySchema.fromDynamicValue(incoming)
// res24: Either[SchemaError, DynamicValue] = Right(
//   Record(
//     IndexedSeq(
//       ("orderId", Primitive(String("ORD-42"))),
//       ("amount", Primitive(Double(99.95)))
//     )
//   )
// )
```

## Integration

`DynamicSchema` connects to several other ZIO Blocks types, each serving a distinct role in the dynamic layer.

### With `DynamicValue`

`DynamicSchema` and `DynamicValue` are the two halves of ZIO Blocks' dynamic layer. `DynamicValue` holds runtime data without compile-time types; `DynamicSchema` holds structural metadata without runtime bindings. Together they enable fully type-erased validation and serialization pipelines. See [DynamicValue](./dynamic-value.md).

### With `BindingResolver`

The primary consumer of `DynamicSchema` in a type-safe context is `DynamicSchema#rebind`, which requires a `BindingResolver` to reattach runtime bindings. See [BindingResolver](./binding-resolver.md) for the complete rebinding API, including `BindingResolver.reflection` for automatic binding discovery.

### With `DynamicOptic`

`DynamicSchema#get` accepts a `DynamicOptic` path to navigate the structural tree. This allows you to inspect nested schemas, extract type information, or validate sub-trees independently. See [DynamicOptic](./dynamic-optic.md).

### With `Schema`

`Schema[A]#toDynamicSchema` is defined on `Schema` and is the standard way to obtain a `DynamicSchema`. See [Schema](./schema.md).

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

**Validate DynamicValues against a DynamicSchema**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaValidationExample.scala))

```bash
sbt "schema-examples/runMain dynamicschema.DynamicSchemaValidationExample"
```

**Serialize and deserialize a DynamicSchema**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaSerializationExample.scala))

```bash
sbt "schema-examples/runMain dynamicschema.DynamicSchemaSerializationExample"
```

**Rebind a restored DynamicSchema to a typed Schema**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaRebindExample.scala))

```bash
sbt "schema-examples/runMain dynamicschema.DynamicSchemaRebindExample"
```

**Complete schema registry pipeline**
([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/dynamicschema/DynamicSchemaRegistryExample.scala))

```bash
sbt "schema-examples/runMain dynamicschema.DynamicSchemaRegistryExample"
```

---

## DynamicValue

`DynamicValue` is a schema-less, dynamically-typed representation of any structured value in ZIO Blocks. It provides a universal data model that can represent any value without requiring compile-time type information, serving as an intermediate representation for serialization, schema evolution, data transformation, and cross-format conversion.

## Overview

The `DynamicValue` type represents all structured values with six cases:

```
DynamicValue
 ├── DynamicValue.Primitive   (scalar values: strings, numbers, booleans, temporal types, etc.)
 ├── DynamicValue.Record      (named fields, analogous to case classes or JSON objects)
 ├── DynamicValue.Variant     (tagged unions, analogous to sealed traits)
 ├── DynamicValue.Sequence    (ordered collections: lists, arrays, vectors)
 ├── DynamicValue.Map         (key-value pairs where keys are also DynamicValues)
 └── DynamicValue.Null        (absence of a value)
```

Key design decisions:

- **Type-agnostic** — Works without compile-time type information
- **Preserves structure** — Maintains full fidelity of the original data
- **Supports rich primitives** — All Java time types, BigDecimal, UUID, Currency, etc.
- **Path-based navigation** — Uses `DynamicOptic` for traversal and modification
- **EJSON toString** — Human-readable output format with type annotations

## DynamicValue Variants

### Primitive

Wraps scalar values in a `PrimitiveValue`:

```scala

// Using convenience constructors
val str = DynamicValue.string("hello")
val num = DynamicValue.int(42)
val flag = DynamicValue.boolean(true)
val pi = DynamicValue.double(3.14159)

// Using the Primitive case directly

val instant = DynamicValue.Primitive(
  PrimitiveValue.Instant(java.time.Instant.now())
)
```

### Record

A collection of named fields, analogous to case classes or JSON objects:

```scala

// Using varargs constructor
val person = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30),
  "active" -> DynamicValue.boolean(true)
)

// Using Chunk constructor
val point = DynamicValue.Record(Chunk(
  ("x", DynamicValue.int(10)),
  ("y", DynamicValue.int(20))
))

// Empty record
val empty = DynamicValue.Record.empty
```

Field order is preserved and significant for equality. Use `sortFields` to normalize for order-independent comparison.

### Variant

A tagged union value, analogous to sealed traits:

```scala

// A Some variant containing a value
val some = DynamicValue.Variant(
  "Some",
  DynamicValue.string("hello")
)

// A None variant with an empty record
val none = DynamicValue.Variant("None", DynamicValue.Record.empty)

// Access case information
some.caseName   // Some("Some")
some.caseValue  // Some(DynamicValue.Primitive(...))
```

### Sequence

An ordered collection of values:

```scala

// Using varargs constructor
val numbers = DynamicValue.Sequence(
  DynamicValue.int(1),
  DynamicValue.int(2),
  DynamicValue.int(3)
)

// Using Chunk constructor
val items = DynamicValue.Sequence(Chunk(
  DynamicValue.string("a"),
  DynamicValue.string("b")
))

// Empty sequence
val empty = DynamicValue.Sequence.empty
```

### Map

Key-value pairs where both keys and values are `DynamicValue`:

```scala

// String keys (common case)
val config = DynamicValue.Map(
  DynamicValue.string("host") -> DynamicValue.string("localhost"),
  DynamicValue.string("port") -> DynamicValue.int(8080)
)

// Non-string keys (unlike Record)
val mapping = DynamicValue.Map(
  DynamicValue.int(1) -> DynamicValue.string("one"),
  DynamicValue.int(2) -> DynamicValue.string("two")
)

// Empty map
val empty = DynamicValue.Map.empty
```

Unlike `Record` which uses String keys, `Map` supports arbitrary `DynamicValue` keys.

### Null

Represents the absence of a value:

```scala

val absent = DynamicValue.Null
```

## PrimitiveValue Types

`PrimitiveValue` is a sealed trait representing all scalar values that can be wrapped in `DynamicValue.Primitive`. Each case preserves full type information:

| Type | Description | Example |
|------|-------------|---------|
| `Unit` | Unit value | `PrimitiveValue.Unit` |
| `Boolean` | Boolean | `PrimitiveValue.Boolean(true)` |
| `Byte` | 8-bit integer | `PrimitiveValue.Byte(127)` |
| `Short` | 16-bit integer | `PrimitiveValue.Short(32767)` |
| `Int` | 32-bit integer | `PrimitiveValue.Int(42)` |
| `Long` | 64-bit integer | `PrimitiveValue.Long(9999999999L)` |
| `Float` | 32-bit float | `PrimitiveValue.Float(3.14f)` |
| `Double` | 64-bit float | `PrimitiveValue.Double(3.14159)` |
| `Char` | Unicode character | `PrimitiveValue.Char('A')` |
| `String` | Text | `PrimitiveValue.String("hello")` |
| `BigInt` | Arbitrary precision integer | `PrimitiveValue.BigInt(BigInt("999..."))` |
| `BigDecimal` | Arbitrary precision decimal | `PrimitiveValue.BigDecimal(BigDecimal("3.14159"))` |
| `Instant` | Timestamp | `PrimitiveValue.Instant(Instant.now())` |
| `LocalDate` | Date without time | `PrimitiveValue.LocalDate(LocalDate.now())` |
| `LocalDateTime` | Date and time | `PrimitiveValue.LocalDateTime(LocalDateTime.now())` |
| `LocalTime` | Time without date | `PrimitiveValue.LocalTime(LocalTime.now())` |
| `Duration` | Time duration | `PrimitiveValue.Duration(Duration.ofHours(1))` |
| `Period` | Date-based period | `PrimitiveValue.Period(Period.ofDays(30))` |
| `DayOfWeek` | Day of week | `PrimitiveValue.DayOfWeek(DayOfWeek.MONDAY)` |
| `Month` | Month | `PrimitiveValue.Month(Month.JANUARY)` |
| `Year` | Year | `PrimitiveValue.Year(Year.of(2024))` |
| `YearMonth` | Year and month | `PrimitiveValue.YearMonth(YearMonth.of(2024, 1))` |
| `MonthDay` | Month and day | `PrimitiveValue.MonthDay(MonthDay.of(1, 15))` |
| `ZoneId` | Time zone | `PrimitiveValue.ZoneId(ZoneId.of("UTC"))` |
| `ZoneOffset` | Time zone offset | `PrimitiveValue.ZoneOffset(ZoneOffset.UTC)` |
| `ZonedDateTime` | Date/time with zone | `PrimitiveValue.ZonedDateTime(ZonedDateTime.now())` |
| `OffsetDateTime` | Date/time with offset | `PrimitiveValue.OffsetDateTime(OffsetDateTime.now())` |
| `OffsetTime` | Time with offset | `PrimitiveValue.OffsetTime(OffsetTime.now())` |
| `UUID` | Universally unique ID | `PrimitiveValue.UUID(UUID.randomUUID())` |
| `Currency` | Currency | `PrimitiveValue.Currency(Currency.getInstance("USD"))` |

## Creating DynamicValues from Typed Values

Use `Schema.toDynamicValue` to convert typed Scala values to `DynamicValue`:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)
val dynamic: DynamicValue = Schema[Person].toDynamicValue(person)
// Record with "name" and "age" fields

// Works with any type that has a Schema
val listDynamic = Schema[List[Int]].toDynamicValue(List(1, 2, 3))
// Sequence of Primitive(Int) values
```

## Converting DynamicValues Back to Typed Values

Use `Schema.fromDynamicValue` to convert `DynamicValue` back to typed Scala values:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val dynamic = DynamicValue.Record(
  "name" -> DynamicValue.string("Bob"),
  "age" -> DynamicValue.int(25)
)

val result: Either[SchemaError, Person] = Schema[Person].fromDynamicValue(dynamic)
// Right(Person("Bob", 25))

// Type mismatch produces an error
val badDynamic = DynamicValue.string("not a person")
val error = Schema[Person].fromDynamicValue(badDynamic)
// Left(SchemaError(...))
```

## Type Information

### DynamicValueType

Each `DynamicValue` has a corresponding `DynamicValueType` for runtime type checking:

```scala

val dv = DynamicValue.Record("x" -> DynamicValue.int(1))

// Check type
dv.is(DynamicValueType.Record)    // true
dv.is(DynamicValueType.Sequence)  // false

// Narrow to specific type
val record: Option[DynamicValue.Record] = dv.as(DynamicValueType.Record)
// Some(Record(...))

// Extract underlying value

val fields: Option[Chunk[(String, DynamicValue)]] = 
  dv.unwrap(DynamicValueType.Record)
```

### Extracting Primitive Values

```scala

val dv = DynamicValue.int(42)

// Extract with specific primitive type
val intValue: Option[Int] = dv.asPrimitive(PrimitiveType.Int(Validation.None))
// Some(42)

val stringValue: Option[String] = dv.asPrimitive(PrimitiveType.String(Validation.None))
// None (type mismatch)
```

## Navigation

### Simple Navigation

Navigate using `get` methods that return `DynamicValueSelection`:

```scala

val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice")),
    DynamicValue.Record("name" -> DynamicValue.string("Bob"))
  )
)

// Navigate to a field
val users = data.get("users")  // DynamicValueSelection

// Navigate to an array element
val firstUser = data.get("users").apply(0)

// Chain navigation
val firstName = data.get("users").apply(0).get("name")

// Extract the value
val name = firstName.one  // Either[SchemaError, DynamicValue]
```

### Path-Based Navigation with DynamicOptic

Use `DynamicOptic` for complex path expressions:

```scala

val data = DynamicValue.Record(
  "company" -> DynamicValue.Record(
    "employees" -> DynamicValue.Sequence(
      DynamicValue.Record("name" -> DynamicValue.string("Alice"))
    )
  )
)

// Build a path
val path = DynamicOptic.root.field("company").field("employees").at(0).field("name")

// Navigate using the path
val result = data.get(path).one  // Right(DynamicValue.Primitive(String("Alice")))
```

### DynamicValueSelection

`DynamicValueSelection` wraps navigation results and provides fluent chaining:

```scala

val selection: DynamicValueSelection = ???

// Terminal operations
selection.one      // Either[SchemaError, DynamicValue] - exactly one value
selection.any      // Either[SchemaError, DynamicValue] - first of many
selection.all      // Either[SchemaError, DynamicValue] - wrap multiple in Sequence
selection.toChunk  // Chunk[DynamicValue] - empty on error

// Type filtering
selection.primitives  // Only Primitive values
selection.records     // Only Record values
selection.sequences   // Only Sequence values
selection.maps        // Only Map values

// Combinators
selection.map(dv => ???)      // Transform values
selection.filter(dv => ???)   // Filter values
selection.flatMap(dv => ???)  // Chain selections
```

## Path-Based Modification

### Modify

Update values at a path:

```scala

val data = DynamicValue.Record(
  "user" -> DynamicValue.Record(
    "name" -> DynamicValue.string("Alice")
  )
)

val path = DynamicOptic.root.field("user").field("name")

// Modify value at path
val updated = data.modify(path)(dv => DynamicValue.string("Bob"))
// Record("user" -> Record("name" -> "Bob"))
```

### Set

Replace a value at a path:

```scala

val data = DynamicValue.Record("x" -> DynamicValue.int(1))
val path = DynamicOptic.root.field("x")

val updated = data.set(path, DynamicValue.int(99))
// Record("x" -> 99)
```

### Delete

Remove a value at a path:

```scala

val data = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.int(2)
)

val updated = data.delete(DynamicOptic.root.field("a"))
// Record("b" -> 2)
```

### Insert

Add a value at a path (fails if path exists):

```scala

val data = DynamicValue.Record("a" -> DynamicValue.int(1))

val updated = data.insert(
  DynamicOptic.root.field("b"),
  DynamicValue.int(2)
)
// Record("a" -> 1, "b" -> 2)
```

### Fallible Operations

Use `*OrFail` variants for operations that should fail explicitly:

```scala

val data = DynamicValue.Record("x" -> DynamicValue.int(1))
val badPath = DynamicOptic.root.field("nonexistent")

val result: Either[SchemaError, DynamicValue] = 
  data.setOrFail(badPath, DynamicValue.int(99))
// Left(SchemaError("Path not found"))
```

## EJSON-like toString Format

`DynamicValue.toString` produces an EJSON (Extended JSON) format that:

- Uses unquoted field names for Records (like Scala syntax)
- Uses quoted string keys for Maps
- Adds `@ {tag: "..."}` annotations for Variants
- Adds `@ {type: "..."}` annotations for typed primitives (Instant, Duration, etc.)

```scala

val person = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30)
)

println(person.toString)
// {
//   name: "Alice",
//   age: 30
// }

val variant = DynamicValue.Variant(
  "Some",
  DynamicValue.string("hello")
)

println(variant.toString)
// "hello" @ {tag: "Some"}

val timestamp = DynamicValue.Primitive(
  PrimitiveValue.Instant(java.time.Instant.ofEpochMilli(1700000000000L))
)

println(timestamp.toString)
// 1700000000000 @ {type: "instant"}
```

Use `toEjson(indent)` to control indentation level.

## Merging Strategies

Merge two `DynamicValue` structures using configurable strategies:

```scala

val left = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.int(2)
)

val right = DynamicValue.Record(
  "b" -> DynamicValue.int(99),
  "c" -> DynamicValue.int(3)
)

// Deep merge (default): recursively merge containers
val merged = left.merge(right, DynamicValueMergeStrategy.Auto)
// Record("a" -> 1, "b" -> 99, "c" -> 3)
```

### Available Strategies

| Strategy | Behavior |
|----------|----------|
| `Auto` | Deep merge: Records by field, Sequences by index, Maps by key. Right wins at leaves. |
| `Replace` | Complete replacement: right value replaces left entirely |
| `KeepLeft` | Always keep left value |
| `Shallow` | Merge only at root level, nested containers replaced |
| `Concat` | Concatenate Sequences instead of merging by index |
| `Custom(f, r)` | Custom function with custom recursion control |

```scala

val list1 = DynamicValue.Sequence(DynamicValue.int(1), DynamicValue.int(2))
val list2 = DynamicValue.Sequence(DynamicValue.int(3))

// Concat sequences instead of index-based merge
val concatted = list1.merge(list2, DynamicValueMergeStrategy.Concat)
// Sequence(1, 2, 3)
```

## Normalization

Transform `DynamicValue` structures for comparison or serialization:

```scala

val data = DynamicValue.Record(
  "z" -> DynamicValue.int(1),
  "a" -> DynamicValue.Null,
  "m" -> DynamicValue.int(2)
)

// Sort fields alphabetically
data.sortFields
// Record("a" -> null, "m" -> 2, "z" -> 1)

// Remove null values
data.dropNulls
// Record("z" -> 1, "m" -> 2)

// Remove empty containers
data.dropEmpty

// Remove Unit primitives
data.dropUnits

// Apply all normalizations
data.normalize
// Sorted, no nulls, no units, no empty containers
```

## Transformation

### Transform Up/Down

Apply functions to all values in a structure:

```scala

val data = DynamicValue.Record(
  "values" -> DynamicValue.Sequence(
    DynamicValue.int(1),
    DynamicValue.int(2)
  )
)

// Bottom-up: children transformed before parents
val doubled = data.transformUp { (path, dv) =>
  dv match {
    case DynamicValue.Primitive(pv: PrimitiveValue.Int) =>
      DynamicValue.int(pv.value * 2)
    case other => other
  }
}

// Top-down: parents transformed before children
val topDown = data.transformDown { (path, dv) => ??? }
```

### Transform Field Names

Rename all record fields:

```scala

val data = DynamicValue.Record(
  "first_name" -> DynamicValue.string("Alice"),
  "last_name" -> DynamicValue.string("Smith")
)

// Convert snake_case to camelCase
val camelCase = data.transformFields { (path, name) =>
  name.split("_").zipWithIndex.map {
    case (word, 0) => word
    case (word, _) => word.capitalize
  }.mkString
}
// Record("firstName" -> "Alice", "lastName" -> "Smith")
```

## Folding

Aggregate values from a `DynamicValue` tree:

```scala

val data = DynamicValue.Record(
  "a" -> DynamicValue.int(1),
  "b" -> DynamicValue.int(2),
  "c" -> DynamicValue.int(3)
)

// Sum all integers
val sum = data.foldUp(0) { (path, dv, acc) =>
  dv match {
    case DynamicValue.Primitive(pv: PrimitiveValue.Int) => acc + pv.value
    case _ => acc
  }
}
// 6
```

## Converting to/from JSON

### To JSON

```scala

val dynamic = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30)
)

val json: Json = dynamic.toJson
// Json.Object with "name" and "age" fields
```

### From JSON

```scala

val json = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")

val dynamic: DynamicValue = json.toDynamicValue
// DynamicValue.Record with "name" and "age" fields
```

## Querying

Search recursively for values matching a predicate:

```scala

val data = DynamicValue.Record(
  "users" -> DynamicValue.Sequence(
    DynamicValue.Record("name" -> DynamicValue.string("Alice"), "active" -> DynamicValue.boolean(true)),
    DynamicValue.Record("name" -> DynamicValue.string("Bob"), "active" -> DynamicValue.boolean(false))
  )
)

// Find all string values
val strings = data.select.query(_.is(DynamicValueType.Primitive))
  .filter(_.primitiveValue.exists(_.isInstanceOf[PrimitiveValue.String]))

// Query with path predicate
val atDepth2 = data.select.queryPath(path => path.nodes.length == 2)
```

## Use Cases

### Schema-less Operations

Work with data when the schema isn't known at compile time:

```scala

def processAnyData(data: DynamicValue): DynamicValue = {
  // Add a timestamp to any record
  data match {
    case r: DynamicValue.Record =>
      DynamicValue.Record(
        r.fields :+ ("processedAt" -> DynamicValue.Primitive(
          PrimitiveValue.Instant(java.time.Instant.now())
        ))
      )
    case other => other
  }
}
```

### Schema Migrations

Transform data between schema versions:

```scala

def migrateV1toV2(data: DynamicValue): DynamicValue = {
  data.transformFields { (path, name) =>
    // Rename deprecated field
    if (name == "userName") "name"
    else name
  }.transformUp { (path, dv) =>
    // Add default for new required field
    dv match {
      case r: DynamicValue.Record if path.nodes.isEmpty =>
        DynamicValue.Record(r.fields :+ ("version" -> DynamicValue.int(2)))
      case other => other
    }
  }
}
```

### Dynamic Queries

Build queries at runtime:

```scala

def buildPath(fields: List[String]): DynamicOptic =
  fields.foldLeft(DynamicOptic.root)(_.field(_))

def getValue(data: DynamicValue, path: List[String]): Option[DynamicValue] =
  data.get(buildPath(path)).one.toOption

// Usage
val data = DynamicValue.Record(
  "user" -> DynamicValue.Record(
    "profile" -> DynamicValue.Record(
      "email" -> DynamicValue.string("alice@example.com")
    )
  )
)

val email = getValue(data, List("user", "profile", "email"))
// Some(DynamicValue.Primitive(String("alice@example.com")))
```

### Cross-Format Conversion

Use `DynamicValue` as an intermediate format:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// JSON -> DynamicValue -> Typed
val json = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")
val dynamic = json.toDynamicValue
val person = Schema[Person].fromDynamicValue(dynamic)

// Typed -> DynamicValue -> JSON
val dynamic2 = Schema[Person].toDynamicValue(Person("Bob", 25))
val json2 = dynamic2.toJson
```

## Comparison and Ordering

`DynamicValue` has a total ordering for sorting and comparison:

```scala

val a = DynamicValue.int(1)
val b = DynamicValue.int(2)

a.compare(b)  // negative
a < b         // true
a >= b        // false

// Type ordering: Primitive < Record < Variant < Sequence < Map < Null
val primitive = DynamicValue.int(1)
val record = DynamicValue.Record.empty
primitive < record  // true
```

## Diff and Patch

Compute differences between `DynamicValue` instances:

```scala

val old = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(30)
)

val new_ = DynamicValue.Record(
  "name" -> DynamicValue.string("Alice"),
  "age" -> DynamicValue.int(31)
)

val patch: DynamicPatch = old.diff(new_)
// Patch that updates "age" from 30 to 31
```

## See Also

- [DynamicSchema](./dynamic-schema.md) — validate `DynamicValue` instances against a structural schema, or transport schemas as `DynamicValue` blobs.

---

## Serialization Formats

ZIO Blocks Schema provides automatic codec derivation for multiple serialization formats. Once you have a `Schema[A]` for your data type, you can derive codecs for any supported format using the unified `Schema.derive(Format)` pattern.

A `Format` is an abstraction that bundles together everything needed to serialize and deserialize data in a specific format (JSON, Avro, Protobuf, etc.).

## Overview

Each format defines the types of input for decoding and output for encoding, as well as the typeclass used as a codec for that format. Each format contains a `Deriver` corresponding to its specific MIME type, which is used to derive codecs from schemas:

```scala
trait Format {
  type DecodeInput
  type EncodeOutput
  type TypeClass[A] <: Codec[DecodeInput, EncodeOutput, A]
  def mimeType: String
  def deriver: Deriver[TypeClass]
}
```

It unifies all metadata related to serialization formats, such as MIME type and codec deriver, in a single place. This allows for a consistent API across different formats when deriving codecs from schemas. Having MIME type information helps with runtime content negotiation and format routing, for example in HTTP servers or message queues.

That is, you can easily call [`Schema[A].derive(format)`](./type-class-derivation.md#using-the-deriver-to-derive-type-class-instances) for any format that implements the `Format` trait, and receive a codec that can encode and decode values of type `A` according to the rules of that format.

Formats are categorized into `BinaryFormat` and `TextFormat`, which specify the types of input and output for encoding and decoding:

```scala
sealed trait Format
abstract class BinaryFormat[...](...) extends Format { ... }
abstract class TextFormat[...](...) extends Format { ... }
```

For example, the `JsonFormat` is a `BinaryFormat` that represents a JSON binary format, where the input for decoding is `ByteBuffer` and the output for encoding is also `ByteBuffer`, the MIME type is `application/json`, and the deriver for generating codecs from schemas is `JsonCodecDeriver`:

```scala
object JsonFormat extends BinaryFormat("application/json", JsonCodecDeriver)
```

## Built-in Formats

Here's a summary of the formats currently supported by ZIO Blocks. Each format provides a `BinaryFormat` object that can be passed to `derive`:

| Format Object       | Codec Type            | MIME Type             | Module                          |
|---------------------|-----------------------|-----------------------|---------------------------------|
| `JsonFormat`        | `JsonCodec[A]`        | `application/json`    | `zio-blocks-schema`             |
| `AvroFormat`        | `AvroCodec[A]`        | `application/avro`    | `zio-blocks-schema-avro`        |
| `BsonFormat`        | `BsonCodec[A]`        | `application/bson`    | `zio-blocks-schema-bson`        |
| `CsvFormat`         | `CsvCodec[A]`         | `text/csv`            | `zio-blocks-schema-csv`         |
| `MessagePackFormat` | `MessagePackCodec[A]` | `application/msgpack` | `zio-blocks-schema-messagepack` |
| `ThriftFormat`      | `ThriftCodec[A]`      | `application/thrift`  | `zio-blocks-schema-thrift`      |
| `ToonFormat`        | `ToonCodec[A]`        | `text/toon`           | `zio-blocks-schema-toon`        |
| `XmlFormat`         | `XmlCodec[A]`         | `application/xml`     | `zio-blocks-schema-xml`         |
| `YamlFormat`        | `YamlCodec[A]`        | `application/yaml`    | `zio-blocks-schema-yaml`        |

## Defining a Custom Format

To add a new serialization format, define a `BinaryFormat` (or `TextFormat`) singleton with a custom `Deriver`:

```scala

// 1. Define your codec base class
abstract class MyCodec[A] extends BinaryCodec[A]

// 2. Implement a Deriver[MyCodec] (see Type-class Derivation docs)
// val myDeriver: Deriver[MyCodec] = ...

// 3. Create the format singleton
// object MyFormat extends BinaryFormat[MyCodec]("application/x-myformat", myDeriver)
```

For details on implementing a `Deriver`, see [Type-class Derivation](./type-class-derivation.md).

## Codec Derivation System

All serialization formats in ZIO Blocks follow the same pattern: given a `Schema[A]`, you derive a codec by calling `derive` with a format object:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive codec for any format (using TOON as an example)
val codec = Schema[Person].derive(ToonFormat)

// Encode to bytes
val bytes: Array[Byte] = codec.encode(Person("Alice", 30))

// Decode from bytes
val result: Either[SchemaError, Person] = codec.decode(bytes)
```

## JSON Format

JSON format is the most commonly used text-based serialization format. See the dedicated [JSON documentation](json.md) for comprehensive coverage of the `Json` ADT, navigation, and transformation features.

### Installation

JSON support is included in the core schema module:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "<version>"
```

### Basic Usage

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)
val bytes: Array[Byte] = person.toJsonBytes
val decoded: Either[SchemaError, Person] = bytes.fromJson[Person]
```

## Avro Format

Apache Avro is a compact binary format with schema evolution support, commonly used in big data systems like Kafka and Spark.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-avro" % "<version>"
```

Requires the Apache Avro library (1.12.x).

### Basic Usage

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive Avro codec
val codec = Schema[Person].derive(AvroFormat)

// Encode to Avro binary format
val person = Person("Alice", 30)
val bytes: Array[Byte] = codec.encode(person)

// Decode from Avro binary format
val decoded: Either[SchemaError, Person] = codec.decode(bytes)
```

### Avro Schema Generation

Each `AvroCodec` exposes an `avroSchema` property containing the Apache Avro schema:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(AvroFormat)
val avroSchema: AvroSchema = codec.avroSchema
println(avroSchema.toString(true))
// {
//   "type": "record",
//   "name": "Person",
//   "fields": [
//     {"name": "name", "type": "string"},
//     {"name": "age", "type": "int"}
//   ]
// }
```

### Avro Type Mappings

| Scala Type | Avro Type |
|------------|-----------|
| `Boolean` | `boolean` |
| `Byte`, `Short`, `Int` | `int` |
| `Long` | `long` |
| `Float` | `float` |
| `Double` | `double` |
| `String`, `Char` | `string` |
| `BigInt` | `bytes` |
| `BigDecimal` | Record (mantissa, scale, precision, roundingMode) |
| `UUID` | 16-byte fixed |
| `Currency` | 3-byte fixed |
| `java.time.*` | Records or primitives |
| Case classes | `record` |
| Sealed traits | `union` |
| `List[A]`, `Set[A]` | `array` |
| `Map[String, V]` | `map` |

### ADT Encoding

Sealed traits are encoded as Avro unions with an integer index prefix:

```scala

sealed trait Shape
case class Circle(radius: Double) extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived
}

val codec = Schema[Shape].derive(AvroFormat)

// The variant index (0 for Circle, 1 for Rectangle) is written first,
// followed by the record data
val circle: Shape = Circle(5.0)
val bytes = codec.encode(circle)
```

## TOON Format (LLM-Optimized)

TOON (Token-Oriented Object Notation) is a line-oriented, indentation-based text format that encodes the JSON data model with explicit structure and minimal quoting. It is 30-60% more compact than JSON, making it particularly efficient for LLM prompts and responses.

### Why TOON?

- **Token efficient**: 30-60% fewer tokens than equivalent JSON
- **Human readable**: Clean, YAML-like syntax without YAML's complexity
- **LLM optimized**: Designed for AI/ML use cases where token count matters
- **Explicit lengths**: Arrays declare their size upfront for reliable parsing
- **Cross-platform**: Works on JVM and Scala.js

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-toon" % "<version>"
```

### Basic Usage

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive TOON codec
val codec = Schema[Person].derive(ToonFormat)

// Encode to TOON
val person = Person("Alice", 30)
val bytes: Array[Byte] = codec.encode(person)
// name: Alice
// age: 30

// Decode from TOON
val decoded: Either[SchemaError, Person] = codec.decode(bytes)
```

### TOON Format Examples

TOON uses indentation and explicit array lengths:

```
# Simple object
name: Alice
age: 30
email: alice@example.com

# Inline primitive arrays (comma-separated)
tags[3]: scala,zio,functional

# Nested object
address:
  street: 123 Main St
  city: Springfield

# Object arrays use list format
orders[2]:
  - id: 1
    total: 99.99
  - id: 2
    total: 149.5

# Or tabular format (more compact)
orders[2]{id,total}:
  1,99.99
  2,149.5
```

### Configuration Options

The `ToonCodecDeriver` provides extensive configuration:

```scala

case class Person(firstName: String, lastName: String)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Custom deriver with snake_case field names
val customDeriver = ToonCodecDeriver
  .withFieldNameMapper(NameMapper.SnakeCase)
  .withArrayFormat(ArrayFormat.Tabular)
  .withDiscriminatorKind(DiscriminatorKind.Field("type"))

val codec = Schema[Person].derive(customDeriver)
// first_name: Alice
// last_name: Smith
```

| Option | Description | Default |
|--------|-------------|---------|
| `withFieldNameMapper` | Transform field names (Identity, SnakeCase, KebabCase) | `Identity` |
| `withCaseNameMapper` | Transform variant/case names | `Identity` |
| `withDiscriminatorKind` | ADT discriminator style (Key, Field, None) | `Key` |
| `withArrayFormat` | Array encoding (Auto, Tabular, Inline, List) | `Auto` |
| `withDelimiter` | Inline array delimiter (Comma, Tab, Pipe) | `Comma` |
| `withRejectExtraFields` | Error on unknown fields during decoding | `false` |
| `withEnumValuesAsStrings` | Encode enum values as strings | `true` |
| `withTransientNone` | Omit None values from output | `true` |
| `withTransientEmptyCollection` | Omit empty collections | `true` |
| `withTransientDefaultValue` | Omit fields with default values | `true` |

### ADT Encoding Styles

```scala

sealed trait Shape
case class Circle(radius: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived
}

// Key discriminator (default)
val keyCodec = Schema[Shape].derive(ToonFormat)
// Circle:
//   radius: 5

// Field discriminator
val fieldDeriver = ToonCodecDeriver
  .withDiscriminatorKind(DiscriminatorKind.Field("type"))
val fieldCodec = Schema[Shape].derive(fieldDeriver)
// type: Circle
// radius: 5
```

## MessagePack Format

MessagePack is an efficient binary serialization format that is more compact than JSON while remaining schema-less and cross-language compatible.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-messagepack" % "<version>"
```

### Basic Usage

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive MessagePack codec
val codec = Schema[Person].derive(MessagePackFormat)

// Encode to MessagePack
val person = Person("Alice", 30)
val bytes: Array[Byte] = codec.encode(person)

// Decode from MessagePack
val decoded: Either[SchemaError, Person] = codec.decode(bytes)
```

### Binary Efficiency

MessagePack provides significant space savings compared to JSON:

- Typically 50-80% of JSON size
- Uses variable-width integer encoding
- No string escaping overhead
- No key quoting or colons/commas

### MessagePack Type Mappings

| Scala Type | MessagePack Type |
|------------|------------------|
| `Unit` | nil |
| `Boolean` | bool |
| `Byte`, `Short`, `Int`, `Long` | int (variable width) |
| `Float` | float32 |
| `Double` | float64 |
| `String`, `Char` | str |
| `Array[Byte]` | bin |
| `List[A]`, `Vector[A]`, `Set[A]` | array |
| `Map[K, V]` | map |
| `Option[A]` | array (0 or 1 element) |
| `Either[A, B]` | map with "left" or "right" key |
| Case classes | map with field names as keys |
| Sealed traits | int index followed by value |

### ADT Encoding

Sealed traits encode a variant index followed by the case value:

```scala

sealed trait Shape
case class Circle(radius: Double) extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived
}

val codec = Schema[Shape].derive(MessagePackFormat)

// Circle is encoded as: 0 followed by {radius: 5.0}
val circle: Shape = Circle(5.0)
val bytes = codec.encode(circle)
```

## BSON Format

BSON (Binary JSON) is the binary format used by MongoDB. The ZIO Blocks BSON module provides integration with the MongoDB BSON library.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-bson" % "<version>"
```

Requires the MongoDB BSON library (5.x).

### Basic Usage

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive BSON encoder/decoder
val encoder: BsonEncoder[Person] = BsonSchemaCodec.bsonEncoder(Schema[Person])
val decoder: BsonDecoder[Person] = BsonSchemaCodec.bsonDecoder(Schema[Person])

// Or get both as a codec
val codec: BsonCodec[Person] = BsonSchemaCodec.bsonCodec(Schema[Person])
```

### MongoDB ObjectId Support

BSON provides native support for MongoDB ObjectIds:

```scala

// Import ObjectId schema

case class Document(_id: ObjectId, title: String)

object Document {
  implicit val schema: Schema[Document] = Schema.derived
}

// ObjectId is encoded using BSON's native OBJECT_ID type
val codec = BsonSchemaCodec.bsonCodec(Schema[Document])
```

### Configuration Options

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Custom configuration
val config = Config
  .withSumTypeHandling(SumTypeHandling.DiscriminatorField("_type"))
  .withIgnoreExtraFields(true)
  .withNativeObjectId(true)

val codec = BsonSchemaCodec.bsonCodec(Schema[Person], config)
```

| Option | Description | Default |
|--------|-------------|---------|
| `withSumTypeHandling` | ADT discrimination strategy | `WrapperWithClassNameField` |
| `withClassNameMapping` | Transform class names | `identity` |
| `withIgnoreExtraFields` | Ignore unknown fields on decode | `true` |
| `withNativeObjectId` | Use native BSON ObjectId type | `false` |

### Sum Type Handling

```scala

// Option 1: Wrapper with class name as field key (default)
SumTypeHandling.WrapperWithClassNameField
// {"Circle": {"radius": 5.0}}

// Option 2: Discriminator field
SumTypeHandling.DiscriminatorField("_type")
// {"_type": "Circle", "radius": 5.0}

// Option 3: No discriminator (tries each case)
SumTypeHandling.NoDiscriminator
```

## Thrift Format

Apache Thrift is a binary protocol format with field ID-based encoding, supporting forward-compatible schema evolution.

### Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-thrift" % "<version>"
```

Requires the Apache Thrift library (0.22.x).

### Basic Usage

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive Thrift codec
val codec = Schema[Person].derive(ThriftFormat)

// Encode to Thrift binary format
val person = Person("Alice", 30)
val bytes: Array[Byte] = codec.encode(person)

// Decode from Thrift binary format
val decoded: Either[SchemaError, Person] = codec.decode(bytes)

// ByteBuffer API
val buffer = ByteBuffer.allocate(1024)
codec.encode(person, buffer)
buffer.flip()
val fromBuffer: Either[SchemaError, Person] = codec.decode(buffer)
```

### Thrift-Specific Features

- **Field ID-based encoding**: Uses 1-based field IDs corresponding to case class field positions
- **Forward compatibility**: Unknown fields are skipped during decoding
- **Out-of-order decoding**: Fields can arrive in any order on the wire
- **TBinaryProtocol**: Uses the standard Thrift binary protocol

### Thrift Type Mappings

| Scala Type | Thrift Type |
|------------|-------------|
| `Unit` | VOID |
| `Boolean` | BOOL |
| `Byte` | BYTE |
| `Short`, `Char` | I16 |
| `Int` | I32 |
| `Long` | I64 |
| `Float`, `Double` | DOUBLE |
| `String` | STRING |
| `BigInt` | Binary (STRING) |
| `BigDecimal` | STRUCT |
| `java.time.*` | STRING (ISO format) or I32 |
| `List[A]` | LIST |
| `Map[K, V]` | MAP |
| Case classes | STRUCT |
| Sealed traits | Indexed variant |

## Supported Types

All formats support the full set of ZIO Blocks Schema primitive types:

**Numeric Types**:
- `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`
- `BigInt`, `BigDecimal`

**Text Types**:
- `String`

**Special Types**:
- `Unit`, `UUID`, `Currency`

**Java Time Types**:
- `Instant`, `LocalDate`, `LocalTime`, `LocalDateTime`
- `OffsetTime`, `OffsetDateTime`, `ZonedDateTime`
- `Duration`, `Period`
- `Year`, `YearMonth`, `MonthDay`
- `DayOfWeek`, `Month`
- `ZoneId`, `ZoneOffset`

**Composite Types**:
- Records (case classes)
- Variants (sealed traits)
- Sequences (`List`, `Vector`, `Set`, `Array`, etc.)
- Maps (`Map[K, V]`)
- Options (`Option[A]`)
- Eithers (`Either[A, B]`)
- Wrappers (newtypes)

## Cross-Platform Support

| Format | JVM | Scala.js |
|--------|-----|----------|
| JSON | ✓ | ✓ |
| TOON | ✓ | ✓ |
| MessagePack | ✓ | ✓ |
| Avro | ✓ | ✗ |
| Thrift | ✓ | ✗ |
| BSON | ✓ | ✗ |

## Error Handling

All formats return `Either[SchemaError, A]` for decoding operations. Errors include path information for debugging:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(ToonFormat)

// Example: decoding invalid bytes
val invalidBytes = "invalid: data\nwrong: format".getBytes
val result = codec.decode(invalidBytes)

result match {
  case Right(person) => println(s"Decoded: $person")
  case Left(error) => 
    // SchemaError includes information about the decode failure
    error.errors.foreach(e => println(s"Error: ${e.message}"))
}
```

---

## HTTP Model

`zio-http-model` is a **pure, zero-dependency HTTP data model** for building HTTP clients and servers. It provides immutable types representing requests, responses, headers, URLs, paths, query parameters, and all HTTP primitives.

The module is designed as a pure data layer:

- **Zero effects**: no streaming, no I/O, no mutable state (except monotonic lazy-parse caches in `Headers`)
- **Zero ZIO dependency**: uses `zio.blocks.chunk.Chunk`, not `zio.Chunk`
- **Single encoding contract**: `Path` and `QueryParams` store values decoded internally, encode only on output
- **Cross-platform**: JVM and Scala.js support
- **Cross-version**: Scala 2.13 and 3.x support

```scala
package zio.http

// Core request/response types
final case class Request(method: Method, url: URL, headers: Headers, body: Body, version: Version)
final case class Response(status: Status, headers: Headers, body: Body, version: Version)

// URL structure
final case class URL(scheme: Option[Scheme], host: Option[String], port: Option[Int],
                     path: Path, queryParams: QueryParams, fragment: Option[String])

final case class Path(segments: Chunk[String], hasLeadingSlash: Boolean, trailingSlash: Boolean)
final class QueryParams private[http] (...)

// HTTP primitives
sealed abstract class Method(val name: String, val ordinal: Int)
opaque type Status = Int  // Scala 3
sealed abstract class Version(val major: Int, val minor: Int)
sealed trait Scheme

// Headers and body
final class Headers private[http] (...)
sealed trait Header
final class Body private (val data: Chunk[Byte], val contentType: ContentType)

// Supporting types
final case class ContentType(mediaType: MediaType, boundary: Option[Boundary], charset: Option[Charset])
final case class ResponseCookie(...), RequestCookie(name: String, value: String)
final case class Form(entries: Chunk[(String, String)])
```

## Motivation

HTTP libraries often couple protocol concerns with effects and streaming, making it difficult to:

- Share data structures across client and server implementations
- Serialize requests/responses for caching or testing
- Work with HTTP primitives without committing to a specific effect system

`zio-http-model` solves this by providing pure data types that:

- Represent all HTTP concepts as immutable values
- Encode only on output (decode on input, store decoded)
- Parse incrementally with lazy caching (`Headers` parses typed headers on first access and caches the result)
- Work with any effect system or none at all

## Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-http-model" % "<version>"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-http-model" % "<version>"
```

Supported Scala versions: 2.13.x and 3.x

## Quick Start

Creating a request with query parameters:

```scala

val url = URL.parse("https://api.example.com/users?active=true").toOption.get
val request = Request.get(url)

val withHeader = request.addHeader("authorization", "Bearer token123")
```

Creating a JSON response:

```scala

val jsonBody = Body.fromString("""{"message":"ok"}""", Charset.UTF8)
val response = Response(
  status = Status.Ok,
  body = jsonBody,
  headers = Headers("content-type" -> "application/json")
)
```

## Method

`Method` represents standard HTTP methods as case objects:

```scala
sealed abstract class Method(val name: String, val ordinal: Int)
```

### Predefined Methods

```scala

val get     = Method.GET
val post    = Method.POST
val put     = Method.PUT
val delete  = Method.DELETE
val patch   = Method.PATCH
val head    = Method.HEAD
val options = Method.OPTIONS
val trace   = Method.TRACE
val connect = Method.CONNECT
```

### Parsing

```scala

Method.fromString("GET")    // Some(Method.GET)
Method.fromString("POST")   // Some(Method.POST)
Method.fromString("CUSTOM") // None (unknown method)
```

### Rendering

```scala

Method.render(Method.GET)  // "GET"
Method.GET.name            // "GET"
Method.GET.toString        // "GET"
```

## Status

`Status` is an opaque type alias for `Int` in Scala 3 (AnyVal wrapper in Scala 2.13), providing zero-allocation status codes with predefined constants.

```scala
opaque type Status = Int  // Scala 3
```

### Predefined Status Codes

Status codes are organized by category:

```scala

// 1xx Informational
Status.Continue           // 100
Status.SwitchingProtocols // 101

// 2xx Success
Status.Ok                 // 200
Status.Created            // 201
Status.NoContent          // 204

// 3xx Redirection
Status.MovedPermanently   // 301
Status.Found              // 302
Status.SeeOther           // 303
Status.NotModified        // 304

// 4xx Client Errors
Status.BadRequest         // 400
Status.Unauthorized       // 401
Status.Forbidden          // 403
Status.NotFound           // 404

// 5xx Server Errors
Status.InternalServerError // 500
Status.BadGateway          // 502
Status.ServiceUnavailable  // 503
```

### Creating Status Codes

```scala

val custom = Status(418)          // I'm a teapot
val ok     = Status.fromInt(200)  // Status.Ok
```

### Status Code Operations

```scala

val status = Status.Ok

status.code              // 200
status.text              // "OK"
status.isSuccess         // true (2xx)
status.isInformational   // false (1xx)
status.isRedirection     // false (3xx)
status.isClientError     // false (4xx)
status.isServerError     // false (5xx)
status.isError           // false (4xx or 5xx)
```

## Version

`Version` represents HTTP protocol versions:

```scala
sealed abstract class Version(val major: Int, val minor: Int)
```

### Predefined Versions

```scala

val v10 = Version.`HTTP/1.0`
val v11 = Version.`HTTP/1.1`
val v20 = Version.`HTTP/2.0`
val v30 = Version.`HTTP/3.0`
```

### Parsing and Rendering

```scala

Version.fromString("HTTP/1.1") // Some(Version.`HTTP/1.1`)
Version.fromString("HTTP/2")   // Some(Version.`HTTP/2.0`)
Version.fromString("HTTP/3")   // Some(Version.`HTTP/3.0`)

Version.render(Version.`HTTP/1.1`) // "HTTP/1.1"
Version.`HTTP/2.0`.text            // "HTTP/2.0"
```

## Scheme

`Scheme` represents URL schemes with support for HTTP, HTTPS, WebSocket (WS, WSS), and custom schemes:

```scala
sealed trait Scheme {
  def text: String
  def defaultPort: Option[Int]
  def isSecure: Boolean
  def isWebSocket: Boolean
}
```

### Predefined Schemes

```scala

val http  = Scheme.HTTP   // http://, port 80
val https = Scheme.HTTPS  // https://, port 443
val ws    = Scheme.WS     // ws://, port 80
val wss   = Scheme.WSS    // wss://, port 443
```

### Scheme Properties

```scala

Scheme.HTTPS.isSecure      // true
Scheme.WS.isWebSocket      // true
Scheme.HTTP.defaultPort    // Some(80)
```

### Custom Schemes

```scala

val custom = Scheme.Custom("git+ssh")
custom.text         // "git+ssh"
custom.defaultPort  // None
```

### Parsing

```scala

Scheme.fromString("https")    // Scheme.HTTPS
Scheme.fromString("wss")      // Scheme.WSS
Scheme.fromString("custom")   // Scheme.Custom("custom")
```

## Charset

`Charset` represents character encodings with JVM-only conversion to `java.nio.charset.Charset`:

```scala
sealed abstract class Charset(val name: String)
```

### Predefined Charsets

```scala

val utf8      = Charset.UTF8       // "UTF-8"
val ascii     = Charset.ASCII      // "US-ASCII"
val iso88591  = Charset.ISO_8859_1 // "ISO-8859-1"
val utf16     = Charset.UTF16      // "UTF-16"
val utf16be   = Charset.UTF16BE    // "UTF-16BE"
val utf16le   = Charset.UTF16LE    // "UTF-16LE"
```

### Parsing

```scala

Charset.fromString("UTF-8")      // Some(Charset.UTF8)
Charset.fromString("utf8")       // Some(Charset.UTF8) (case-insensitive)
Charset.fromString("ISO-8859-1") // Some(Charset.ISO_8859_1)
Charset.fromString("LATIN1")     // Some(Charset.ISO_8859_1) (alias)
```

## Boundary

`Boundary` represents multipart form-data boundaries:

```scala

val boundary = Boundary("----WebKitFormBoundary7MA4YWxkTrZu0gW")
boundary.value    // "----WebKitFormBoundary7MA4YWxkTrZu0gW"
boundary.toString // "----WebKitFormBoundary7MA4YWxkTrZu0gW"
```

### Generating Boundaries

```scala

val generated = Boundary.generate  // random 24-character alphanumeric string
```

## PercentEncoder

`PercentEncoder` provides RFC 3986 percent-encoding for URL components. Each URL component has different encoding rules:

```scala

val segment = PercentEncoder.encode("hello world", ComponentType.PathSegment)
// "hello%20world"

val queryKey = PercentEncoder.encode("filter[name]", ComponentType.QueryKey)
// "filter%5Bname%5D"

val decoded = PercentEncoder.decode("hello%20world")
// "hello world"
```

### Component Types

The encoder recognizes these component types:

- `PathSegment`: path segments between `/`
- `QueryKey`: query parameter names
- `QueryValue`: query parameter values
- `Fragment`: fragment identifiers after `#`
- `UserInfo`: userinfo in authority

Each type has specific rules for which characters must be percent-encoded.

## ContentType

`ContentType` combines a media type with optional charset and boundary parameters:

```scala

val json = ContentType(MediaTypes.application.`json`)

val htmlUtf8 = ContentType(
  mediaType = MediaTypes.text.`html`,
  charset = Some(Charset.UTF8)
)

val multipart = ContentType(
  mediaType = MediaTypes.multipart.`form-data`,
  boundary = Some(Boundary("----boundary"))
)
```

### Parsing

```scala

ContentType.parse("application/json")
// Right(ContentType(MediaType("application", "json")))

ContentType.parse("text/html; charset=utf-8")
// Right(ContentType(..., charset = Some(Charset.UTF8)))

ContentType.parse("multipart/form-data; boundary=abc123")
// Right(ContentType(..., boundary = Some(Boundary("abc123"))))

ContentType.parse("")
// Left("Invalid content type: cannot be empty")
```

### Rendering

```scala

val ct = ContentType(
  MediaTypes.text.`plain`,
  charset = Some(Charset.UTF8)
)

ct.render  // "text/plain; charset=UTF-8"
```

### Predefined Content Types

```scala

val json   = ContentType.`application/json`
val plain  = ContentType.`text/plain`
val html   = ContentType.`text/html`
val binary = ContentType.`application/octet-stream`
```

## Path

`Path` represents URL paths with decoded segments stored internally:

```scala
final case class Path(
  segments: Chunk[String],
  hasLeadingSlash: Boolean,
  trailingSlash: Boolean
)
```

Paths use a **single encoding contract**: decode on input, store decoded, encode on output.

### Creating Paths

```scala

val empty = Path.empty                    // ""
val root  = Path.root                     // "/"
val users = Path("/users")                // segments: ["users"], leading slash: true
val api   = Path("api/v1/users/")         // segments: ["api", "v1", "users"], trailing slash: true
```

### Parsing Encoded Paths

`Path.fromEncoded` decodes percent-encoded segments:

```scala

val path = Path.fromEncoded("/hello%20world/foo%2Fbar")
// Path(Chunk("hello world", "foo/bar"), hasLeadingSlash = true, trailingSlash = false)

path.segments(0)  // "hello world" (decoded)
path.segments(1)  // "foo/bar" (decoded)
```

### Building Paths

```scala

val base = Path("/api")
val extended = base / "users" / "123"  // "/api/users/123"

val combined = Path("/api") ++ Path("v1/users")  // "/api/v1/users"
```

### Encoding Paths

```scala

val path = Path(Chunk("hello world", "foo/bar"), hasLeadingSlash = true, trailingSlash = false)

path.encode  // "/hello%20world/foo%2Fbar" (percent-encoded)
path.render  // "/hello world/foo/bar" (decoded for display)
```

### Path Properties

```scala

val path = Path("/api/v1/users/")

path.isEmpty           // false
path.nonEmpty          // true
path.length            // 3 (number of segments)
path.hasLeadingSlash   // true
path.trailingSlash     // true
```

### Path Navigation

```scala

val path = Path("/api/v1/users")

// Slash manipulation
path.addLeadingSlash     // same (already has one)
path.dropLeadingSlash    // "api/v1/users" (relative)
path.addTrailingSlash    // "/api/v1/users/"
path.dropTrailingSlash   // same (already no trailing slash)

// Inspecting
path.isRoot              // false (root is "/" with no segments)
Path.root.isRoot         // true

// Prefix checking
path.startsWith(Path("api/v1"))  // true

// Slicing
path.drop(1)       // "/v1/users" (drop first segment)
path.take(2)       // "/api/v1" (take first 2 segments)
path.dropRight(1)  // "/api/v1" (drop last segment)
path.initial       // "/api/v1" (all but last)
path.last          // Some("users")
path.reverse       // "/users/v1/api"
```

## QueryParams

`QueryParams` stores query parameters with multiple values per key:

```scala
final class QueryParams private[http] (
  private val keys: Array[String],
  private val vals: Array[Chunk[String]],
  val size: Int
)
```

Like `Path`, query parameters store decoded values internally and encode only on output.

### Creating QueryParams

```scala

val empty = QueryParams.empty

val params = QueryParams(
  "name" -> "Alice",
  "age" -> "30",
  "active" -> "true"
)
```

### Parsing Encoded Query Strings

```scala

val params = QueryParams.fromEncoded("name=Alice%20Smith&age=30&active=true")

params.getFirst("name")    // Some("Alice Smith") (decoded)
params.getFirst("age")     // Some("30")
params.getFirst("active")  // Some("true")
```

### Accessing Values

```scala

val params = QueryParams(
  "color" -> "red",
  "color" -> "blue",
  "size" -> "large"
)

params.get("color")       // Some(Chunk("red", "blue"))
params.getFirst("color")  // Some("red")
params.getFirst("size")   // Some("large")
params.getFirst("other")  // None
params.has("color")       // true
```

### Modifying QueryParams

```scala

val params = QueryParams("a" -> "1", "b" -> "2")

val added = params.add("c", "3")         // adds "c=3"
val set   = params.set("a", "100")       // replaces all "a" values
val removed = params.remove("b")         // removes all "b" entries
```

### Encoding

```scala

val params = QueryParams(
  "name" -> "Alice Smith",
  "filter[status]" -> "active"
)

params.encode  // "name=Alice%20Smith&filter%5Bstatus%5D=active"
```

### Converting to List

```scala

val params = QueryParams("a" -> "1", "a" -> "2", "b" -> "3")
params.toList  // List(("a", "1"), ("a", "2"), ("b", "3"))
```

## URL

`URL` combines scheme, host, port, path, query parameters, and fragment:

```scala
final case class URL(
  scheme: Option[Scheme],
  host: Option[String],
  port: Option[Int],
  path: Path,
  queryParams: QueryParams,
  fragment: Option[String]
)
```

### Parsing URLs

```scala

val absolute = URL.parse("https://api.example.com:8080/users?active=true#results")
// Right(URL(
//   scheme = Some(Scheme.HTTPS),
//   host = Some("api.example.com"),
//   port = Some(8080),
//   path = Path("/users"),
//   queryParams = QueryParams("active" -> "true"),
//   fragment = Some("results")
// ))

val relative = URL.parse("/api/users?page=2")
// Right(URL(
//   scheme = None,
//   host = None,
//   port = None,
//   path = Path("/api/users"),
//   queryParams = QueryParams("page" -> "2"),
//   fragment = None
// ))
```

The parser handles:

- IPv6 hosts in brackets: `http://[::1]:8080/`
- Userinfo: `https://user:pass@example.com/` (userinfo is skipped)
- Relative URLs: `/path?query`
- Fragment identifiers: `#section`

### Building URLs

```scala

val base = URL.root  // http://localhost/

val extended = base / "api" / "users"  // adds path segments

val withQuery = extended ?? ("active", "true") ?? ("page", "1")
// http://localhost/api/users?active=true&page=1
```

### URL from Path

```scala

val path = Path("/api/users")
val url = URL.fromPath(path)  // relative URL with just path
```

### Encoding URLs

```scala

val url = URL.parse("https://example.com/hello world?name=Alice Smith").toOption.get

url.encode  // "https://example.com/hello%20world?name=Alice%20Smith"
url.toString  // same as encode
```

### URL Properties

```scala

val absolute = URL.parse("https://example.com/").toOption.get
val relative = URL.parse("/api/users").toOption.get

absolute.isAbsolute  // true (has scheme)
relative.isRelative  // true (no scheme)
```

### URL Transformation

```scala

val url = URL.parse("https://api.example.com:8080/users?page=1").toOption.get

// Setting components
url.host("other.com")           // changes host
url.port(9090)                  // changes port
url.scheme(Scheme.HTTP)          // changes scheme
url.path(Path("/v2/users"))      // replaces path
url.fragment("top")              // sets fragment

// Adding paths
url.addPath("123")             // appends segment: /users/123
url.addPath(Path("v2/items"))  // appends multi-segment path

// Query manipulation
url.addQueryParams(QueryParams("sort" -> "name"))
url.updateQueryParams(_.remove("page"))

// Slash manipulation (delegates to path)
url.addLeadingSlash
url.dropTrailingSlash

// Derived properties
url.hostPort   // Some("api.example.com:8080")
url.relative   // strips scheme/host/port, keeps path and query
```

## Header

`Header` is a trait representing typed HTTP headers:

```scala
trait Header {
  def headerName: String
  def renderedValue: String
}
```

Each header type has a companion object implementing `Header.Typed[H]` for parsing and rendering.

### Predefined Header Types

```scala

val contentType   = headers.ContentType
val accept        = headers.Accept
val authorization = headers.Authorization
val host          = headers.Host
val userAgent     = headers.UserAgent
val cacheControl  = headers.CacheControl
val contentLength = headers.ContentLength
val location      = headers.Location
val setCookie     = headers.SetCookieHeader
val cookie        = headers.CookieHeader
```

### Creating Typed Headers

```scala

val ct = headers.ContentType(
  ContentType(MediaTypes.application.`json`, charset = Some(Charset.UTF8))
)

val auth = headers.Authorization.Bearer("token123")

val host = headers.Host("api.example.com", Some(8080))
```

### Parsing Headers

```scala

headers.ContentType.parse("application/json; charset=utf-8")
// Right(headers.ContentType(...))

headers.Host.parse("example.com:443")
// Right(headers.Host("example.com", Some(443)))

headers.ContentLength.parse("1024")
// Right(headers.ContentLength(1024))

headers.ContentLength.parse("-1")
// Left("Invalid content-length: -1")
```

### Custom Headers

```scala

val custom = Header.Custom("x-request-id", "abc-123")
custom.headerName     // "x-request-id"
custom.renderedValue  // "abc-123"
```

## Headers

`Headers` is a flat array-based collection with lazy monotonic parsing:

```scala
final class Headers private[http] (
  private val names: Array[String],
  private val rawValues: Array[String],
  private val parsed: Array[AnyRef],  // null -> unparsed, value -> cached
  val size: Int
)
```

When you call `get[H]`, the header is parsed once and cached. Subsequent calls return the cached result.

### Creating Headers

```scala

val empty = Headers.empty

val headers = Headers(
  "content-type" -> "application/json",
  "authorization" -> "Bearer token",
  "x-request-id" -> "abc-123"
)
```

### Getting Typed Headers

```scala

val headers = Headers(
  "content-type" -> "application/json",
  "content-length" -> "1024"
)

val ct = headers.get(h.ContentType)
// Some(h.ContentType(...)) (parsed and cached)

val cl = headers.get(h.ContentLength)
// Some(h.ContentLength(1024)) (parsed and cached)

val auth = headers.get(h.Authorization)
// None (not present)
```

### Getting Raw Values

```scala

val headers = Headers("x-custom" -> "value")

headers.rawGet("x-custom")  // Some("value")
headers.rawGet("missing")   // None
```

### Getting All Headers of a Type

Some headers can appear multiple times (like `Set-Cookie`):

```scala

val headers = Headers(
  "set-cookie" -> "session=abc",
  "set-cookie" -> "preference=dark"
)

val cookies = headers.getAll(h.SetCookieHeader)
// Chunk(h.SetCookieHeader(...), h.SetCookieHeader(...))
```

### Modifying Headers

```scala

val headers = Headers("a" -> "1", "b" -> "2")

val added = headers.add("c", "3")        // adds "c: 3"
val set   = headers.set("a", "100")      // replaces all "a" values
val removed = headers.remove("b")        // removes all "b" entries
val has = headers.has("a")               // true
```

### Converting to List

```scala

val headers = Headers("a" -> "1", "b" -> "2")
headers.toList  // List(("a", "1"), ("b", "2"))
```

### Combining Headers

```scala

val auth = Headers("authorization" -> "Bearer token")
val cors = Headers("access-control-allow-origin" -> "*")

val combined = auth ++ cors  // Headers(authorization: ..., access-control-allow-origin: ...)

combined.contains("authorization")  // true (alias for `has`)
combined.toChunk                    // Chunk(("authorization", ...), ("access-control-allow-origin", ...))
```

## Body

`Body` wraps a materialized `Chunk[Byte]` with a content type:

```scala
final class Body private (
  val data: Chunk[Byte],
  val contentType: ContentType
)
```

### Creating Bodies

`Body.empty` provides an empty body with default `application/octet-stream` content type:

```scala

val empty = Body.empty
// Body(data = Chunk.empty, contentType = application/octet-stream)
```

`Body.fromString` creates a body with `text/plain` content type:

```scala

val fromString = Body.fromString("Hello, World!", Charset.UTF8)
// Content-Type: text/plain; charset=UTF-8
```

`Body.fromArray` creates a body with default `application/octet-stream` content type:

```scala

val fromBytes = Body.fromArray(Array[Byte](1, 2, 3))
// Content-Type: application/octet-stream
```

`Body.fromChunk` creates a body from a `Chunk[Byte]` with optional content type:

```scala

val chunk = Chunk[Byte](1, 2, 3, 4, 5)
val body = Body.fromChunk(chunk)
// Content-Type: application/octet-stream (default)

val jsonBody = Body.fromChunk(chunk, ContentType(MediaTypes.application.`json`))
// Content-Type: application/json
```

### Reading Bodies

`Body` provides direct access to data and content type:

```scala

val body = Body.fromString("Hello!", Charset.UTF8)

body.length           // 6
body.isEmpty          // false
body.nonEmpty         // true
body.asString()       // "Hello!" (UTF-8 default)
body.asString(Charset.ASCII)  // "Hello!" (explicit charset)
body.data             // Chunk[Byte](72, 101, 108, 108, 111, 33)
body.contentType      // ContentType(text/plain; charset=UTF-8)
```

## Cookie

Cookies are split into `RequestCookie` and `ResponseCookie` with different structures:

```scala
final case class RequestCookie(name: String, value: String)

final case class ResponseCookie(
  name: String,
  value: String,
  domain: Option[String],
  path: Option[Path],
  maxAge: Option[Long],
  isSecure: Boolean,
  isHttpOnly: Boolean,
  sameSite: Option[SameSite]
)
```

### SameSite

```scala

val strict = SameSite.Strict
val lax    = SameSite.Lax
val none   = SameSite.None_  // underscore avoids conflict with scala.None
```

### Parsing Request Cookies

```scala

val cookies = Cookie.parseRequest("session=abc123; preference=dark")
// Chunk(RequestCookie("session", "abc123"), RequestCookie("preference", "dark"))
```

### Parsing Response Cookies

```scala

val cookie = Cookie.parseResponse("session=abc; Domain=example.com; Path=/; Secure; HttpOnly; SameSite=Strict")
// Right(ResponseCookie(
//   name = "session",
//   value = "abc",
//   domain = Some("example.com"),
//   path = Some(Path("/")),
//   isSecure = true,
//   isHttpOnly = true,
//   sameSite = Some(SameSite.Strict)
// ))
```

### Rendering Cookies

```scala

val requestCookies = Chunk(
  RequestCookie("session", "abc"),
  RequestCookie("theme", "dark")
)
Cookie.renderRequest(requestCookies)
// "session=abc; theme=dark"

val responseCookie = ResponseCookie(
  name = "session",
  value = "xyz",
  domain = Some("example.com"),
  path = Some(Path("/")),
  maxAge = Some(3600),
  isSecure = true,
  isHttpOnly = true,
  sameSite = Some(SameSite.Strict)
)
Cookie.renderResponse(responseCookie)
// "session=xyz; Domain=example.com; Path=/; Max-Age=3600; Secure; HttpOnly; SameSite=Strict"
```

## Form

`Form` represents URL-encoded form data:

```scala
final case class Form(entries: Chunk[(String, String)])
```

### Creating Forms

```scala

val empty = Form.empty

val form = Form(
  "username" -> "alice",
  "password" -> "secret",
  "remember" -> "true"
)
```

### Accessing Form Data

```scala

val form = Form(
  "tag" -> "scala",
  "tag" -> "functional",
  "page" -> "1"
)

form.get("tag")      // Some("scala") (first value)
form.getAll("tag")   // Chunk("scala", "functional")
form.get("page")     // Some("1")
form.get("missing")  // None
```

### Modifying Forms

```scala

val form = Form("a" -> "1")
val added = form.add("b", "2")  // Form(("a", "1"), ("b", "2"))
```

### Encoding and Parsing

```scala

val form = Form(
  "name" -> "Alice Smith",
  "email" -> "alice@example.com"
)

val encoded = form.encode
// "name=Alice%20Smith&email=alice%40example.com"

val parsed = Form.fromString(encoded)
// Form with decoded entries
```

## FormField

`FormField` is a sealed trait for multipart form fields, supporting simple key-value pairs, text parts with optional metadata, and binary parts:

```scala

// Simple key-value field
val simple = FormField.Simple("username", "alice")

// Text field with optional content type and filename
val text = FormField.Text(
  name = "bio",
  value = "Hello world",
  contentType = Some(ContentType(MediaTypes.text.`plain`)),
  filename = None
)

// Binary field with content type and optional filename
val binary = FormField.Binary(
  name = "avatar",
  data = Chunk.fromArray(Array[Byte](1, 2, 3)),
  contentType = ContentType(MediaTypes.image.`png`),
  filename = Some("avatar.png")
)

// All variants share a common `name` accessor
simple.name  // "username"
text.name    // "bio"
binary.name  // "avatar"
```

## Request

`Request` combines all HTTP request components:

```scala
final case class Request(
  method: Method,
  url: URL,
  headers: Headers,
  body: Body,
  version: Version
)
```

### Creating Requests

```scala

val url = URL.parse("https://api.example.com/users").toOption.get
val getRequest = Request.get(url)

val jsonBody = Body.fromString("""{"name":"Alice"}""", Charset.UTF8)
val postRequest = Request.post(url, jsonBody)
```

### Request Factory Methods

```scala

val url = URL.parse("https://api.example.com/resource/1").toOption.get
val body = Body.fromString("""{"name":"updated"}""")

Request.get(url)              // GET with empty body
Request.post(url, body)       // POST with body
Request.put(url, body)        // PUT with body
Request.patch(url, body)      // PATCH with body
Request.delete(url)           // DELETE with empty body
Request.head(url)             // HEAD with empty body
Request.options(url)          // OPTIONS with empty body
```

### Modifying Requests

All modification methods return a new `Request`—the original is unchanged:

```scala

val request = Request.get(URL.parse("https://api.example.com/users").toOption.get)

// Adding/modifying headers
request.addHeader("Accept", "application/json")
request.addHeaders(Headers("X-A" -> "1", "X-B" -> "2"))
request.setHeader("Accept", "text/html")       // replaces existing
request.removeHeader("Accept")

// Replacing components
request.body(Body.fromString("data"))
request.url(URL.parse("/other").toOption.get)
request.method(Method.POST)
request.version(Version.`HTTP/2.0`)
// Functional updates
request.updateHeaders(_.add("X-Custom", "value"))
request.updateUrl(_ / "123")   // appends path segment
```

Full control:

```scala

val url = URL.parse("https://api.example.com/users").toOption.get
val body = Body.fromString("""{"name":"Alice"}""", Charset.UTF8)

val request = Request(
  method = Method.POST,
  url = url,
  headers = Headers(
    "content-type" -> "application/json",
    "authorization" -> "Bearer token123"
  ),
  body = body,
  version = Version.`HTTP/1.1`
)
```

### Accessing Request Data

```scala

val request = Request.get(URL.parse("/api/users?page=1").toOption.get)

request.path         // Path("/api/users")
request.queryParams  // QueryParams("page" -> "1")
request.contentType  // Option[ContentType]
request.header(h.Authorization)  // Option[h.Authorization]
```

## Response

`Response` represents HTTP responses:

```scala
final case class Response(
  status: Status,
  headers: Headers,
  body: Body,
  version: Version
)
```

### Creating Responses

```scala

val ok = Response.ok  // 200 OK, empty body

val notFound = Response.notFound  // 404 Not Found, empty body
```

### Response Factory Methods

```scala

// Status-only responses
Response.ok                    // 200
Response.notFound              // 404
Response.badRequest            // 400
Response.unauthorized          // 401
Response.forbidden             // 403
Response.internalServerError   // 500
Response.serviceUnavailable    // 503

// Responses with bodies
Response.text("Hello, World!")    // 200 with text/plain body
Response.json("""{'ok':true}""")  // 200 with application/json in headers and body
// Redirects
Response.redirect("/new-location")                     // 307 Temporary Redirect
Response.redirect("/new-location", isPermanent = true) // 308 Permanent Redirect
Response.seeOther("/other")                            // 303 See Other
```

Note that `Response.json` creates bodies with `application/json` content-type on the `Body` itself, not just in the headers.

### Modifying Responses

```scala

val response = Response.ok

// Adding/modifying headers
response.addHeader("X-Custom", "value")
response.setHeader("X-Custom", "new-value")
response.removeHeader("X-Custom")

// Replacing components
response.body(Body.fromString("data"))
response.status(Status.Created)
response.version(Version.`HTTP/2.0`)
// Functional update
response.updateHeaders(_.add("X-Request-Id", "abc"))

// Add a Set-Cookie header
response.addCookie(ResponseCookie("session", "abc123"))
```

Full control:

```scala

val jsonBody = Body.fromString("""{"message":"created"}""", Charset.UTF8)

val response = Response(
  status = Status.Created,
  headers = Headers(
    "content-type" -> "application/json",
    "location" -> "/users/123"
  ),
  body = jsonBody,
  version = Version.`HTTP/1.1`
)
```

### Accessing Response Data

```scala

val response = Response.ok

response.status.code                      // 200
response.status.isSuccess                 // true
response.contentType                      // Option[ContentType]
response.header(h.ContentType)       // Option[h.ContentType]
response.header(h.Location)          // Option[h.Location]
```

## Advanced Usage

### Building a Complete HTTP Exchange

```scala

// Build request
val url = URL.parse("https://api.example.com/users").toOption.get
val requestBody = Body.fromString("""{"name":"Alice","age":30}""", Charset.UTF8)

val request = Request(
  method = Method.POST,
  url = url,
  headers = Headers(
    "content-type" -> "application/json",
    "authorization" -> "Bearer abc123",
    "user-agent" -> "MyClient/1.0"
  ),
  body = requestBody,
  version = Version.`HTTP/1.1`
)

// Build response
val responseBody = Body.fromString("""{"id":123,"name":"Alice","age":30}""", Charset.UTF8)

val response = Response(
  status = Status.Created,
  headers = Headers(
    "content-type" -> "application/json",
    "location" -> "/users/123"
  ),
  body = responseBody,
  version = Version.`HTTP/1.1`
)
```

### URL Building with Fluent API

```scala

val url = URL.parse("https://api.example.com").toOption.get

val extended = (url / "v1" / "users" / "123") ?? ("include", "profile") ?? ("include", "posts")

extended.encode
// "https://api.example.com/v1/users/123?include=profile&include=posts"
```

### Typed Header Access

```scala

val headers = Headers(
  "content-type" -> "application/json; charset=utf-8",
  "content-length" -> "1024",
  "authorization" -> "Bearer token"
)

// Type-safe header access with parsing
val ct = headers.get(h.ContentType)
ct.map(_.value.charset)  // Some(Some(Charset.UTF8))

val cl = headers.get(h.ContentLength)
cl.map(_.length)  // Some(1024)

// Raw access
headers.rawGet("authorization")  // Some("Bearer token")
```

### Cookie Management

```scala

// Parse cookies from request header
val cookieHeader = "session=abc; theme=dark"
val requestCookies = Cookie.parseRequest(cookieHeader)

// Create response with Set-Cookie headers
val sessionCookie = ResponseCookie(
  name = "session",
  value = "xyz123",
  path = Some(Path("/")),
  maxAge = Some(3600),
  isSecure = true,
  isHttpOnly = true,
  sameSite = Some(SameSite.Strict)
)

val response = Response(
  status = Status.Ok,
  headers = Headers(
    "set-cookie" -> Cookie.renderResponse(sessionCookie)
  ),
  body = Body.empty
)
```

### Form Submission

```scala

val form = Form(
  "username" -> "alice",
  "password" -> "secret",
  "remember" -> "true"
)

val formBody = Body.fromString(form.encode, Charset.UTF8)

val request = Request(
  method = Method.POST,
  url = URL.parse("/login").toOption.get,
  headers = Headers(
    "content-type" -> "application/x-www-form-urlencoded"
  ),
  body = formBody,
  version = Version.`HTTP/1.1`
)
```

## Design Principles

### Single Encoding Contract

`Path` and `QueryParams` store decoded values internally. Encoding happens only at output boundaries:

- `Path.fromEncoded(s)` decodes, stores decoded segments
- `Path.encode` encodes segments for transmission
- `QueryParams.fromEncoded(s)` decodes, stores decoded key-value pairs
- `QueryParams.encode` encodes for transmission

This eliminates double-encoding bugs and clarifies responsibilities.

### Lazy Header Parsing

`Headers` stores raw string values and parses typed headers on first access. Parsed results are cached in a parallel `Array[AnyRef]` for O(1) subsequent lookups. This design:

- Avoids parsing headers that are never accessed
- Avoids re-parsing the same header multiple times
- Supports unknown/custom headers without parsing failures

### No Streaming

Bodies are fully materialized `Chunk[Byte]`. Streaming is left to higher-level HTTP libraries that compose with this data model. This keeps the model simple and effect-free.

### Zero ZIO Dependency

The module uses `zio.blocks.chunk.Chunk` instead of `zio.Chunk`, making it usable in any Scala project without ZIO.

## Schema-Based Typed Access (zio-http-model-schema)

The `zio-http-model-schema` module provides schema-based extraction of query parameters and headers with automatic decoding and validation.

### Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-http-model-schema" % "<version>"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-http-model-schema" % "<version>"
```

### Imports

Import the schema module to enable extension methods:

```scala

```

### Query Parameter Extraction

`QueryParams` gains schema-based extraction methods via implicit conversions:

```scala

val url = URL.parse("/api/users?page=2&tag=scala&tag=fp").toOption.get
val params = url.queryParams

// Extract single value with automatic decoding
params.query[Int]("page")
// Right(2)

// Extract all values for a key
params.queryAll[String]("tag")
// Right(Chunk("scala", "fp"))

// Extract with default fallback
params.queryOrElse[Int]("limit", 10)
// 10 - uses default since "limit" not present
```

### Header Extraction

`Headers` gains schema-based extraction methods:

```scala

val request = Request.get(URL.parse("/").toOption.get)
  .addHeader("x-page", "5")
  .addHeader("x-tag", "scala")
  .addHeader("x-tag", "functional")

val headers = request.headers

// Extract single header value
headers.header[Int]("x-page")
// Right(5)

// Extract all header values
headers.headerAll[String]("x-tag")
// Right(Chunk("scala", "functional"))

// Extract with default fallback
headers.headerOrElse[Int]("x-limit", 100)
// 100
```

### Request and Response Extensions

`Request` and `Response` gain schema-based extraction methods that delegate to their query parameters and headers:

```scala

// Request query parameter extraction
val request = Request.get(URL.parse("/search?q=zio&limit=20").toOption.get)

request.query[String]("q")
// Right("zio")

request.query[Int]("limit")
// Right(20)

// Response header extraction via schema
val response = Response.ok.addHeader("x-correlation-id", "abc-123")

val responseOps = new ResponseSchemaOps(response)
responseOps.header[String]("x-correlation-id")
```

### Error Handling

Schema-based extraction returns `Either[QueryParamError, A]` or `Either[HeaderError, A]` for explicit error handling:

```scala

val params = QueryParams("name" -> "Alice", "age" -> "invalid")

params.query[String]("name") match {
  case Right(name) => println(s"Name: $name")
  case Left(QueryParamError.Missing(key)) => println(s"Missing key: $key")
  case Left(QueryParamError.Malformed(key, value, cause)) =>
    println(s"Failed to parse $key=$value: $cause")
}

params.query[Int]("age") match {
  case Right(age) => println(s"Age: $age")
  case Left(QueryParamError.Missing(key)) => println(s"Missing key: $key")
  case Left(QueryParamError.Malformed(key, value, cause)) =>
    println(s"Failed to parse $key=$value: $cause")
}
```

### Supported Types

The schema module provides built-in `Schema` instances for common types. Any type with a `Schema[T]` can be extracted:

- **Primitives**: `String`, `Int`, `Long`, `Boolean`, `Double`, `Float`, `Short`, `Byte`, `Char`
- **Big Numbers**: `BigInt`, `BigDecimal`
- **UUID**: `java.util.UUID`

For custom types, define a `Schema[T]` instance using schema derivation or manual construction.

---

## JsonDiffer

`JsonDiffer` is a **diff algorithm for JSON values** that computes the minimal [`JsonPatch`](./json-patch.md) transforming one `Json` value into another. It is the foundation of `JsonPatch.diff`.

```scala
object JsonDiffer {
  def diff(source: Json, target: Json): JsonPatch
}
```

## Overview

`JsonDiffer` solves a fundamental problem: given two `Json` values, what is the most compact representation of the changes from one to the other?

Instead of transmitting or storing the entire new value, `JsonDiffer` emits a `JsonPatch` containing only the differences. For each type of change — numeric updates, string mutations, array reordering, field additions — it selects the most space-efficient representation:

```
Original JSON              Target JSON
┌──────────────────┐      ┌──────────────────┐
│ {                │      │ {                │
│   "name": "Alice"│      │   "name": "Alice"│
│   "age": 25      │─────→│   "age": 26      │
│   "city": "NYC"  │ diff │   "city": "NYC"  │
│ }                │      │ }                │
└──────────────────┘      └──────────────────┘
        │
        ▼
   JsonPatch {
     ObjectEdit(
       Modify("age", NumberDelta(1))
     )
   }
```

The core guarantee: `JsonDiffer.diff(source, target).apply(source) == Right(target)` always holds.

## Motivation

Transmitting or storing entire JSON documents wastes bandwidth, disk space, and network latency when only a fraction of the data changes. `JsonDiffer` enables:

- **Efficient data synchronization** — send only what changed, not the entire document
- **Change tracking and audit logs** — record every mutation as a first-class value
- **Optimistic concurrency control** — detect conflicts by analyzing patches rather than full documents
- **Replay and time-travel debugging** — reconstruct any historical state by applying patches in sequence
- **Compression** — patches are often 10–100× smaller than the full target value

## The Diffing Algorithm

`JsonDiffer.diff` adapts its strategy per `Json` type, choosing the most compact representation for each kind of change.

### Type Mismatches

When the types differ (e.g., number to string, object to array), `JsonDiffer` emits `Op.Set` to replace the value entirely. This is the only compact choice when the types diverge:

```scala

val numberToString = JsonDiffer.diff(Json.Number(42), Json.String("hello"))
val objectToArray  = JsonDiffer.diff(
  Json.Object("x" -> Json.Number(1)),
  Json.Array(Json.Number(1), Json.Number(2))
)
```

Both patches consist of a single `Set` operation:

```scala

numberToString.apply(Json.Number(42))
objectToArray.apply(Json.Object("x" -> Json.Number(1)))
```

### Numbers

For numeric changes, `JsonDiffer` emits `NumberDelta` — representing the difference between old and new values as a delta rather than replacing the entire number:

```scala

val increment = JsonDiffer.diff(Json.Number(100), Json.Number(105))
val decrement = JsonDiffer.diff(Json.Number(50), Json.Number(48))
```

The patches store deltas, not full new values:

```scala

increment.apply(Json.Number(100))
decrement.apply(Json.Number(50))
```

### Strings

For strings, `JsonDiffer` chooses between two strategies:

1. **LCS (Longest Common Subsequence) edit operations** — if the edits are more compact than the full new string.
2. **Full replacement with `Set`** — if the string has changed so much that storing the entire new value is smaller.

The algorithm computes common characters, then emits `Insert` and `Delete` operations (with `Append` and `Modify` existing as supported patch ops but not currently produced by `JsonDiffer`). It only emits the edits if their byte size is smaller than the new string's length:

```scala

// Prefix insertion — common suffix preserved
val addPrefix = JsonDiffer.diff(Json.String("world"), Json.String("hello world"))

// Complete replacement — almost no common subsequence
val replacement = JsonDiffer.diff(Json.String("abc"), Json.String("xyz"))
```

The first patch uses `StringEdit` (compact), the second uses `Set` (more efficient):

```scala
addPrefix.apply(Json.String("world"))
replacement.apply(Json.String("abc"))
```

### Arrays

For arrays, `JsonDiffer` uses **LCS-aligned insert, delete, and append operations** to transform the old array into the new one. It aligns common elements and generates the minimum sequence of mutations:

```scala

// Append elements at the end
val append = JsonDiffer.diff(
  Json.Array(Json.Number(1), Json.Number(2)),
  Json.Array(Json.Number(1), Json.Number(2), Json.Number(3))
)

// Delete and reorder — LCS finds the common elements
val reorder = JsonDiffer.diff(
  Json.Array(Json.Number(1), Json.Number(2), Json.Number(3)),
  Json.Array(Json.Number(3), Json.Number(2), Json.Number(1))
)
```

Both patches use efficient `ArrayEdit` operations:

```scala
append.apply(Json.Array(Json.Number(1), Json.Number(2)))
reorder.apply(Json.Array(Json.Number(1), Json.Number(2), Json.Number(3)))
```

### Objects

For objects, `JsonDiffer` compares field-by-field:

- Fields in the target but not in the source become `Add` operations.
- Fields in the source but not in the target become `Remove` operations.
- Fields in both are recursively diffed — if the values differ, a `Modify` with a sub-patch is emitted.

```scala

val original = Json.Object(
  "name" -> Json.String("Alice"),
  "age"  -> Json.Number(25),
  "city" -> Json.String("NYC")
)

val updated = Json.Object(
  "name"  -> Json.String("Alice"),
  "age"   -> Json.Number(26),
  "email" -> Json.String("alice@example.com")
)

val patch = JsonDiffer.diff(original, updated)
```

The patch contains `Add`, `Remove`, and `Modify` operations:

```scala
patch.apply(original)
```

### Nested Structures

`JsonDiffer` handles deeply nested objects and arrays recursively. Each field or element is diffed independently, producing compact patches at every level:

```scala

val original = Json.Object(
  "user" -> Json.Object(
    "name" -> Json.String("Alice"),
    "scores" -> Json.Array(Json.Number(95), Json.Number(87))
  )
)

val updated = Json.Object(
  "user" -> Json.Object(
    "name" -> Json.String("Alice"),
    "scores" -> Json.Array(Json.Number(95), Json.Number(88), Json.Number(92))
  )
)

val patch = JsonDiffer.diff(original, updated)
```

The patch navigates to the nested array and emits only the array changes:

```scala
patch.apply(original)
```

## Core Operations

`JsonDiffer` exposes a single public operation: `diff`, which computes the minimal `JsonPatch` that transforms `source` into `target`. Returns an empty patch if the values are already equal:

```scala
object JsonDiffer {
  def diff(source: Json, target: Json): JsonPatch
}
```

The roundtrip property always holds — applying the patch to the source always yields the target:

```scala

val source = Json.Object("x" -> Json.Number(10), "y" -> Json.Number(20))
val target = Json.Object("x" -> Json.Number(10), "y" -> Json.Number(21))
val patch  = JsonDiffer.diff(source, target)
```

```scala
patch.apply(source) == Right(target)
// res13: Boolean = true
```

`JsonDiffer.diff` is also available as `JsonPatch.diff` and as the `Json#diff` extension method:

```scala

// Via JsonPatch companion
val p1 = JsonPatch.diff(Json.Number(1), Json.Number(2))

// Via Json extension method
val p2 = Json.Number(1).diff(Json.Number(2))
```

## Integration

`JsonDiffer` is the implementation behind the public `JsonPatch.diff` API. You typically interact with it through `JsonPatch.diff` or the `Json#diff` extension method rather than calling `JsonDiffer.diff` directly.

The relationship is simple: `JsonPatch.diff` delegates to `JsonDiffer.diff` and wraps the result in a `JsonPatch` for further composition and application.

Once you have a `JsonPatch` from `JsonDiffer`, use the full `JsonPatch` API to apply it with different modes, compose multiple patches, or convert to/from `DynamicPatch`:

```scala

val source = Json.Object("count" -> Json.Number(0))
val target = Json.Object("count" -> Json.Number(1))
val patch  = JsonDiffer.diff(source, target)
```

Apply with different failure-handling modes:

```scala
patch.apply(source, PatchMode.Strict)
patch.apply(source, PatchMode.Lenient)
patch.apply(source, PatchMode.Clobber)
```

Compose patches with `++`:

```scala
val patch2 = JsonDiffer.diff(target, Json.Object("count" -> Json.Number(2)))
val combined = patch ++ patch2
```

## Implementation Notes

`JsonDiffer.diff` uses the LCS (Longest Common Subsequence) algorithm for both strings and arrays, ensuring that the number of edit operations is minimized. For strings, it compares character sequences; for arrays, it compares JSON elements by structural equality.

The LCS-based approach is particularly effective for arrays and strings with significant common subsequences — a common pattern in real-world data mutation scenarios (e.g., adding an item to a list, inserting a few characters into a string).

:::note
The LCS algorithm has O(n·m) time complexity, where n and m are the lengths of the two sequences. For very large strings or arrays, consider whether you need the minimal patch or can accept a faster approximation.
:::

## Running the Examples

`JsonDiffer` is the foundation of the `JsonPatch` API. Runnable examples demonstrating `JsonPatch.diff` (which internally uses `JsonDiffer`) are available in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run JsonPatch examples with sbt:**

These examples demonstrate how `JsonDiffer.diff` computes minimal patches through the public `JsonPatch.diff` API:

```bash
sbt "schema-examples/runMain jsonpatch.JsonPatchDiffAndApplyExample"
```

```bash
sbt "schema-examples/runMain jsonpatch.JsonPatchOperationsExample"
```

```bash
sbt "schema-examples/runMain jsonpatch.JsonPatchCompositionExample"
```

```bash
sbt "schema-examples/runMain jsonpatch.CompleteJsonPatchExample"
```

---

## JsonPatch

`JsonPatch` is an **untyped, composable patch** for [`Json`](./json.md) values. It represents a sequence of operations that transform one `Json` value into another — computed automatically via a diff algorithm or constructed manually. The two fundamental operations are `JsonPatch.diff` to compute a patch between two `Json` values, and `JsonPatch#apply` to apply it.

`JsonPatch`:
- is a pure value — applying it never mutates the input
- is composable via `++`, sequencing two patches one after another
- supports three failure-handling modes: `Strict`, `Lenient`, and `Clobber`
- carries its own `Schema` instances for full serialization support
- converts bidirectionally to/from `DynamicPatch` for use in generic patching pipelines

The `JsonPatch` type wraps a sequence of operations:

```scala
final case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp])
```

## Motivation

In most systems, updating JSON data means transmitting the entire new value — even when only a single field changed. `JsonPatch` solves this by representing changes as a **first-class value** that can be:

- **Transmitted efficiently** — send only what changed, not the entire document
- **Stored for audit logs** — record every change for compliance, debugging, or undo
- **Composed** — merge multiple changes into a single atomic patch
- **Serialized** — persist patches to disk or a message queue and replay them later

```
Source JSON                    Target JSON
┌─────────────────────┐        ┌─────────────────────┐
│ { "name": "Alice",  │        │ { "name": "Alice",  │
│   "age": 25,        │──diff──│   "age": 26,        │
│   "city": "NYC" }   │        │   "city": "NYC" }   │
└─────────────────────┘        └─────────────────────┘
           │
           ▼
      JsonPatch {
        ObjectEdit(
          Modify("age", NumberDelta(1))
        )
      }
           │
           ▼ apply
┌─────────────────────┐
│ { "name": "Alice",  │
│   "age": 26,        │
│   "city": "NYC" }   │
└─────────────────────┘
```

The "hello world" for `JsonPatch` is diff-then-apply. We compute a patch from `source` to `target`, then verify that applying it to `source` reproduces `target`:

```scala

val source = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(25))
val target = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(26))

val patch: JsonPatch = JsonPatch.diff(source, target)
```

Applying the patch to `source` always yields `Right(target)`:

```scala
patch.apply(source) == Right(target)
// res1: Boolean = true
```

## Creating Patches

There are three ways to create a `JsonPatch`: compute one automatically with `JsonPatch.diff`, construct one manually with `JsonPatch.root` or `JsonPatch.apply`, or start from the identity patch `JsonPatch.empty`.

### `JsonPatch.diff`

Computes the minimal `JsonPatch` that transforms `source` into `target`. Uses a smart diff strategy per value type — see the [Diffing Algorithm](#diffing-algorithm) section for details:

```scala
object JsonPatch {
  def diff(source: Json, target: Json): JsonPatch
}
```

`JsonPatch.diff` is also available as the `Json#diff` extension method:

```scala

// Via companion object
val p1 = JsonPatch.diff(Json.Number(10), Json.Number(15))

// Via extension method on Json
val p2 = Json.Number(10).diff(Json.Number(15))

// Nested object diff produces minimal ObjectEdit
val p3 = JsonPatch.diff(
  Json.Object("a" -> Json.Number(1), "b" -> Json.Number(2)),
  Json.Object("a" -> Json.Number(1), "b" -> Json.Number(9))
)
// p3 only touches "b", leaves "a" unchanged
```

### `JsonPatch.root`

Creates a patch with a single operation applied at the root of the value:

```scala
object JsonPatch {
  def root(operation: JsonPatch.Op): JsonPatch
}
```

For example, we can replace the root entirely, increment a number, or add a field to a root object:

```scala

// Replace the entire value
val replaceAll = JsonPatch.root(Op.Set(Json.Null))

// Increment a number at the root
val increment = JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1))))

// Add a field to a root object
val addField = JsonPatch.root(Op.ObjectEdit(Chunk(ObjectOp.Add("active", Json.Boolean(true)))))
```

### `JsonPatch.apply`

Creates a patch with a single operation applied at the specified `DynamicOptic` path. Use this when targeting a nested location within the value:

```scala
object JsonPatch {
  def apply(path: DynamicOptic, operation: JsonPatch.Op): JsonPatch
}
```

Paths are built fluently on `DynamicOptic.root` using `.field(name)` to navigate object fields and `.at(index)` to navigate array elements. For instance, to increment `age` inside a `user` object:

```scala

val agePath  = DynamicOptic.root.field("user").field("age")
val agePatch = JsonPatch(agePath, Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1))))
val nested   = Json.Object("user" -> Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(25)))
```

Applying the patch navigates to the nested `age` field and increments it:

```scala
agePatch.apply(nested)
// res5: Either[SchemaError, Json] = Right(
//   Object(
//     IndexedSeq(
//       (
//         "user",
//         Object(IndexedSeq(("name", String("Alice")), ("age", Number(26))))
//       )
//     )
//   )
// )
```

### `JsonPatch.empty`

The empty patch. Applying it to any `Json` value returns that value unchanged. `JsonPatch.empty` is the identity element for `++`:

```scala
object JsonPatch {
  val empty: JsonPatch
}
```

`JsonPatch.empty` is useful as a neutral starting point when building patches conditionally:

```scala

```

Both `JsonPatch#isEmpty` and applying `JsonPatch.empty` confirm the identity property:

```scala
JsonPatch.empty.isEmpty
// res7: Boolean = true
JsonPatch.empty.apply(Json.Number(42))
// res8: Either[SchemaError, Json] = Right(Number(42))
(JsonPatch.empty ++ JsonPatch.empty).isEmpty
// res9: Boolean = true
```

### `JsonPatch.fromDynamicPatch`

Converts a generic `DynamicPatch` to a `JsonPatch`. Returns `Left[SchemaError]` for operations not representable in JSON:
- Temporal deltas (`InstantDelta`, `DurationDelta`, etc.) — JSON has no native time type
- Non-string map keys — JSON object keys must always be strings

All numeric delta types (`IntDelta`, `LongDelta`, `DoubleDelta`, etc.) are widened to `NumberDelta(BigDecimal)`:

```scala
object JsonPatch {
  def fromDynamicPatch(patch: DynamicPatch): Either[SchemaError, JsonPatch]
}
```

The round-trip through `DynamicPatch` preserves numeric deltas:

```scala

val original: JsonPatch                  = JsonPatch.diff(Json.Number(1), Json.Number(2))
val dynPatch: DynamicPatch               = original.toDynamicPatch
val back: Either[SchemaError, JsonPatch] = JsonPatch.fromDynamicPatch(dynPatch)
```

The roundtrip succeeds and the recovered patch equals the original:

```scala
back == Right(original)
// res11: Boolean = true
```

## Core Operations

`JsonPatch` exposes operations for applying patches, composing them, and converting between formats. The three groups of operations are applying, composing, and converting.

### Applying Patches

The primary way to use a `JsonPatch` is to call `JsonPatch#apply` or the `Json#patch` extension method, both of which accept an optional `PatchMode` argument.

#### `apply`

Applies this patch to a `Json` value. Returns `Right` with the patched value on success, or `Left[SchemaError]` on failure. The `mode` parameter controls failure handling — see [`PatchMode`](#patchmode):

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def apply(value: Json, mode: PatchMode = PatchMode.Strict): Either[SchemaError, Json]
}
```

`apply` is also available via the `Json#patch` extension method. Both forms produce the same result:

```scala

val applyJson  = Json.Object("score" -> Json.Number(10))
val applyPatch = JsonPatch.root(Op.ObjectEdit(Chunk(
  ObjectOp.Modify("score", JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(5)))))
)))
```

The direct call and the extension method are equivalent:

```scala
applyPatch.apply(applyJson)
// res13: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("score", Number(15))))
// )
applyJson.patch(applyPatch)
// res14: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("score", Number(15))))
// )
applyJson.patch(applyPatch, PatchMode.Lenient)
// res15: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("score", Number(15))))
// )
```

#### `isEmpty`

Returns `true` if this patch contains no operations:

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def isEmpty: Boolean
}
```

A patch computed between two identical values also produces an empty patch:

```scala

```

```scala
JsonPatch.empty.isEmpty
// res17: Boolean = true
JsonPatch.diff(Json.Number(1), Json.Number(1)).isEmpty
// res18: Boolean = true
```

### Composing Patches

`++` is the principal operator for building complex patches from smaller, focused ones. The `JsonPatch.empty` value is the identity element for `++`:

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def ++(that: JsonPatch): JsonPatch
}
```

Concatenating two patches applies `this` first, then `that`. This allows building a single patch that updates multiple fields independently:

```scala

val renamePatch = JsonPatch(
  DynamicOptic.root.field("name"),
  Op.Set(Json.String("Bob"))
)
val incrAgePatch = JsonPatch(
  DynamicOptic.root.field("age"),
  Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1)))
)
val combinedPatch = renamePatch ++ incrAgePatch
val personJson    = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(25))
```

The combined patch applies both operations in sequence:

```scala
combinedPatch.apply(personJson)
// res20: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("name", String("Bob")), ("age", Number(26))))
// )
```

### Converting

`toDynamicPatch` converts a `JsonPatch` to a [`DynamicPatch`](./patch.md). This is always safe — every JSON operation maps to a corresponding dynamic operation. `NumberDelta` widens to `BigDecimalDelta`:

```scala
case class JsonPatch(ops: Chunk[JsonPatch.JsonPatchOp]) {
  def toDynamicPatch: DynamicPatch
}
```

To convert in the opposite direction, use `JsonPatch.fromDynamicPatch` — see [Creating Patches](#creating-patches) above:

```scala

val patch: JsonPatch  = JsonPatch.diff(Json.Number(1), Json.Number(5))
val dyn: DynamicPatch = patch.toDynamicPatch
```

## PatchMode

`PatchMode` controls how `JsonPatch#apply` reacts when an operation's precondition is not met (e.g., a field is missing, or `ObjectOp.Add` targets a key that already exists):

| Mode | Behaviour |
|------|-----------|
| `PatchMode.Strict` (default) | Returns `Left[SchemaError]` on the first failure |
| `PatchMode.Lenient` | Silently skips failing operations; returns `Right` with partial result |
| `PatchMode.Clobber` | Overwrites on conflicts; forces through missing-field errors where possible |

`ObjectOp.Add` fails in `Strict` mode when the key already exists. In `Lenient` mode the conflicting add is silently skipped; in `Clobber` mode it overwrites the existing value:

```scala

val modeJson  = Json.Object("a" -> Json.Number(1))
val modePatch = JsonPatch.root(Op.ObjectEdit(Chunk(ObjectOp.Add("a", Json.Number(99)))))
```

The three modes produce different outcomes for the same conflicting patch:

```scala
modeJson.patch(modePatch, PatchMode.Strict)
// res23: Either[SchemaError, Json] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(IndexedSeq()),
//         expectation = "Key 'a' already exists in object"
//       )
//     )
//   )
// )
modeJson.patch(modePatch, PatchMode.Lenient)
// res24: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("a", Number(1))))
// )
modeJson.patch(modePatch, PatchMode.Clobber)
// res25: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("a", Number(99))))
// )
```

## Operation Types

A `JsonPatch` is a sequence of `JsonPatchOp` values. Each `JsonPatchOp` pairs a `DynamicOptic` path with an `Op`:

```scala
final case class JsonPatchOp(path: DynamicOptic, operation: Op)
```

The full `Op` hierarchy covers five cases, from full replacement to fine-grained array and object edits:

```
Op (sealed trait)
 ├── Op.Set                — replace the target value entirely
 ├── Op.PrimitiveDelta     — numeric increment or string edit
 │    ├── PrimitiveOp.NumberDelta
 │    └── PrimitiveOp.StringEdit
 │         ├── StringOp.Insert
 │         ├── StringOp.Delete
 │         ├── StringOp.Append
 │         └── StringOp.Modify
 ├── Op.ArrayEdit          — insert / append / delete / modify array elements
 │    ├── ArrayOp.Insert
 │    ├── ArrayOp.Append
 │    ├── ArrayOp.Delete
 │    └── ArrayOp.Modify
 ├── Op.ObjectEdit         — add / remove / modify object fields
 │    ├── ObjectOp.Add
 │    ├── ObjectOp.Remove
 │    └── ObjectOp.Modify
 └── Op.Nested             — groups a sub-patch under a shared path prefix
```

### `Op.Set`

Replaces the target value with a new `Json` value, regardless of the current value. Works on any `Json` type:

```scala
final case class Set(value: Json) extends Op
```

`Op.Set` can replace across types — for example, replacing a number with a string, or resetting a nested field to `null`:

```scala

val setString  = JsonPatch.root(Op.Set(Json.String("replaced")))
val setNull    = JsonPatch(DynamicOptic.root.field("status"), Op.Set(Json.Null))
val withStatus = Json.Object("status" -> Json.String("active"), "id" -> Json.Number(1))
```

Both patches replace their target regardless of its current type:

```scala
setString.apply(Json.Number(123))
// res27: Either[SchemaError, Json] = Right(String("replaced"))
setNull.apply(withStatus)
// res28: Either[SchemaError, Json] = Right(
//   Object(IndexedSeq(("status", null), ("id", Number(1))))
// )
```

### `Op.PrimitiveDelta`

Applies a primitive mutation to a scalar value — either a numeric increment (`NumberDelta`) or a sequence of string edits (`StringEdit`):

```scala
final case class PrimitiveDelta(op: PrimitiveOp) extends Op
```

#### `PrimitiveOp.NumberDelta`

Adds `delta` to a `Json.Number`. Use a negative value to subtract. Fails if the target is not a `Json.Number`:

```scala
final case class NumberDelta(delta: BigDecimal) extends PrimitiveOp
```

Positive deltas increment; negative deltas decrement:

```scala

val inc = JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(5))))
val dec = JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(-3))))
```

```scala
inc.apply(Json.Number(10))
// res30: Either[SchemaError, Json] = Right(Number(15))
dec.apply(Json.Number(10))
// res31: Either[SchemaError, Json] = Right(Number(7))
```

#### `PrimitiveOp.StringEdit`

Applies a sequence of `StringOp` operations to a `Json.String`. `JsonPatch.diff` generates `StringEdit` automatically when it is more compact than a full `Set`:

```scala
final case class StringEdit(ops: Chunk[StringOp]) extends PrimitiveOp
```

The `StringOp` cases:

| Case | Parameters | Effect |
|------|-----------|--------|
| `StringOp.Insert(index, text)` | position, text | Inserts `text` before character `index` |
| `StringOp.Delete(index, length)` | position, count | Removes `length` characters starting at `index` |
| `StringOp.Append(text)` | text | Appends `text` to the end |
| `StringOp.Modify(index, length, text)` | position, count, text | Replaces `length` characters at `index` with `text` |

We can insert a prefix before the first character using `StringOp.Insert`:

```scala

val insertPatch = JsonPatch.root(
  Op.PrimitiveDelta(PrimitiveOp.StringEdit(Chunk(StringOp.Insert(0, "Hello, "))))
)
```

```scala
insertPatch.apply(Json.String("world"))
// res33: Either[SchemaError, Json] = Right(String("Hello, world"))
```

:::tip
For most use cases, let `JsonPatch.diff` generate `StringEdit` automatically. The diff algorithm uses an LCS (Longest Common Subsequence) comparison and only emits `StringEdit` when it produces fewer bytes than a plain `Set`.
:::

### `Op.ArrayEdit`

Applies a sequence of `ArrayOp` operations to a `Json.Array`. Operations are applied in order, and each one sees the result of the previous:

```scala
final case class ArrayEdit(ops: Chunk[ArrayOp]) extends Op
```

The `ArrayOp` cases:

| Case | Parameters | Effect |
|------|-----------|--------|
| `ArrayOp.Insert(index, values)` | position, elements | Inserts `values` before `index` |
| `ArrayOp.Append(values)` | elements | Appends `values` to the end |
| `ArrayOp.Delete(index, count)` | position, count | Removes `count` elements starting at `index` |
| `ArrayOp.Modify(index, op)` | position, op | Applies `op` to the element at `index` |

Multiple `ArrayOp`s in a single `ArrayEdit` can be combined — here we transform `[1, 2, 3]` into `[0, 1, 2, 4]` in one pass:

```scala

val arrayPatch = JsonPatch.root(Op.ArrayEdit(Chunk(
  ArrayOp.Insert(0, Chunk(Json.Number(0))),
  ArrayOp.Delete(3, 1),
  ArrayOp.Append(Chunk(Json.Number(4)))
)))
val originalArr = Json.Array(Json.Number(1), Json.Number(2), Json.Number(3))
```

```scala
arrayPatch.apply(originalArr)
// res35: Either[SchemaError, Json] = Right(
//   Array(IndexedSeq(Number(0), Number(1), Number(2), Number(4)))
// )
```

:::note
Array indices in `ArrayOp.Delete` and `ArrayOp.Modify` refer to the state of the array **after** all preceding ops in the same `ArrayEdit` have been applied.
:::

### `Op.ObjectEdit`

Applies a sequence of `ObjectOp` operations to a `Json.Object`. Operations are applied in order:

```scala
final case class ObjectEdit(ops: Chunk[ObjectOp]) extends Op
```

The `ObjectOp` cases:

| Case | Parameters | Effect |
|------|-----------|--------|
| `ObjectOp.Add(key, value)` | field name, value | Adds a new field; fails in `Strict` mode if key exists |
| `ObjectOp.Remove(key)` | field name | Removes an existing field |
| `ObjectOp.Modify(key, patch)` | field name, sub-patch | Applies `patch` recursively to the field value |

A single `ObjectEdit` can add, remove, and modify fields in one operation:

```scala

val originalObj = Json.Object(
  "name"  -> Json.String("Alice"),
  "age"   -> Json.Number(25),
  "city"  -> Json.String("NYC")
)

val objPatch = JsonPatch.root(Op.ObjectEdit(Chunk(
  ObjectOp.Add("email", Json.String("alice@example.com")),
  ObjectOp.Remove("city"),
  ObjectOp.Modify("age", JsonPatch.root(Op.PrimitiveDelta(PrimitiveOp.NumberDelta(BigDecimal(1)))))
)))
```

```scala
objPatch.apply(originalObj)
// res37: Either[SchemaError, Json] = Right(
//   Object(
//     IndexedSeq(
//       ("name", String("Alice")),
//       ("age", Number(26)),
//       ("email", String("alice@example.com"))
//     )
//   )
// )
```

### `Op.Nested`

Groups a sub-patch under a shared path prefix. `JsonPatch.diff` emits `Nested` automatically when multiple operations share a common navigation path — this avoids repeating the full path in each `JsonPatchOp`:

```scala
final case class Nested(patch: JsonPatch) extends Op
```

You rarely need to construct `Nested` manually; it is primarily an internal optimization used by the diff algorithm.

## Diffing Algorithm

`JsonPatch.diff` (and its alias `Json#diff`) delegate to [`JsonDiffer.diff`](./json-differ.md), which selects the most compact representation for each type of change:

| Value type | Change | Strategy |
|------------|--------|----------|
| Any | No change | No operation emitted |
| `Json.Number` | Value changed | `NumberDelta` — stores the numeric difference |
| `Json.String` | Value changed | `StringEdit` via LCS if smaller; otherwise `Set` |
| `Json.Array` | Elements changed | `ArrayEdit` with LCS-aligned `Insert`/`Delete`/`Append`/`Modify` |
| `Json.Object` | Fields changed | `ObjectEdit` with recursive per-field diff |
| Any | Type changed | `Set` — full replacement |

:::tip
`JsonPatch.diff` followed by `JsonPatch#apply` is always a lossless roundtrip: for any `source` and `target`, `JsonPatch.diff(source, target).apply(source) == Right(target)`.
:::

## JsonPatch vs RFC 6902 JSON Patch

ZIO Blocks' `JsonPatch` is **not** an implementation of [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902). The two share the same motivation but differ in design:

| | ZIO Blocks `JsonPatch` | RFC 6902 JSON Patch |
|---|---|---|
| Operations | Typed ADT (`Op.Set`, `Op.ArrayEdit`, …) | String-tagged JSON objects (`"op": "replace"`) |
| Paths | `DynamicOptic` (typed, composable) | JSON Pointer strings (`"/a/b/0"`) |
| Number changes | `NumberDelta` (stores diff) | `replace` (stores full new value) |
| String changes | LCS-based `StringEdit` | `replace` only |
| Array changes | LCS-aligned insert/delete | `add`, `remove`, `replace` at absolute indices |
| Serialization | Via ZIO Blocks `Schema` in any format | Always JSON |
| Composition | `++` operator | Array concatenation |

Use `JsonPatch` when working within ZIO Blocks. For interoperability with RFC 6902 tooling, convert the patch to JSON using the built-in `Schema` instances and reformat as needed.

## Advanced Usage

`JsonPatch`'s composability and first-class serializability unlock patterns beyond simple point-in-time updates.

### Building a Change Log

Because `JsonPatch` is a pure value with a `Schema`, we can serialize every change and replay or audit it later:

```scala

// Every mutation is a patch — store it instead of overwriting
val v0 = Json.Object("count" -> Json.Number(0))
val v1 = Json.Object("count" -> Json.Number(1))
val v2 = Json.Object("count" -> Json.Number(2))

val log: List[JsonPatch] = List(
  JsonPatch.diff(v0, v1),
  JsonPatch.diff(v1, v2)
)

// Replay: reconstruct any historical state
val replay = log.foldLeft(v0: Json)((state, patch) => patch.apply(state).getOrElse(state))
assert(replay == v2)
```

### Composing Targeted Sub-Patches

We can build a single patch that updates multiple nested fields by combining focused per-field patches with `++`:

```scala

def setField(field: String, value: Json): JsonPatch =
  JsonPatch(DynamicOptic.root.field(field), Op.Set(value))

val fieldPatch =
  setField("status", Json.String("active")) ++
  setField("updatedAt", Json.String("2025-01-01"))

val doc = Json.Object("status" -> Json.String("draft"), "id" -> Json.Number(42))
```

Applying the composed patch updates both fields in one step:

```scala
fieldPatch.apply(doc)
// res40: Either[SchemaError, Json] = Left(
//   SchemaError(
//     List(
//       MissingField(source = DynamicOptic(IndexedSeq()), fieldName = "updatedAt")
//     )
//   )
// )
```

## Integration

`JsonPatch` integrates with `Json`, `DynamicPatch`, and the ZIO Blocks serialization system. Each integration point is covered below.

### With `Json`

`Json` exposes two extension methods as entry points into `JsonPatch`:

```scala

val source = Json.Object("x" -> Json.Number(1))
val target = Json.Object("x" -> Json.Number(2))

val patch: JsonPatch  = source.diff(target)          // compute patch
val result            = source.patch(patch)          // apply (Strict)
val lenient           = source.patch(patch, PatchMode.Lenient)
```

See [Json](./json.md) for the complete `Json` API.

### With `DynamicPatch`

`JsonPatch` and `DynamicPatch` are bidirectionally convertible. This is useful when patches originate from the typed `Patch[S]` system and need to be applied to raw JSON:

```scala

val jsonPatch: JsonPatch                  = JsonPatch.diff(Json.Number(1), Json.Number(3))

// JsonPatch → DynamicPatch (always succeeds)
val dynPatch: DynamicPatch               = jsonPatch.toDynamicPatch

// DynamicPatch → JsonPatch (may fail for temporal ops or non-string keys)
val back: Either[SchemaError, JsonPatch] = JsonPatch.fromDynamicPatch(dynPatch)
```

See [Patching](./patch.md) for the typed `Patch[S]` API.

### Serialization

`JsonPatch` ships with `Schema` instances for all nested operation types, enabling round-trip serialization via any ZIO Blocks codec:

```scala

val schema: Schema[JsonPatch] = implicitly[Schema[JsonPatch]]
```

See [Codec & Format](./codec.md) for how to derive and use codecs.

## Examples

Runnable examples are in `schema-examples/src/main/scala/jsonpatch/`:

| File | Topic |
|------|-------|
| `JsonPatchDiffAndApplyExample.scala` | `JsonPatch.diff`, `Json#diff`, `Json#patch`, roundtrip guarantee |
| `JsonPatchManualBuildExample.scala` | `JsonPatch.root`, path-based patches, `JsonPatch.empty` |
| `JsonPatchOperationsExample.scala` | All `Op` types — `Set`, `NumberDelta`, `StringEdit`, `ArrayEdit`, `ObjectEdit` |
| `JsonPatchCompositionExample.scala` | `++`, `PatchMode`, `toDynamicPatch`, `fromDynamicPatch` |
| `CompleteJsonPatchExample.scala` | Collaborative document editor with a full patch log, replay, and sync |

Run any example with:

```bash
sbt "schema-examples/runMain jsonpatch.CompleteJsonPatchExample"
```

---

## JSON Schema

`JsonSchema` provides first-class support for [JSON Schema 2020-12](https://json-schema.org/specification-links.html#2020-12) in ZIO Blocks. It enables parsing, construction, validation, and serialization of JSON Schemas as native Scala values.

## Overview

The `JsonSchema` type is a sealed ADT representing all valid JSON Schema documents:

```
JsonSchema
 ├── JsonSchema.True          (accepts all values - equivalent to {})
 ├── JsonSchema.False         (rejects all values - equivalent to {"not": {}})
 └── JsonSchema.Object        (full schema with all keywords)
```

Key features:

- **Full JSON Schema 2020-12 support** - All standard vocabularies (core, applicator, validation, format, meta-data)
- **Type-safe construction** - Smart constructors and builder pattern
- **Validation** - Validate JSON values against schemas with detailed error messages
- **Round-trip serialization** - Parse from JSON and serialize back without loss
- **Combinators** - Compose schemas with `&&` (allOf), `||` (anyOf), `!` (not)
- **817 of 844 official tests passing** (97%+)

## Deriving JSON Schema from Schema

The most common use case is deriving a JSON Schema from an existing `Schema[A]`.

### Basic Derivation

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Get JSON Schema directly from Schema
val jsonSchema: JsonSchema = Schema[Person].toJsonSchema

// The derived schema validates JSON values
val valid = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(30))
val invalid = Json.Object("name" -> Json.Number(123))

jsonSchema.conforms(valid)   // true
jsonSchema.conforms(invalid) // false
```

### Through JsonCodec

For more control, derive through `JsonCodec`:

```scala

case class User(email: String, active: Boolean)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

// Derive codec first, then get JSON Schema
val codec = Schema[User].derive(JsonFormat)
val jsonSchema = codec.toJsonSchema
```

## Creating Schemas

### Boolean Schemas

The simplest schemas accept or reject all values:

```scala

// Accepts any valid JSON value
val acceptAll = JsonSchema.True

// Rejects all JSON values
val rejectAll = JsonSchema.False
```

### Type Schemas

Create schemas that validate specific JSON types:

```scala

// Single type
val stringSchema = JsonSchema.ofType(JsonSchemaType.String)
val numberSchema = JsonSchema.ofType(JsonSchemaType.Number)
val integerSchema = JsonSchema.ofType(JsonSchemaType.Integer)
val booleanSchema = JsonSchema.ofType(JsonSchemaType.Boolean)
val arraySchema = JsonSchema.ofType(JsonSchemaType.Array)
val objectSchema = JsonSchema.ofType(JsonSchemaType.Object)
val nullSchema = JsonSchema.ofType(JsonSchemaType.Null)

// Convenience aliases
val isNull = JsonSchema.nullSchema
val isBoolean = JsonSchema.boolean
```

### String Schemas

Create schemas for string validation:

```scala

// String with length constraints (compile-time validated literals)
val username = JsonSchema.string(
  NonNegativeInt.literal(3),
  NonNegativeInt.literal(20)
)

// String with pattern
val hexColor = JsonSchema.string(
  pattern = RegexPattern.unsafe("^#[0-9a-fA-F]{6}$")
)

// String with format
val email = JsonSchema.string(format = Some("email"))
val dateTime = JsonSchema.string(format = Some("date-time"))
val uuid = JsonSchema.string(format = Some("uuid"))
```

### Numeric Schemas

Create schemas for number validation:

```scala

// Number with range
val percentage = JsonSchema.number(
  minimum = Some(BigDecimal(0)),
  maximum = Some(BigDecimal(100))
)

// Integer with exclusive bounds
val positiveInt = JsonSchema.integer(
  exclusiveMinimum = Some(BigDecimal(0))
)

// Number divisible by a value
val evenNumber = JsonSchema.integer(
  multipleOf = PositiveNumber.fromInt(2)
)
```

### Array Schemas

Create schemas for array validation:

```scala

// Array of strings
val stringArray = JsonSchema.array(
  items = Some(JsonSchema.ofType(JsonSchemaType.String))
)

// Array with length constraints
val shortList = JsonSchema.array(
  JsonSchema.ofType(JsonSchemaType.Number),
  NonNegativeInt.literal(1),
  NonNegativeInt.literal(5)
)

// Array with unique items
val uniqueNumbers = JsonSchema.array(
  items = Some(JsonSchema.ofType(JsonSchemaType.Number)),
  uniqueItems = Some(true)
)

// Tuple-like array with prefixItems
val point2D = JsonSchema.array(
  prefixItems = Some(NonEmptyChunk(
    JsonSchema.ofType(JsonSchemaType.Number),
    JsonSchema.ofType(JsonSchemaType.Number)
  ))
)
```

### Object Schemas

Create schemas for object validation:

```scala

// Object with properties
val person = JsonSchema.obj(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String),
    "age" -> JsonSchema.ofType(JsonSchemaType.Integer)
  )),
  required = Some(Set("name"))
)

// Object with no additional properties
val strictPerson = JsonSchema.obj(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String),
    "age" -> JsonSchema.ofType(JsonSchemaType.Integer)
  )),
  required = Some(Set("name")),
  additionalProperties = Some(JsonSchema.False)
)
```

### Enum and Const

```scala

// Enum of string values
val status = JsonSchema.enumOfStrings(NonEmptyChunk("pending", "active", "completed"))

// Enum of mixed values
val mixed = JsonSchema.enumOf(NonEmptyChunk(
  Json.String("auto"),
  Json.Number(0),
  Json.Boolean(true)
))

// Constant value
val alwaysTrue = JsonSchema.constOf(Json.Boolean(true))
```

## Schema Combinators

### Logical Composition

Combine schemas using logical operators:

```scala

val stringSchema = JsonSchema.ofType(JsonSchemaType.String)
val numberSchema = JsonSchema.ofType(JsonSchemaType.Number)
val nullSchema = JsonSchema.ofType(JsonSchemaType.Null)

// allOf - must match all schemas
val stringAndNotEmpty = stringSchema && JsonSchema.string(
  minLength = Some(zio.blocks.schema.json.NonNegativeInt.literal(1))
)

// anyOf - must match at least one schema
val stringOrNumber = stringSchema || numberSchema

// not - must not match the schema
val notNull = !nullSchema

// Combining operators
val nullableString = stringSchema || nullSchema
```

### Nullable Schemas

Make any schema nullable:

```scala

val stringSchema = JsonSchema.ofType(JsonSchemaType.String)

// Accepts string or null
val nullableString = stringSchema.withNullable
```

## Conditional Schemas

### if/then/else

Apply different schemas based on conditions:

```scala

// If type is string, require minLength
val conditionalSchema = JsonSchema.Object(
  `if` = Some(JsonSchema.ofType(JsonSchemaType.String)),
  `then` = Some(JsonSchema.string(minLength = Some(NonNegativeInt.literal(1)))),
  `else` = Some(JsonSchema.True)
)
```

### Dependent Schemas

Apply schemas when properties are present:

```scala

// If "credit_card" exists, require "billing_address"
val paymentSchema = JsonSchema.Object(
  properties = Some(ChunkMap(
    "credit_card" -> JsonSchema.ofType(JsonSchemaType.String),
    "billing_address" -> JsonSchema.ofType(JsonSchemaType.String)
  )),
  dependentRequired = Some(ChunkMap(
    "credit_card" -> Set("billing_address")
  ))
)
```

## Validation

### Basic Validation

```scala

val schema = JsonSchema.obj(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String),
    "age" -> JsonSchema.integer(minimum = Some(BigDecimal(0)))
  )),
  required = Some(Set("name"))
)

val validJson = Json.Object(
  "name" -> Json.String("Alice"),
  "age" -> Json.Number(30)
)

val invalidJson = Json.Object(
  "age" -> Json.Number(-5)
)

// Using check() - returns Option[SchemaError]
schema.check(validJson)   // None (valid)
schema.check(invalidJson) // Some(SchemaError(...))

// Using conforms() - returns Boolean
schema.conforms(validJson)   // true
schema.conforms(invalidJson) // false
```

### Validation Options

Control validation behavior:

```scala

val schema = JsonSchema.string(format = Some("email"))
val value = Json.String("not-an-email")

// With format validation (default)
val strictOptions = ValidationOptions.formatAssertion
schema.check(value, strictOptions) // Some(error)

// Without format validation (format as annotation only)
val lenientOptions = ValidationOptions.annotationOnly
schema.check(value, lenientOptions) // None
```

### Error Messages

Validation errors include path information:

```scala

val schema = JsonSchema.obj(
  properties = Some(ChunkMap(
    "users" -> JsonSchema.array(
      items = Some(JsonSchema.obj(
        properties = Some(ChunkMap(
          "email" -> JsonSchema.string(format = Some("email"))
        ))
      ))
    )
  ))
)

val invalid = Json.Object(
  "users" -> Json.Array(
    Json.Object("email" -> Json.String("invalid"))
  )
)

schema.check(invalid) match {
  case Some(err) => println(err.message)
  // "String 'invalid' is not a valid email address"
  case None => println("Valid")
}
```

## Parsing and Serialization

### Parsing from JSON

```scala

// From JSON string
val parsed = JsonSchema.parse("""
  {
    "type": "object",
    "properties": {
      "name": { "type": "string" }
    },
    "required": ["name"]
  }
""")

// From Json value
val json = Json.Object(
  "type" -> Json.String("string"),
  "minLength" -> Json.Number(1)
)
val fromJson = JsonSchema.fromJson(json)
```

### Serializing to JSON

```scala

val schema = JsonSchema.string(
  NonNegativeInt.literal(1),
  NonNegativeInt.literal(100)
)

val json = schema.toJson
// {"type":"string","minLength":1,"maxLength":100}

val jsonString = json.print
```

## Format Validation

The following formats are supported for validation:

| Format | Description | Example |
|--------|-------------|---------|
| `date-time` | RFC 3339 date-time | `2024-01-15T10:30:00Z` |
| `date` | RFC 3339 full-date | `2024-01-15` |
| `time` | RFC 3339 full-time | `10:30:00Z` |
| `email` | Email address | `user@example.com` |
| `uuid` | RFC 4122 UUID | `550e8400-e29b-41d4-a716-446655440000` |
| `uri` | RFC 3986 URI | `https://example.com/path` |
| `uri-reference` | RFC 3986 URI-reference | `/path/to/resource` |
| `ipv4` | IPv4 address | `192.168.1.1` |
| `ipv6` | IPv6 address | `2001:db8::1` |
| `hostname` | RFC 1123 hostname | `example.com` |
| `regex` | ECMA-262 regex | `^[a-z]+$` |
| `duration` | ISO 8601 duration | `P3Y6M4DT12H30M5S` |
| `json-pointer` | RFC 6901 JSON Pointer | `/foo/bar/0` |

Format validation is enabled by default. Use `ValidationOptions.annotationOnly` to treat `format` as annotation only (per JSON Schema spec).

## Unevaluated Properties and Items

JSON Schema 2020-12 introduces `unevaluatedProperties` and `unevaluatedItems` for validating properties/items not matched by any applicator keyword:

```scala

// Reject any properties not defined in properties or patternProperties
val strictObject = JsonSchema.Object(
  properties = Some(ChunkMap(
    "name" -> JsonSchema.ofType(JsonSchemaType.String)
  )),
  unevaluatedProperties = Some(JsonSchema.False)
)

// Reject extra array items not matched by prefixItems or items
val strictArray = JsonSchema.Object(
  prefixItems = Some(NonEmptyChunk(
    JsonSchema.ofType(JsonSchemaType.String),
    JsonSchema.ofType(JsonSchemaType.Number)
  )),
  unevaluatedItems = Some(JsonSchema.False)
)
```

## Schema Object Fields

`JsonSchema.Object` supports all JSON Schema 2020-12 keywords:

### Core Vocabulary
- `$id`, `$schema`, `$anchor`, `$dynamicAnchor`
- `$ref`, `$dynamicRef` (limited support - see Limitations)
- `$defs`, `$comment`

### Applicator Vocabulary
- `allOf`, `anyOf`, `oneOf`, `not`
- `if`, `then`, `else`
- `properties`, `patternProperties`, `additionalProperties`
- `propertyNames`, `dependentSchemas`
- `prefixItems`, `items`, `contains`

### Unevaluated Vocabulary
- `unevaluatedProperties`, `unevaluatedItems`

### Validation Vocabulary
- `type`, `enum`, `const`
- `multipleOf`, `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`
- `minLength`, `maxLength`, `pattern`
- `minItems`, `maxItems`, `uniqueItems`, `minContains`, `maxContains`
- `minProperties`, `maxProperties`, `required`, `dependentRequired`

### Format Vocabulary
- `format`

### Content Vocabulary
- `contentEncoding`, `contentMediaType`, `contentSchema`

### Meta-Data Vocabulary
- `title`, `description`, `default`, `deprecated`
- `readOnly`, `writeOnly`, `examples`

## Limitations

### Not Implemented

The following features require reference resolution and are **not supported**:

| Feature | Description |
|---------|-------------|
| `$ref` to external URIs | References to other files or URLs |
| `$dynamicRef` / `$dynamicAnchor` | Dynamic reference resolution |
| `$id` resolution | Base URI changing and resolution |
| Remote references | Fetching schemas from URLs |
| Recursive schemas via `$ref` | Self-referential schemas using references |

Local `$ref` within the same schema is partially supported for `#/$defs/...` references only.

### Known Edge Cases

| Case | Behavior |
|------|----------|
| Float/integer numeric equality | `1.0` is not treated as equal to `1` for `const`/`enum` |
| String length | Measured in codepoints, not grapheme clusters |
| Some `unevaluatedItems` with `contains` | Edge cases involving item evaluation tracking |

### Test Suite Compliance

The implementation passes **817 of 844 tests** (97%+) from the official JSON Schema Test Suite for draft2020-12. The remaining tests require reference resolution features listed above.

## Complete Example

```scala

// Define a complex schema
val userSchema = JsonSchema.obj(
  properties = Some(ChunkMap(
    "id" -> JsonSchema.string(format = Some("uuid")),
    "email" -> JsonSchema.string(format = Some("email")),
    "name" -> JsonSchema.string(
      NonNegativeInt.literal(1),
      NonNegativeInt.literal(100)
    ),
    "age" -> JsonSchema.integer(
      minimum = Some(BigDecimal(0)),
      maximum = Some(BigDecimal(150))
    ),
    "roles" -> JsonSchema.array(
      items = Some(JsonSchema.enumOfStrings(
        NonEmptyChunk("admin", "user", "guest")
      )),
      minItems = Some(NonNegativeInt.literal(1)),
      uniqueItems = Some(true)
    ),
    "metadata" -> JsonSchema.obj(
      additionalProperties = Some(JsonSchema.ofType(JsonSchemaType.String))
    )
  )),
  required = Some(Set("id", "email", "name", "roles")),
  additionalProperties = Some(JsonSchema.False)
)

// Validate some data
val validUser = Json.Object(
  "id" -> Json.String("550e8400-e29b-41d4-a716-446655440000"),
  "email" -> Json.String("alice@example.com"),
  "name" -> Json.String("Alice"),
  "roles" -> Json.Array(Json.String("admin"), Json.String("user"))
)

val invalidUser = Json.Object(
  "id" -> Json.String("not-a-uuid"),
  "email" -> Json.String("invalid-email"),
  "name" -> Json.String(""),
  "roles" -> Json.Array(),
  "extra" -> Json.String("not allowed")
)

userSchema.conforms(validUser)   // true
userSchema.conforms(invalidUser) // false

// Get detailed errors
userSchema.check(invalidUser) match {
  case Some(err) => println(err.message)
  case None => println("Valid!")
}

// Serialize the schema
val schemaJson = userSchema.toJson.print
// Can be sent to other tools, stored, or shared
```

## Cross-Platform Support

`JsonSchema` works across 2 platforms:

- **JVM** - Full functionality
- **Scala.js** - Browser and Node.js

All features work identically across platforms.

---

## Json

`Json` is an algebraic data type (ADT) for representing JSON values in ZIO Blocks. It provides a type-safe, schema-free way to work with JSON data, enabling navigation, transformation, merging, and querying without losing fidelity.

## Overview

The `Json` type represents all valid JSON values with six cases:

```
Json
 ├── Json.Object   (key-value pairs, order-preserving)
 ├── Json.Array    (ordered sequence of values)
 ├── Json.String   (text)
 ├── Json.Number   (arbitrary precision via BigDecimal)
 ├── Json.Boolean  (true/false)
 └── Json.Null     (null)
```

Key design decisions:

- **Objects use `Vector[(String, Json)]`** to preserve insertion order while providing order-independent equality
- **Numbers use `BigDecimal`** to preserve precision for financial and scientific data
- **All navigation returns `JsonSelection`** for fluent, composable chaining

## Creating JSON Values

### Using Constructors

```scala

// Object with named fields
val person = Json.Object(
  "name" -> Json.String("Alice"),
  "age" -> Json.Number(30),
  "active" -> Json.Boolean(true)
)

// Array of values
val numbers = Json.Array(Json.Number(1), Json.Number(2), Json.Number(3))

// Primitive values
val name = Json.String("Bob")
val count = Json.Number(42)
val flag = Json.Boolean(false)
val nothing = Json.Null
```

### Parsing JSON Strings

```scala

// Safe parsing (returns Either)
val parsed: Either[SchemaError, Json] = Json.parse("""{"name": "Alice", "age": 30}""")

// Unsafe parsing (throws on error)
val json = Json.parseUnsafe("""{"items": [1, 2, 3]}""")
```

### String Interpolators

ZIO Blocks provides compile-time validated string interpolators for JSON:

```scala

// JSON literal with compile-time validation
val person = json"""{"name": "Alice", "age": 30}"""

// With Scala value interpolation
val name = "Bob"
val age = 25
val person2 = json"""{"name": $name, "age": $age}"""

// Path interpolator for navigation
val path = p".users[0].name"
```

The `json"..."` interpolator validates JSON syntax at compile time, catching errors before runtime.

## Type Testing and Access

### Unified Type Operations

The `Json` type provides unified methods for type testing and narrowing with path-dependent return types.
`JsonType` also implements `Json => Boolean`, so it can be used directly as a predicate for filtering.

```scala

val json: Json = Json.parseUnsafe("""{"count": 42}""")

// Type testing with is()
json.is(JsonType.Object)  // true
json.is(JsonType.Array)   // false

// Type narrowing with as() - returns Option[jsonType.Type]
val obj: Option[Json.Object] = json.as(JsonType.Object)  // Some(Json.Object(...))
val arr: Option[Json.Array] = json.as(JsonType.Array)    // None

// Value extraction with unwrap() - returns Option[jsonType.Unwrap]
val str: Json = Json.String("hello")
val strValue: Option[String] = str.unwrap(JsonType.String)  // Some("hello")

val num: Json = Json.Number(42)
val numValue: Option[BigDecimal] = num.unwrap(JsonType.Number)  // Some(42)

// JsonType as predicate - use directly in selection query
val strings = json.select.query(JsonType.String)  // all string values in the JSON tree
```

### Direct Value Access

```scala

val obj = Json.Object("a" -> Json.Number(1))
obj.fields  // Chunk(("a", Json.Number(1)))

val arr = Json.Array(Json.Number(1), Json.Number(2))
arr.elements  // Chunk(Json.Number(1), Json.Number(2))
```

## Navigation

### Simple Navigation

Navigate into objects by key and arrays by index:

```scala

val json = Json.parseUnsafe("""{
  "users": [
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25}
  ]
}""")

// Navigate to a field
val users = json.get("users")  // JsonSelection

// Navigate to an array element
val firstUser = json.get("users")(0)  // JsonSelection

// Chain navigation
val firstName = json.get("users")(0).get("name")  // JsonSelection

// Extract the value
val name: Either[SchemaError, String] = firstName.as[String]  // Right("Alice")
```

### Path-Based Navigation with DynamicOptic

Use `DynamicOptic` paths for complex navigation:

```scala

val json = Json.parseUnsafe("""{
  "company": {
    "employees": [
      {"name": "Alice", "department": "Engineering"},
      {"name": "Bob", "department": "Sales"}
    ]
  }
}""")

// Using path interpolator
val path = p".company.employees[0].name"
val name = json.get(path).as[String]  // Right("Alice")

// Equivalent to chained navigation
val sameName = json.get("company").get("employees")(0).get("name").as[String]
```

## JsonSelection

`JsonSelection` is a fluent wrapper for navigation results, enabling composable chaining:

```scala

val json = Json.parseUnsafe("""{"users": [{"name": "Alice"}]}""")

// Fluent chaining
val result: JsonSelection = json
 .get("users")
 .arrays
 .apply(0)
 .get("name")
 .strings

// Extract values
result.as[String]  // Right("Alice")
result.one         // Right(Json.String("Alice"))
result.isSuccess   // true
result.isFailure   // false
```

### Terminal Operations

```scala

val selection: JsonSelection = ???

// Get single value (exactly one required)
val oneValue: Either[SchemaError, Json] = selection.one
// Get any single value (first of many)
val anyValue: Either[SchemaError, Json] = selection.any
// Get all values condensed (wraps multiple in array)
val allValues: Either[SchemaError, Json] = selection.all

// Get underlying result
val underlying: Option[zio.blocks.chunk.Chunk[Json]] = selection.values
val asChunk: zio.blocks.chunk.Chunk[Json] = selection.toChunk  // empty on error

// Decode to specific types
val asString: Either[SchemaError, String] = selection.as[String]
val asBigDecimal: Either[SchemaError, BigDecimal] = selection.as[BigDecimal]
val asBoolean: Either[SchemaError, Boolean] = selection.as[Boolean]
val asInt: Either[SchemaError, Int] = selection.as[Int]
val asLong: Either[SchemaError, Long] = selection.as[Long]
val asDouble: Either[SchemaError, Double] = selection.as[Double]
```

## Modification

### Setting Values

```scala

val json = Json.parseUnsafe("""{"user": {"name": "Alice", "age": 30}}""")

// Set a value at a path
val updated = json.set(p".user.name", Json.String("Bob"))
// {"user": {"name": "Bob", "age": 30}}

// Set with failure handling
val result = json.setOrFail(p".user.email", Json.String("alice@example.com"))
// Left(SchemaError) - path doesn't exist
```

### Modifying Values

```scala

val json = Json.parseUnsafe("""{"count": 10}""")

// Modify with a function
val incremented = json.modify(p".count") {
  case Json.Number(n) => Json.Number(n + 1)
  case other => other
}
// {"count": 11}

// Modify with failure on missing path
val result = json.modifyOrFail(p".count") {
  case Json.Number(n) => Json.Number(n * 2)
}
// Right({"count": 20})
```

### Deleting Values

```scala

val json = Json.parseUnsafe("""{"a": 1, "b": 2, "c": 3}""")

// Delete a field
val withoutB = json.delete(p".b")
// {"a": 1, "c": 3}

// Delete with failure handling
val result = json.deleteOrFail(p".missing")
// Left(SchemaError) - path doesn't exist
```

### Inserting Values

```scala

val json = Json.parseUnsafe("""{"existing": 1}""")

// Insert a new field
val withNew = json.insert(p".newField", Json.String("value"))
// {"existing": 1, "newField": "value"}
```

## Transformation

### Transform Up (Bottom-Up)

Transform children before parents:

```scala

val json = Json.parseUnsafe("""{"values": [1, 2, 3]}""")

// Double all numbers
val doubled = json.transformUp { (path, value) =>
  value match {
    case Json.Number(n) => Json.Number(n * 2)
    case other => other
  }
}
// {"values": [2, 4, 6]}
```

### Transform Down (Top-Down)

Transform parents before children:

```scala

val json = Json.parseUnsafe("""{"items": [{"x": 1}, {"x": 2}]}""")

// Add a field to all objects
val withId = json.transformDown { (path, value) =>
  value match {
    case Json.Object(fields) if !fields.exists(_._1 == "id") =>
      new Json.Object(("id" -> Json.String(path.toString)) +: fields)
    case other => other
  }
}
```

### Transform Keys

Rename object keys throughout the structure:

```scala

val json = Json.parseUnsafe("""{"user_name": "Alice", "user_age": 30}""")

// Convert snake_case to camelCase
val camelCase = json.transformKeys { (path, key) =>
  key.split("_").zipWithIndex.map {
    case (word, 0) => word
    case (word, _) => word.capitalize
  }.mkString
}
// {"userName": "Alice", "userAge": 30}
```

## Filtering

### Filter Values

Keep only values matching a predicate using `retain`, or remove values using `prune`:

```scala

val json = Json.parseUnsafe("""{"a": 1, "b": null, "c": 2, "d": null}""")

// Remove nulls using prune (removes values matching predicate)
val noNulls = json.prune(_.is(JsonType.Null))
// {"a": 1, "c": 2}

// Keep only numbers using retain (keeps values matching predicate)
val onlyNumbers = json.retain(_.is(JsonType.Number))
// {"a": 1, "c": 2}
```

### Project Paths

Extract only specific paths:

```scala

val json = Json.parseUnsafe("""{
  "user": {"name": "Alice", "email": "alice@example.com", "password": "secret"},
  "metadata": {"created": "2024-01-01"}
}""")

// Keep only specific fields
val projected = json.project(p".user.name", p".user.email")
// {"user": {"name": "Alice", "email": "alice@example.com"}}
```

### Partition

Split based on a predicate:

```scala

val json = Json.parseUnsafe("""{"a": 1, "b": "text", "c": 2}""")

// Separate numbers from non-numbers
val (numbers, nonNumbers) = json.partition(_.is(JsonType.Number))
// numbers: {"a": 1, "c": 2}
// nonNumbers: {"b": "text"}
```

## Folding

### Fold Up (Bottom-Up)

Accumulate values from children to parents:

```scala

val json = Json.parseUnsafe("""{"values": [1, 2, 3, 4, 5]}""")

// Sum all numbers
val sum = json.foldUp(BigDecimal(0)) { (path, value, acc) =>
  value match {
    case n: Json.Number => acc + n.value
    case _ => acc
  }
}
// sum = 15
```

### Fold Down (Top-Down)

Accumulate values from parents to children:

```scala

val json = Json.parseUnsafe("""{"a": {"b": {"c": 1}}}""")

// Collect all paths
val paths = json.foldDown(Vector.empty[DynamicOptic]) { (path, value, acc) =>
  acc :+ path
}
```

## Merging

Combine two JSON values using different strategies:

```scala

val base = Json.parseUnsafe("""{"a": 1, "b": {"x": 10}}""")
val overlay = Json.parseUnsafe("""{"b": {"y": 20}, "c": 3}""")

// Auto strategy (default) - deep merge objects, concat arrays
val merged = base.merge(overlay)
// {"a": 1, "b": {"x": 10, "y": 20}, "c": 3}

// Shallow merge (only top-level)
val shallow = base.merge(overlay, MergeStrategy.Shallow)

// Replace (right wins)
val replaced = base.merge(overlay, MergeStrategy.Replace)
// {"b": {"y": 20}, "c": 3}

// Concat arrays
val concat = base.merge(overlay, MergeStrategy.Concat)

// Custom strategy
val custom = base.merge(overlay, MergeStrategy.Custom { (path, left, right) =>
  // Your merge logic here
  right
})
```

### Merge Strategies

| Strategy | Objects | Arrays | Primitives |
|----------|---------|--------|------------|
| `Auto` | Deep merge | Concatenate | Replace |
| `Deep` | Recursive merge | Concatenate | Replace |
| `Shallow` | Top-level only | Concatenate | Replace |
| `Replace` | Right wins | Right wins | Right wins |
| `Concat` | Merge keys | Concatenate | Replace |
| `Custom(f)` | User-defined | User-defined | User-defined |

## Normalization

Clean up JSON values:

```scala

val json = Json.parseUnsafe("""{
  "z": 1,
  "a": null,
  "m": {"empty": {}},
  "b": 2
}""")

// Sort object keys alphabetically
val sorted = json.sortKeys
// {"a": null, "b": 2, "m": {"empty": {}}, "z": 1}

// Remove null values
val noNulls = json.dropNulls
// {"z": 1, "m": {"empty": {}}, "b": 2}

// Remove empty objects and arrays
val noEmpty = json.dropEmpty
// {"z": 1, "a": null, "b": 2}

// Apply all normalizations
val normalized = json.normalize
// {"b": 2, "z": 1}
```

## Encoding and Decoding

### Built-in Codecs

```scala

// Primitives
Schema[String].jsonCodec
Schema[Int].jsonCodec
Schema[Long].jsonCodec
Schema[Double].jsonCodec
Schema[Boolean].jsonCodec
Schema[BigDecimal].jsonCodec

// Collections
Schema[List[Int]].jsonCodec
Schema[Vector[String]].jsonCodec
Schema[Map[String, Int]].jsonCodec
Schema[Option[String]].jsonCodec

// Java time/util types
Schema[java.time.Instant].jsonCodec
Schema[java.time.LocalDate].jsonCodec
Schema[java.time.ZonedDateTime].jsonCodec
Schema[java.util.UUID].jsonCodec
```

### Encoding/Decoding of Primitives

```scala

// Encode Scala values to Json
val intJson = 42.toJson  // Json.Number(42)
val strJson = "hello".toJson  // Json.String("hello")

// Decode Json to Scala values
val intResult = intJson.as[Int]  // Right(42)
val strResult = strJson.as[String]  // Right("hello")
```

### Encoding/Decoding of Case Classes

For complex types, use Schema-based derivation:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)
val json = person.toJson
val decoded = json.as[Person]
```

### Extension Syntax

When a `Schema` is in scope, you can use convenient extension methods directly on values:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)

// Convert to Json AST
val json = person.toJson  // Json.Object(...)

// Convert directly to JSON string
val jsonString = person.toJsonString  // {"name":"Alice","age":30}

// Convert to UTF-8 bytes
val jsonBytes = person.toJsonBytes  // Array[Byte]

// Parse JSON string back to a typed value
val parsed = """{"name":"Bob","age":25}""".fromJson[Person]  // Right(Person("Bob", 25))

// Parse from bytes
val fromBytes = jsonBytes.fromJson[Person]  // Right(Person("Alice", 30))
```

These extension methods provide a more ergonomic API compared to explicitly creating encoders/decoders.

### Using the `as` Method

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val json = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")

// Decode to a specific type
val person: Either[SchemaError, Person] = json.as[Person]

// Unsafe version (throws on error)
val personUnsafe: Person = json.asUnsafe[Person]
```

## Printing JSON

### Basic Printing

```scala

val json = Json.Object("name" -> Json.String("Alice"), "age" -> Json.Number(30))

// Compact output
val compact: String = json.print
// {"name":"Alice","age":30}
```

### With Writer Config

```scala

val json = Json.Object("name" -> Json.String("Alice"))

// Pretty-printed output (2-space indentation)
val pretty = json.print(WriterConfig.withIndentionStep2)
// {
//   "name": "Alice"
// }

// Custom indentation
val indented4 = json.print(WriterConfig.withIndentionStep(4))
```

### WriterConfig Options

`WriterConfig` controls JSON output formatting:

| Option | Default | Description |
|--------|---------|-------------|
| `indentionStep` | `0` | Spaces per indentation level (0 = compact) |
| `escapeUnicode` | `false` | Escape non-ASCII characters as `\uXXXX` |
| `preferredBufSize` | `32768` | Internal buffer size in bytes |

```scala

// Compact output (default)
val compact = WriterConfig

// Pretty-printed with 2-space indentation
val pretty = WriterConfig.withIndentionStep(2)

// Escape Unicode for ASCII-only output
val ascii = WriterConfig.withEscapeUnicode(true)

// Combine options
val custom = WriterConfig
  .withIndentionStep(2)
  .withEscapeUnicode(true)
  .withPreferredBufSize(65536)
```

### ReaderConfig Options

`ReaderConfig` controls JSON parsing behavior:

| Option | Default | Description |
|--------|---------|-------------|
| `preferredBufSize` | `32768` | Preferred byte buffer size |
| `preferredCharBufSize` | `4096` | Preferred char buffer size for strings |
| `maxBufSize` | `33554432` | Maximum byte buffer size (32MB) |
| `maxCharBufSize` | `4194304` | Maximum char buffer size (4MB) |
| `checkForEndOfInput` | `true` | Error on trailing non-whitespace |

```scala

// Default configuration
val default = ReaderConfig

// Allow trailing content (useful for streaming)
val lenient = ReaderConfig.withCheckForEndOfInput(false)

// Increase buffer sizes for large documents
val largeDoc = ReaderConfig
  .withPreferredBufSize(65536)
  .withPreferredCharBufSize(8192)
```

### To Bytes

```scala

val json = Json.Object("x" -> Json.Number(1))

// As byte array
val bytes: Array[Byte] = json.printBytes
```

## Query Operations

### Query with Predicate

Find all values matching a condition:

```scala

val json = Json.parseUnsafe("""{
  "users": [
    {"name": "Alice", "active": true},
    {"name": "Bob", "active": false},
    {"name": "Charlie", "active": true}
  ]
}""")

// Find all active users using queryBoth on a selection
val activeUsers = json.select.queryBoth { (path, value) =>
  value.get("active").as[Boolean].getOrElse(false)
}
```

### Convert to Key-Value Pairs

Flatten to path-value pairs:

```scala

val json = Json.parseUnsafe("""{"a": {"b": 1, "c": 2}}""")

val pairs: Chunk[(DynamicOptic, Json)] = json.toKV
// Chunk(
//   ($.a.b, Json.Number(1)),
//   ($.a.c, Json.Number(2))
// )
```

## Comparison and Equality

### Object Equality

Objects are compared **order-independently** (keys are compared as sorted sets):

```scala

val obj1 = Json.parseUnsafe("""{"a": 1, "b": 2}""")
val obj2 = Json.parseUnsafe("""{"b": 2, "a": 1}""")

obj1 == obj2  // true (order-independent)
```

### Ordering

JSON values have a total ordering for sorting:

```scala

val values = List(
  Json.String("z"),
  Json.Number(1),
  Json.Null,
  Json.Boolean(true)
)

// Sort by type, then by value
val sorted = values.sortWith((a, b) => a.compare(b) < 0)
// [null, true, 1, "z"]
```

Type ordering: Null < Boolean < Number < String < Array < Object

## JSON Diffing

`JsonDiffer` computes the difference between two JSON values, producing a [`JsonPatch`](./json-patch.md) that transforms the source into the target:

```scala

val source = Json.parseUnsafe("""{"name": "Alice", "age": 30}""")
val target = Json.parseUnsafe("""{"name": "Alice", "age": 31, "active": true}""")

// Compute the diff
val patch: JsonPatch = JsonPatch.diff(source, target)

// The patch describes the minimal changes:
// - NumberDelta for age: 30 -> 31
// - Add field "active": true
```

The differ uses optimal operations:
- **NumberDelta** for numeric changes (stores the delta, not the new value)
- **StringEdit** for string changes when edits are more compact than replacement
- **ArrayEdit** with LCS-based Insert/Delete operations for arrays
- **ObjectEdit** with Add/Remove/Modify operations for objects

## JSON Patching

`JsonPatch` represents a sequence of operations that transform a JSON value. Patches are composable and can be applied with different failure modes:

### Computing and Applying Patches

```scala

val original = Json.parseUnsafe("""{"count": 10, "items": ["a", "b"]}""")
val modified = Json.parseUnsafe("""{"count": 15, "items": ["a", "b", "c"]}""")

// Compute the patch
val patch = JsonPatch.diff(original, modified)

// Apply with default (Strict) mode - fails on any precondition violation
val result1: Either[SchemaError, Json] = patch(original)

// Apply with Lenient mode - skips failing operations
val result2 = patch(original, PatchMode.Lenient)

// Apply with Clobber mode - forces changes on conflicts
val result3 = patch(original, PatchMode.Clobber)
```

### Patch Modes

| Mode | Behavior |
|------|----------|
| `Strict` | Fail immediately on any precondition violation |
| `Lenient` | Skip operations that fail preconditions |
| `Clobber` | Force changes, overwriting on conflicts |

### Composing Patches

```scala

val patch1 = JsonPatch.diff(
  Json.parseUnsafe("""{"x": 1}"""),
  Json.parseUnsafe("""{"x": 2}""")
)

val patch2 = JsonPatch.diff(
  Json.parseUnsafe("""{"x": 2}"""),
  Json.parseUnsafe("""{"x": 2, "y": 3}""")
)

// Compose patches - applies patch1, then patch2
val combined = patch1 ++ patch2

// Apply the combined patch
val result = combined(Json.parseUnsafe("""{"x": 1}"""))
// Right({"x": 2, "y": 3})
```

### Converting to DynamicPatch

`JsonPatch` can be converted to and from `DynamicPatch` for interoperability with the typed patching system:

```scala

val jsonPatch: JsonPatch = ???

// Convert to DynamicPatch
val dynamicPatch: DynamicPatch = jsonPatch.toDynamicPatch

// Convert from DynamicPatch (may fail for unsupported operations)
val restored: Either[SchemaError, JsonPatch] = JsonPatch.fromDynamicPatch(dynamicPatch)
```

## Conversion to DynamicValue

Convert JSON to ZIO Blocks' semi-structured `DynamicValue`:

```scala

val json = Json.parseUnsafe("""{"name": "Alice"}""")

val dynamic: DynamicValue = json.toDynamicValue
```

This enables interoperability with other ZIO Blocks formats (Avro, TOON, etc.).

## Error Handling

### SchemaError

Errors include path information for debugging:

```scala

val json = Json.parseUnsafe("""{"users": [{"name": "Alice"}]}""")

val result = json.get("users")(5).get("name").as[String]
// Left(SchemaError: Index 5 out of bounds at path $.users[5])
```

### Error Properties

```scala

val error: SchemaError = ???

error.message            // Error description
error.errors.head.source // DynamicOptic path to error location
```

## Cross-Platform Support

The `Json` type works across 2 platforms:

- **JVM** - Full functionality
- **Scala.js** - Browser and Node.js

String interpolators use compile-time validation that works on both platforms too.

---

## Lazy

The `Lazy[A]` data type represents a deferred computation that produces a value of type `A`. Unlike Scala's built-in `lazy val`, ZIO Blocks' `Lazy` provides a powerful, monadic abstraction with explicit error handling, memoization, and stack-safe evaluation through trampolining:

```scala
sealed trait Lazy[+A] {
  def force: A
  def isEvaluated: Boolean
  def map[B](f: A => B): Lazy[B]
  def flatMap[B](f: A => Lazy[B]): Lazy[B]
  // ... more operations
}

object Lazy {
  def apply[A](expression: => A): Lazy[A]
  def fail(throwable: Throwable): Lazy[Nothing]
  // ... more constructors
}
```

Once a `Lazy` computation is evaluated, the result is cached and `isEvaluated` becomes `true`. Subsequent calls to `force` return the cached value without re-executing the computation, as long as the computed result is not `null`.

Note: the current implementation uses `null` internally as the sentinel for “not evaluated”. If a `Lazy` computation legitimately returns `null`, it will be recomputed on every `force` call and `isEvaluated` will remain `false`. To benefit from memoization, prefer non-null results (for example, use `Option[A]` instead of returning `null`).
The `force` method uses trampolining (an explicit stack) to evaluate deeply nested `Lazy` computations without risking stack overflow.

## Why Lazy Exists?

During type-class derivation, instances for nested types must be created before they are used. `Lazy` allows the derivation machinery to build a structure of deferred computations, resolving them only when the final instance is forced.

When deriving a type-class instance for a complex type, the derivation machinery needs to:

1. **Traverse the schema tree**: Visit each node (records, variants, sequences, etc.)
2. **Create instances for nested types**: Before creating an instance for a parent type, instances for child types must exist
3. **Handle recursion**: For recursive types, the instance for the recursive reference must be deferred

Here's a simplified view of how `Lazy` enables this:

```scala
// In DerivationBuilder
def transformRecord[A](
  fields: IndexedSeq[Term[F, A, ?]],
  metadata: F[BindingType.Record, A],
  // ...
): Lazy[Reflect.Record[G, A]] = Lazy {
  // Get instances for field types (may trigger evaluation of other Lazy values)
  val fieldInstances = fields.map { field =>
    D.instance(field.value.metadata)  // Returns Lazy[TC[FieldType]]
  }

  // Create the record instance
  val instance = deriver.deriveRecord(fields, /* ... */)

  new Reflect.Record(fields, ..., new BindingInstance(metadata, instance), ...)
}
```

The `BindingInstance` class wraps both a `Binding` and a `Lazy[TC[A]]`:

```scala
final case class BindingInstance[TC[_], T, A](
  binding: Binding[T, A],
  instance: Lazy[TC[A]]
)
```

This allows the derivation to build a complete tree of `Lazy` instances, which are only forced when the final type-class instance is needed.

## Creating Lazy Values

### Basic Construction

Create a `Lazy` value by passing a by-name expression to `Lazy.apply`:

```scala

// The expression is NOT evaluated here
val lazyInt: Lazy[Int] = Lazy {
  println("Computing...")
  42
}

println(lazyInt.isEvaluated)  // false

// Now the expression is evaluated
val result = lazyInt.force    // prints "Computing..."
println(result)               // 42
println(lazyInt.isEvaluated)  // true

// Subsequent calls return the cached value
val result2 = lazyInt.force   // does NOT print "Computing..."
println(result2)              // 42
```

### Creating Failed Lazy Values

Use `Lazy.fail` to create a `Lazy` that will throw an exception when forced:

```scala
val failingLazy: Lazy[Int] = 
  Lazy.fail(new RuntimeException("Something went wrong"))
```

## Transforming Lazy Values

`Lazy` is a monad, supporting `map`, `flatMap`, and other familiar operations.

### map

Transform the value inside a `Lazy` without forcing it:

```scala
val lazyInt: Lazy[Int] = Lazy(42)
val lazyString: Lazy[String] = lazyInt.map(_.toString)

println(lazyString.force)       // "42"
```

### flatMap

Chain `Lazy` computations together:

```scala
val lazyA: Lazy[Int] = Lazy(10)
val lazyB: Lazy[Int] = Lazy(20)

val lazySum: Lazy[Int] = lazyA.flatMap(a => lazyB.map(b => a + b))

println(lazySum.force)  // 30
```

Using for-comprehension syntax:

```scala
val result: Lazy[String] = for {
  x <- Lazy(10)
  y <- Lazy(20)
  z <- Lazy(30)
} yield s"Sum: ${x + y + z}"

println(result.force)  // "Sum: 60"
```

### as

Replace the value with a constant, discarding the original:

```scala
val lazy42: Lazy[Int] = Lazy(42)
val lazyHello: Lazy[String] = lazy42.as("Hello")

println(lazyHello.force)  // "Hello"
```

### unit

Discard the value, keeping only the side effects:

```scala
val lazyWithSideEffect: Lazy[Int] = Lazy {
  println("Side effect!")
  42
}

val lazyUnit: Lazy[Unit] = lazyWithSideEffect.unit
lazyUnit.force  // prints "Side effect!", returns ()
```

### flatten
 
Flatten a nested `Lazy[Lazy[A]]` into `Lazy[A]`:

```scala
val nested: Lazy[Lazy[Int]] = Lazy(Lazy(42))
val flat: Lazy[Int] = nested.flatten

println(flat.force)  // 42
```

### zip

Combine two `Lazy` values into a tuple:

```scala
val lazyA: Lazy[Int] = Lazy(1)
val lazyB: Lazy[String] = Lazy("hello")

val lazyPair: Lazy[(Int, String)] = lazyA.zip(lazyB)

println(lazyPair.force)  // (1, "hello")
```

## Error Handling

### catchAll

Recover from errors by providing an alternative `Lazy`:

```scala
val failing: Lazy[Int] = Lazy(throw new RuntimeException("oops"))
val recovered: Lazy[Int] = failing.catchAll(_ => Lazy(0))

println(recovered.force)  // 0
```

The error handler receives the thrown exception:

```scala
val failing: Lazy[Int] = Lazy(throw new RuntimeException("specific error"))
val handled: Lazy[Int] = failing.catchAll { error =>
  println(s"Caught: ${error.getMessage}")
  Lazy(-1)
}

println(handled.force)  // prints "Caught: specific error", returns -1
```

### ensuring

Run a finalizer regardless of success or failure:

```scala
var resourceClosed = false

val computation: Lazy[Int] = Lazy {
  42
}.ensuring(Lazy {
  resourceClosed = true
})

println(computation.force)  // 42
println(resourceClosed)     // true
```

The finalizer runs even when the main computation fails:

```scala
var finalizerRan = false

val failing: Lazy[Int] = Lazy[Int] {
  throw new RuntimeException("error")
}.ensuring(Lazy {
  finalizerRan = true
})

try {
  failing.force
} catch {
  case _: RuntimeException => ()
}

println(finalizerRan)  // true
```

## Working with Collections

### collectAll

Convert an `IndexedSeq[Lazy[A]]` into a `Lazy[IndexedSeq[A]]`:

```scala
val lazies: IndexedSeq[Lazy[Int]] = IndexedSeq(Lazy(1), Lazy(2), Lazy(3))
val collected: Lazy[IndexedSeq[Int]] = Lazy.collectAll(lazies)

println(collected.force)  // IndexedSeq(1, 2, 3)
```

### foreach

Apply a function that returns `Lazy` to each element of a collection:

```scala
val numbers: IndexedSeq[Int] = IndexedSeq(1, 2, 3)
val doubled: Lazy[IndexedSeq[Int]] = Lazy.foreach(numbers)(n => Lazy(n * 2))

println(doubled.force)  // IndexedSeq(2, 4, 6)
```

## How `force` Works: A Deep Dive

The `force` method is the heart of the `Lazy` data type. It evaluates the deferred computation and returns the result. Understanding how it works is essential for understanding `Lazy`.

### Internal Structure

`Lazy` has three internal components:

1. **`Defer[A](thunk: () => A)`**: Represents a deferred computation. The `thunk` is a function that, when called, produces the value.

2. **`FlatMap[A, B](first: Lazy[A], cont: Cont[A, B])`**: Represents a chained computation where `first` must be evaluated, then its result is passed to the continuation `cont`.

3. **`Cont[A, B](ifSuccess: A => Lazy[B], ifError: Throwable => Lazy[B])`**: A continuation that handles both success and error cases.

Additionally, each `Lazy` instance has mutable fields for memoization:
- `value: Any` - Stores the cached successful result
- `error: Throwable` - Stores the cached exception

### Why Trampolining?

Without trampolining, deeply nested `flatMap` chains would overflow the stack:

```scala
// This would overflow without trampolining
var lazy: Lazy[Int] = Lazy(0)
for (_ <- 1 to 100000) {
  lazy = lazy.flatMap(n => Lazy(n + 1))
}
lazy.force  // Works! Returns 100000
```

The trampolining approach converts recursive calls into an iterative loop with an explicit stack (`List[Cont[Any, Any]]`), consuming constant stack space regardless of nesting depth.

## Comparison with Scala's `lazy val`

| Feature            | `Lazy[A]`                    | Scala `lazy val`             |
|--------------------|------------------------------|------------------------------|
| Monadic operations | Yes (`map`, `flatMap`)       | No                           |
| Error handling     | Yes (`catchAll`, `ensuring`) | No                           |
| Stack safety       | Yes (trampolined `flatMap`)  | N/A (no monadic chaining)    |
| Composable         | Yes                          | Limited                      |

## API Reference

### Constructors

| Method                    | Description                                                         |
|---------------------------|---------------------------------------------------------------------|
| `Lazy(expr: => A)`        | Create a `Lazy` from a by-name expression                          |
| `Lazy.fail(e: Throwable)` | Create a `Lazy` that fails with the given exception                |
| `Lazy.collectAll(values)` | Combine an `IndexedSeq` of `Lazy` into a single `Lazy`             |
| `Lazy.foreach(values)(f)` | Apply a function to each element of an `IndexedSeq`, collecting results |

Note: For other collection types, convert them using `.toIndexedSeq` before calling `Lazy.collectAll` or `Lazy.foreach`.
### Instance Methods

| Method                                       | Description                               |
|----------------------------------------------|-------------------------------------------|
| `force: A`                                          | Evaluate and return the result (memoized) |
| `isEvaluated: Boolean`                              | Check if the value has been computed      |
| `map[B](f: A => B): Lazy[B]`                        | Transform the value                       |
| `flatMap[B](f: A => Lazy[B]): Lazy[B]`              | Chain computations                        |
| `flatten: Lazy[B]`                                  | Flatten nested `Lazy`                     |
| `as[B](b: => B): Lazy[B]`                           | Replace the value with a constant         |
| `unit: Lazy[Unit]`                                  | Discard the value                         |
| `zip[B](that: Lazy[B]): Lazy[(A, B)]`               | Combine with another `Lazy`               |
| `catchAll[B >: A](f: Throwable => Lazy[B]): Lazy[B]` | Handle errors                            |
| `ensuring(finalizer: Lazy[Any]): Lazy[A]`           | Run finalizer on completion               |

### Equality and Hashing

`Lazy` values are compared by forcing both values and comparing the results:

```scala
Lazy(42) == Lazy(42)  // true (forces both)
Lazy(42).hashCode == Lazy(42).hashCode  // true
```

Note: Comparing `Lazy` values will force their evaluation. Also, `hashCode` forces the value and then calls `.hashCode` on it; if a `Lazy` evaluates to `null`, `hashCode` will throw a `NullPointerException`, and repeated equality checks may re-run the thunk because `null` is used internally as the “not yet evaluated” sentinel.

---

## MediaType

`MediaType` is a **type-safe representation of IANA media types** (also known as MIME types). It captures structured metadata about content types including compressibility, binary/text classification, and associated file extensions. 

ZIO Blocks MediaType is designed to be a comprehensive and efficient implementation for handling media types in Scala applications, especially those involving HTTP content negotiation, file handling, and data serialization.

`MediaType`:

- is a zero-dependency data type in the `zio-blocks-mediatype` module
- ships with 2,600+ predefined IANA media types auto-generated from the [mime-db](https://github.com/jshttp/mime-db) database
- provides a `mediaType"..."` string interpolator with compile-time validation
- supports wildcard and case-insensitive matching
- is cross-platform (JVM and Scala.js) and cross-version (Scala 2.13 and 3.x)

```scala
final case class MediaType(
  mainType: String,
  subType: String,
  compressible: Boolean = false,
  binary: Boolean = false,
  fileExtensions: List[String] = Nil,
  extensions: Map[String, String] = Map.empty,
  parameters: Map[String, String] = Map.empty
)
```

## Motivation

Media types are fundamental to content negotiation in HTTP, file type detection, and serialization format selection. Working with raw strings like `"application/json"` is error-prone — typos go undetected, metadata (is it compressible? binary? what file extensions?) must be tracked separately, and matching logic must account for wildcards and case insensitivity.

`MediaType` solves these problems by providing:

- **Structured representation** — main type, subtype, and parameters as distinct fields
- **Rich metadata** — compressibility, binary/text classification, and file extensions baked in
- **Compile-time safety** — the `mediaType"..."` interpolator catches malformed types at compile time
- **Correct matching** — wildcard and case-insensitive matching with parameter awareness

```text
                    MediaType
           ┌──────────┼──────────┐
        mainType    subType   parameters
           │          │          │
     "application"  "json"   Map("charset" -> "utf-8")
           │
     ┌─────┴──────────────────────────────────┐
     │  compressible = true                   │
     │  binary       = false                  │
     │  fileExtensions = List("json", "map")  │
     └────────────────────────────────────────┘
```

A quick example:

```scala

// Compile-time validated media type
val json = mediaType"application/json"

// Look up by file extension
val detected = MediaType.forFileExtension("png")
// Some(MediaType("image", "png", binary = true, ...))

// Wildcard matching
val textAny = mediaType"text/*"
val html    = mediaType"text/html"
textAny.matches(html) // true
```

## Installation

Add the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-mediatype" % "0.0.33"
```

For cross-platform projects (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-mediatype" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

### Direct Construction

We can create a `MediaType` by specifying the main type and subtype directly. All other fields have sensible defaults:

```scala

// Minimal — just mainType and subType
val plain = MediaType("text", "plain")
// compressible = false, binary = false, fileExtensions = Nil, ...

// With all fields
val json = MediaType(
  mainType       = "application",
  subType        = "json",
  compressible   = true,
  binary         = false,
  fileExtensions = List("json", "map"),
  extensions     = Map("source" -> "iana"),
  parameters     = Map("charset" -> "utf-8")
)
```

### Parsing from a String

`MediaType.parse` parses a standard media type string in the format `mainType/subType[; key=value]*`:

```scala

// Simple type
val json: Either[String, MediaType] = MediaType.parse("application/json")
// Right(MediaType("application", "json", compressible = true, ...))

// With parameters
val html = MediaType.parse("text/html; charset=utf-8")
// Right(MediaType("text", "html", ..., parameters = Map("charset" -> "utf-8")))

// Predefined instances are reused — parse returns the same object
val parsed = MediaType.parse("application/json").toOption.get
parsed eq MediaTypes.application.`json` // true (reference equality)

// Invalid input returns Left with an error message
MediaType.parse("")               // Left("Invalid media type: cannot be empty")
MediaType.parse("applicationjson") // Left("Invalid media type: must contain '/' separator")
MediaType.parse("/json")          // Left("Invalid media type: main type cannot be empty")
MediaType.parse("application/")   // Left("Invalid media type: subtype cannot be empty")
```

### Unsafe Parsing

When we are certain the input is valid, `unsafeFromString` returns a `MediaType` directly or throws an `IllegalArgumentException`:

```scala

val json = MediaType.unsafeFromString("application/json")
// MediaType("application", "json", compressible = true, ...)

// Throws IllegalArgumentException for invalid input:
// MediaType.unsafeFromString("invalid")
```

### String Interpolator

The `mediaType"..."` interpolator validates the media type at compile time. Invalid types produce compile errors, not runtime failures:

```scala

val json     = mediaType"application/json"
val htmlUtf8 = mediaType"text/html; charset=utf-8"
val wildcard = mediaType"*/*"
val vendor   = mediaType"text/vnd.api+json"
```

The interpolator reuses predefined instances when available (reference equality with `MediaTypes` constants). It does not support variable interpolation — only literal strings are accepted.

:::note
The `mediaType` interpolator is available after importing `zio.blocks.mediatype._`. Invalid inputs produce clear compile-time errors:

```text
mediaType""                → "Invalid media type: cannot be empty"
mediaType"applicationjson" → "Invalid media type: must contain '/' separator"
mediaType"/json"           → "Invalid media type: main type cannot be empty"
mediaType"application/"    → "Invalid media type: subtype cannot be empty"
```
:::

### File Extension Lookup

`MediaType.forFileExtension` finds a `MediaType` by its associated file extension. The lookup is case-insensitive and strips a leading `.` if present:

```scala

MediaType.forFileExtension("json")  // Some(MediaType("application", "json", ...))
MediaType.forFileExtension(".html") // Some(MediaType("text", "html", ...))
MediaType.forFileExtension("PNG")   // Some(MediaType("image", "png", ...))
MediaType.forFileExtension("jpg")   // Some(MediaType("image", "jpeg", ...))
MediaType.forFileExtension("pdf")   // Some(MediaType("application", "pdf", ...))

MediaType.forFileExtension("xyz123") // None (unknown extension)
MediaType.forFileExtension("")       // None
MediaType.forFileExtension(".")      // None
```

## Predefined Media Types

The `MediaTypes` object contains 2,600+ predefined media type constants auto-generated from the [jshttp/mime-db](https://github.com/jshttp/mime-db) database. Types are organized by main type category:

| Category     | Object                    | Examples                             |
|--------------|---------------------------|--------------------------------------|
| Application  | `MediaTypes.application`  | `json`, `pdf`, `xml`, `octet-stream` |
| Audio        | `MediaTypes.audio`        | `mpeg`, `ogg`, `wav`                 |
| Chemical     | `MediaTypes.chemical`     | `x-cif`, `x-pdb`                     |
| Font         | `MediaTypes.font`         | `woff`, `woff2`, `otf`               |
| Image        | `MediaTypes.image`        | `png`, `jpeg`, `gif`, `svg+xml`      |
| Message      | `MediaTypes.message`      | `rfc822`, `partial`                  |
| Model        | `MediaTypes.model`        | `gltf+json`, `stl`                   |
| Multipart    | `MediaTypes.multipart`    | `form-data`, `mixed`                 |
| Text         | `MediaTypes.text`         | `html`, `css`, `plain`, `csv`        |
| Video        | `MediaTypes.video`        | `mp4`, `webm`, `ogg`                 |
| X-Conference | `MediaTypes.x_conference` | `x-cooltalk`                         |
| X-Shader     | `MediaTypes.x_shader`     | `x-vertex`, `x-fragment`             |
| Wildcard     | `MediaTypes.any`          | `*/*`                                |

### Accessing Predefined Types

Since many subtype names contain special characters (hyphens, dots, plus signs), predefined constants use backtick identifiers:

```scala

// Common application types
val json      = MediaTypes.application.`json`
val pdf       = MediaTypes.application.`pdf`
val xmlType   = MediaTypes.application.`xml`

// Common text types
val html      = MediaTypes.text.`html`
val css       = MediaTypes.text.`css`
val plain     = MediaTypes.text.`plain`

// Common image types
val png       = MediaTypes.image.`png`
val jpeg      = MediaTypes.image.`jpeg`

// Wildcard (matches any type)
val any       = MediaTypes.any
```

### Listing All Types

Each category object has an `all` field returning a `List[MediaType]` of all types in that category. The top-level `allMediaTypes` aggregates every category:

```scala

// All types in a category
val appTypes: List[zio.blocks.mediatype.MediaType] = MediaTypes.application.all

// All 2,600+ predefined types
val everything: List[zio.blocks.mediatype.MediaType] = MediaTypes.allMediaTypes
```

### Predefined Metadata

Each predefined instance comes with rich metadata from the IANA registry:

```scala

val json = MediaTypes.application.`json`
json.compressible   // true
json.binary         // false
json.fileExtensions // List("json", "map")

val jpeg = MediaTypes.image.`jpeg`
jpeg.compressible   // false
jpeg.binary         // true
jpeg.fileExtensions // List("jpg", "jpeg", "jpe")

val html = MediaTypes.text.`html`
html.compressible   // true
html.binary         // false
html.fileExtensions // List("html", "htm", "shtml")
```

## Core Operations

### `fullType`

Returns the complete media type string by combining `mainType` and `subType` with a `/` separator:

```scala

val mt = MediaType("application", "json")
mt.fullType // "application/json"

MediaType("*", "*").fullType  // "*/*"
MediaType("text", "*").fullType // "text/*"
```

### `matches`

Compares two `MediaType` values with support for wildcards, case insensitivity, and parameter matching:

```scala
final case class MediaType(...) {
  def matches(other: MediaType, ignoreParameters: Boolean = false): Boolean
}
```

The matching rules are:

1. **Wildcard matching** — `"*"` in either `mainType` or `subType` matches any value
2. **Case-insensitive** — `"APPLICATION/JSON"` matches `"application/json"`
3. **Parameter subset** — when `ignoreParameters = false` (the default), all parameters in `this` must exist in `other` with matching values (case-insensitive). Extra parameters in `other` are allowed.

```scala

val json    = mediaType"application/json"
val textAll = mediaType"text/*"
val html    = mediaType"text/html"
val any     = mediaType"*/*"

// Exact match
json.matches(json) // true

// Wildcard matching
any.matches(json)      // true — */* matches anything
textAll.matches(html)  // true — text/* matches text/html
textAll.matches(json)  // false — text/* does not match application/json

// Case-insensitive
MediaType("APPLICATION", "JSON").matches(json) // true

// Parameter matching
val htmlUtf8  = MediaType("text", "html", parameters = Map("charset" -> "utf-8"))
val htmlLatin = MediaType("text", "html", parameters = Map("charset" -> "iso-8859-1"))
val htmlFull  = MediaType("text", "html", parameters = Map("charset" -> "utf-8", "boundary" -> "xxx"))

htmlUtf8.matches(htmlLatin)                          // false — charset mismatch
htmlUtf8.matches(htmlLatin, ignoreParameters = true) // true  — parameters ignored
htmlUtf8.matches(htmlFull)                           // true  — subset match (charset matches)
```

### `forFileExtension`

Looks up a `MediaType` by file extension. The lookup is case-insensitive and strips a leading `.` if present:

```scala
object MediaType {
  def forFileExtension(ext: String): Option[MediaType]
}
```

```scala

MediaType.forFileExtension("json")  // Some(MediaType("application", "json", ...))
MediaType.forFileExtension(".html") // Some(MediaType("text", "html", ...))
MediaType.forFileExtension("PNG")   // Some(MediaType("image", "png", ...))
MediaType.forFileExtension("")      // None
MediaType.forFileExtension("xyz")   // None
```

:::tip
`forFileExtension` gives priority to `text/*` types when an extension maps to multiple types. For example, `"js"` maps to `text/javascript` rather than `application/javascript`.
:::

### `parse`

Parses a media type string into a `MediaType`, returning `Left` with an error message for invalid input:

```scala
object MediaType {
  def parse(s: String): Either[String, MediaType]
}
```

When the parsed type matches a predefined instance, that instance is returned (preserving reference equality and all metadata). Parameters from the input string are merged into the result:

```scala

// Returns predefined instance with full metadata
val json = MediaType.parse("application/json")
// Right(MediaType("application", "json", compressible=true, binary=false, ...))

// Parameters are parsed and attached
val result = MediaType.parse("multipart/form-data; boundary=abc; charset=utf-8")
// Right(MediaType(..., parameters = Map("boundary" -> "abc", "charset" -> "utf-8")))

// Unknown types get a fresh instance
val custom = MediaType.parse("custom/x-my-format")
// Right(MediaType("custom", "x-my-format"))

// Malformed parameters (no "=") are silently ignored
val partial = MediaType.parse("text/html; charset=utf-8; malformed")
// Right(MediaType(..., parameters = Map("charset" -> "utf-8")))
```

### `unsafeFromString`

Like `parse`, but throws `IllegalArgumentException` instead of returning `Left`:

```scala
object MediaType {
  def unsafeFromString(s: String): MediaType
}
```

For valid input, it returns the corresponding `MediaType` (reusing predefined instances when possible). For invalid input, it throws an exception with a descriptive message:

```scala

val json = MediaType.unsafeFromString("application/json")

// Throws IllegalArgumentException:
// MediaType.unsafeFromString("not-a-media-type")
```

## Advanced Usage

### Content Negotiation

We can use `matches` with wildcard types to implement HTTP-style content negotiation:

```scala

def negotiate(
  accept: List[MediaType],
  available: List[MediaType]
): Option[MediaType] =
  available.find(avail => accept.exists(_.matches(avail, ignoreParameters = true)))

val accept    = List(mediaType"text/*", mediaType"application/json")
val available = List(mediaType"application/xml", mediaType"application/json", mediaType"text/html")

negotiate(accept, available)
// Some(MediaType("application", "json", ...))
```

### File Type Detection

Combine `forFileExtension` with file path processing to detect content types:

```scala

def detectContentType(filename: String): Option[MediaType] = {
  val ext = filename.lastIndexOf('.') match {
    case -1 => ""
    case i  => filename.substring(i + 1)
  }
  MediaType.forFileExtension(ext)
}

detectContentType("report.pdf")   // Some(MediaType("application", "pdf", ...))
detectContentType("photo.jpg")    // Some(MediaType("image", "jpeg", ...))
detectContentType("data.json")    // Some(MediaType("application", "json", ...))
detectContentType("Makefile")     // None
```

---

## Modifier

`Modifier` is a sealed trait that provides a mechanism to attach metadata and configuration to schema elements. Modifiers serve as annotations for record fields, variant cases, and reflect values, enabling format-specific customization without polluting domain types.

Modifiers are designed to be **pure data** values that can be serialized, making them ideal for runtime introspection and cross-process schema exchange. When deriving schemas, modifiers are collected and attached to the corresponding fields or types, allowing codecs to read and interpret them accordingly. They are extended with `StaticAnnotation` to also support annotation syntax:

```scala
sealed trait Modifier extends StaticAnnotation
object Modifier {
  sealed trait Term    extends Modifier
  // ... term modifiers (transient, rename, alias, config) ...
  sealed trait Reflect extends Modifier
  // ... reflect modifiers (config) ...
}
```

Modifiers can be applied in two ways:

1. **Programmatic API**: Using the `Schema#modifier` and `Schema#modifiers` methods to attach modifiers to the entire schema or, for field-level modifiers, attach them to specific fields using optics when deriving codecs. This approach keeps your domain types clean and allows you to separate schema configuration from your data model:

```scala

// Clean domain type - zero dependencies
case class User(
  id: String,
  name: String,
  cache: Map[String, String] = Map.empty
)

// Modifiers applied separately to schema and codecs
object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema
    .derived[User]
    .modifier(Modifier.config("db.table-name", "users"))

   implicit val jsonCodec: JsonCodec[User] =
     schema
       .deriving(JsonCodecDeriver)
       .modifier(User.name, Modifier.rename("username"))
       .modifier(User.cache, Modifier.transient())
       .derive

  lazy val id   : Lens[User, String]              = $(_.id)
  lazy val name : Lens[User, String]              = $(_.name)
  lazy val cache: Lens[User, Map[String, String]] = $(_.cache)
}
```

In the above example, we derived a JSON codec for `User` and applied the `rename` and `transient` modifiers to the `name` and `cache` fields respectively, while keeping the domain type free of any schema-related annotations. Now when encoding a `User` to JSON, the `name` field will be serialized as `username`, and the `cache` field will be omitted. During decoding, the codec will look for `username` in the input JSON and populate the `name` field accordingly:

```scala
val user = User(
  id = "123",
  name = "Alice",
  cache = Map("lastLogin" -> "2024-06-01T12:00:00Z")
)
val json: String = User.jsonCodec.encodeToString(user)
println(json)
// Prints: {"id":"123","username":"Alice"}
val decodedUser: Either[SchemaError, User] = User.jsonCodec.decode(json)
println(decodedUser)
// Prints: Right(User(123,Alice,Map()))
```

Please note that when deriving codecs, you can access these modifiers programmatically, allowing you to build custom logic based on the presence of certain modifiers. For example, your SQL codec could check for the presence of `db.table-name` in the schema modifiers to determine which table to read from or write to.

2. **Annotation Syntax**: Using the `@` syntax to annotate fields and cases directly in your case classes and sealed traits. These annotations are processed during schema derivation to attach the corresponding modifiers to the schema elements. At runtime, you can access these modifiers through the `Reflect` structure of the schema.

```scala

@Modifier.config("db.table-name", "users")
case class User(
  id: String,
  @Modifier.rename("username") name: String,
  @Modifier.transient() cache: Map[String, String] = Map.empty
)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] =
    Schema.derived[User]

   implicit val jsonCodec: JsonCodec[User] =
     schema
       .derive(JsonCodecDeriver)
}
```

In this example, we applied the same modifiers as in the programmatic example, but using annotation syntax directly on the case class fields. Let's try encoding and decoding a `User` instance:

```scala
val user = User(
  id = "123",
  name = "Alice",
  cache = Map("lastLogin" -> "2024-06-01T12:00:00Z")
)

val json: String = User.jsonCodec.encodeToString(user)
println(json)
// Prints: {"id":"123","username":"Alice"}
val decodedUser: Either[SchemaError, User] = User.jsonCodec.decode(json)
println(decodedUser)
// Prints: Right(User(123,Alice,Map()))
```

## Modifier Hierarchy

Modifiers are organized into two main categories:

1. **Term modifiers** - annotate record fields or variant cases (the data structure elements)
2. **Reflect modifiers** - annotate schemas/reflect values themselves (the metadata about types)

```
Modifier
 ├── Modifier.Term     (annotates record fields and variant cases)
 │    ├── transient()      : exclude from serialization
 │    ├── rename(name)     : change serialized name
 │    ├── alias(name)      : add alternative name
 │    └── config(key, val) : attach key-value metadata
 └── Modifier.Reflect  (annotates reflect values / types)
      └── config(key, val) : attach key-value metadata
```

As you can see, `config` is the only modifier that extends both `Term` and `Reflect`, allowing it to be used on both fields and types.

## Term Modifiers

Term modifiers annotate record fields and variant cases. They are used to control how individual fields or cases are serialized and deserialized, as well as to attach additional metadata that can be interpreted by codecs or other tools.

### transient

The `transient` modifier marks a field as transient, meaning it will be excluded from serialization. This is useful for computed fields, caches, or sensitive data that shouldn't be persisted.

### rename

The `rename` modifier changes the serialized name of a field or variant case. This is useful when the field name in your Scala code differs from the expected name in the serialized format.

```scala

case class Person(
  @Modifier.rename("user_name") name: String,
  @Modifier.rename("user_age") age: Int
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

You can also use `rename` on variant cases to customize the discriminator value:

```scala

sealed trait PaymentMethod

object PaymentMethod {
  @Modifier.rename("credit_card")
  case class CreditCard(number: String, cvv: String) extends PaymentMethod

  @Modifier.rename("bank_transfer")
  case class BankTransfer(iban: String) extends PaymentMethod

  implicit val schema: Schema[PaymentMethod] = Schema.derived
}
```

### alias

The `alias` modifier provides an alternative name for a term during decoding. This is useful for supporting multiple names during schema evolution or data migration.

```scala

case class MyClass(
  @Modifier.rename("NewName")
  @Modifier.alias("OldName")
  @Modifier.alias("LegacyName")
  value: String
)

object MyClass {
  implicit val schema: Schema[MyClass] = Schema.derived
}
```

With this configuration:
- **Encoding** always uses the `rename` value: `"NewName"`
- **Decoding** accepts any of: `"NewName"`, `"OldName"`, or `"LegacyName"`

This pattern is particularly useful when migrating data formats without breaking compatibility with existing data.

### config

The `config` modifier attaches arbitrary key-value metadata to a term (record fields or variant cases) or a type itself. The convention for keys is `<format>.<property>`, allowing format-specific configuration.

```scala

case class Event(
  @Modifier.config("protobuf.field-id", "1") id: Long,
  @Modifier.config("protobuf.field-id", "2") name: String
)

object Event {
  implicit val schema: Schema[Event] = Schema.derived
}
```

The `config` modifier extends both `Term` and `Reflect`, making it usable on both fields and types. We will discuss using `config` on types in the reflect modifiers section below.

## Reflect Modifiers

Reflect modifiers annotate reflect values (types themselves). Currently, only `config` is a reflect modifier.

### config

You can attach configuration to the type itself using the `Schema#modifier` method:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
    .modifier(Modifier.config("db.table-name", "person_table"))
    .modifier(Modifier.config("schema.version", "v2"))
}
```

Or add multiple modifiers at once:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
    .modifiers(
      Seq(
        Modifier.config("db.table-name", "person_table"),
        Modifier.config("schema.version", "v2")
      )
    )
}
```

Or annotate the case class directly:

```scala

@Modifier.config("db.table-name", "person_table")
@Modifier.config("schema.version", "v2")
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

## Programmatic Modifier Access

You can access modifiers programmatically through the `Reflect` structure:

```scala

case class Person(
  @Modifier.rename("full_name") name: String,
  @Modifier.transient cache: String = ""
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Access field modifiers through the reflect
val reflect = Schema[Person].reflect
reflect match {
  case record: Reflect.Record[_, _] =>
    record.fields.foreach { field =>
      println(s"Field: ${field.name}")
      println(s"Modifiers: ${field.modifiers}")
    }
  case _ => ()
}
```

## Built-in Schema Support

All modifier types have built-in `Schema` instances, enabling them to be serialized and deserialized:

```scala

// Schema instances for individual modifiers
Schema[Modifier.transient]
Schema[Modifier.rename]
Schema[Modifier.alias]
Schema[Modifier.config]

// Schema instances for modifier traits
Schema[Modifier.Term]
Schema[Modifier.Reflect]
Schema[Modifier]
```

This means you can serialize modifiers as part of your schema metadata, allowing you to persist and exchange schema information with full modifier details.

[//]: # (## Format Support)
[//]: # ()
[//]: # (Different serialization formats interpret modifiers according to their semantics.)
[//]: # ()
[//]: # (TODO: Add a table comparing how each modifier is supported across formats like JSON, BSON, Avro, Protobuf, etc. For example:)
[//]: # (| Modifier    | JSON                 | BSON                 | Avro              | Protobuf          |)
[//]: # (|-------------|----------------------|----------------------|-------------------|-------------------|)
[//]: # (| `transient` | Field omitted        | Field omitted        | Field omitted     | Field omitted     |)
[//]: # (| `rename`    | Custom field name    | Custom field name    | Custom field name | Custom field name |)
[//]: # (| `alias`     | Accepts alternatives | Accepts alternatives | -                 | -                 |)
[//]: # (| `config`    | Format-specific      | Format-specific      | Format-specific   | Format-specific   |)

## Best Practices

1. **Use `rename` for external APIs**: When integrating with external systems that use different naming conventions (snake_case vs camelCase), use `rename` to match the expected format.

2. **Use `alias` for migrations**: When evolving your data model, add `alias` modifiers to support reading old data while writing with new names.

3. **Use `transient` sparingly**: Only mark fields as transient when they are truly derived or temporary. Remember that transient fields need default values.

4. **Use namespaced keys for `config`**: Follow the `<format>.<property>` convention to avoid conflicts between different formats or tools.

---

## Optics

Optics are a fundamental feature of ZIO Blocks that enable type-safe, composable access and modification of nested data structures. What sets ZIO Blocks apart is its implementation of **reflective optics** — a novel construct that combines the operational capabilities of traditional optics with embedded structural metadata, enabling both data manipulation AND introspection.

:::tip
For a practical walkthrough of building query DSLs with optics, see the [Writing a Query DSL with Reified Optics](../guides/query-dsl-reified-optics.md) guide.
:::

## What Are Optics?

Optics are abstractions that allow you to focus on a specific part of a data structure. They provide a way to **view**, **update**, and **traverse** nested fields in immutable data types without boilerplate code:

Every optic has two type parameters:

```
Optic[S, A]
      │  │
      │  └── Focus: The "little thing" being accessed
      └───── Source: The "big thing" containing the focus
```

The `S` is the source type from which data is accessed or modified. The `A` is the focus type or the target type of the optic.

The terminology comes from physical optics — like using a magnifying glass to focus on a small part of something larger.

## Reflective Optics

[//]: # (Optics are **reified fields, cases, and for-loops**.)

Traditional optics are defined purely by functions — they give you **capabilities** (get/set) but no **knowledge** about the structure being accessed. ZIO Blocks introduces **reflective optics** which embed the structure of both the source and focus as first-class reified schemas:

```
sealed trait Optic[S, A] {
  def source: Reflect.Bound[S]
  def focus: Reflect.Bound[A]
  
  // Other common operations
}
```

The `source` and `focus` methods return `Reflect.Bound` instances that describe the schema of the source and focus types, respectively. This means that every optic carries with it detailed information about the data shapes it operates on.

This enables capabilities that ordinary optics have never had before:

- **Amazing Error Messages** — Know exactly where and why an optic operation failed, because you know which part of the structure was being accessed during the failure
- **DSL Integration** — Optics as first-class values enables us to write query DSLs that can introspect the data model being queried. So we can integrate that DSL with underlying storage systems (SQL, NoSQL, etc.) without losing type safety.

The `OpticCheck` data type is an error reporting mechanism for reflective optics. It captures detailed diagnostic information when optic operations fail, solving a long-standing pain point in traditional optics libraries where failures silently return `None` without explanation. It provides detailed context about where and why the replacement failed. It includes the expected and actual cases, the full optic path, and the actual value encountered.

## The Optic Type Hierarchy

ZIO Blocks provides four primary optic types, each designed for a specific data shape:

1. **`Lens[S, A]`** — Focuses on a single field within a record (case class)
2. **`Prism[S, A <: S]`** — Focuses on a specific case within a sum type (sealed trait/enum)
3. **`Optional[S, A]`** — It behaves like a combination of `Lens` and `Prism`; focuses on a value that may or may not exist. Like a `Lens`, it can focus on a part of a larger structure and (if present) get and set it. Like a `Prism`, the focus may or may not exist.
4. **`Traversal[S, A]`** — Focuses on zero or more elements within a collection (List, Vector, Set, Map, etc.)

```text
                              ┌─────────────────┐ 
                              │   Optic[S, A]   │
                              │   (abstract)    │
                              └────────┬────────┘
                                       │
         ┌───────────────────┬───────────────────┬───────────────────┐
         │                   │                   │                   │         
         ▼                   ▼                   ▼                   ▼         
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   Lens[S, A]    │ │  Prism[S, A]    │ │ Optional[S, A]  │ │Traversal[S, A]  │
│                 │ │   (A <: S)      │ │                 │ │                 │
│ Fields in       │ │ Cases in        │ │ Combinations of │ │ Elements in     │
│ Records         │ │ Enums/Variants  │ │ Lens + Prism    │ │ Collections     │
└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘
```

## Lens

A **Lens** focuses on a single field within a record (case class). It always succeeds because records always have all their fields.

Lens has two primary operations: 

| Method    | Signature     | Description                             |
|-----------|---------------|-----------------------------------------|
| `get`     | `S => A`      | Extract the field value                 |
| `replace` | `(S, A) => S` | Create new structure with updated field |

```scala
sealed trait Lens[S, A] extends Optic[S, A] {
  def get(s: S): A
  def replace(s: S, a: A): S
}
```

Once you have a lens, you can use it to extract or replace the focused field immutably. There are two main approaches to creating lenses: manual construction and automatic macro-based derivation. Let's explore each.

### Manual Lens Construction

To create a lens manually for a field, you can use the following constructor:

```scala
object Lens {
  def apply[S, A](
    source   : Reflect.Record.Bound[S],
    focusTerm: Term.Bound[S, A]
  ): Lens[S, A] = ???
}
```

It takes a `Reflect.Record.Bound[S]` representing the schema of the source type `S`, and a `Term.Bound[S, A]` representing the specific field within `S` that the lens will focus on, and finally returns a lens of type from `S` to `A`.

Assume you have an `Address` case class like this with the schema derived:

```scala
case class Address(street: String, city: String, zipCode: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}
```

You can create a lens to focus on the `street` field like this:

```scala
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
  
  val street: Lens[Address, String] =
    Lens[Address, String](
      Schema[Address].reflect.asRecord.get,
      Schema[String].reflect.asTerm("street")
    )
}
```

Now you can use the lens to get or replace the `street` field:

```scala
val address = Address("123 Main St", "Springfield", "12345")
val streetName: String = Address.street.get(address) // "123 Main St"
val updatedAddress: Address = Address.street.replace(address, "456 Elm St")
// Address("456 Elm St", "Springfield", "12345")
```

### Automatic Macro-Based Lens Derivation

While manual lens construction gives you fine-grained control, ZIO Blocks provides macro-based derivation as the **preferred approach** for creating lenses.

The `optic` macro inside the `CompanionOptics` trait creates a lens using intuitive selector syntax that mirrors standard Scala field access:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  
  val name: Lens[Person, String] = optic(_.name)
  val age : Lens[Person, Int   ] = optic(_.age)
}
```

The macro inspects the selector expression `_.name`, validates that it corresponds to a valid field path, and generates the appropriate lens. This approach is type-safe—the compiler verifies that the field exists and has the correct type.

For nested structures, the `optic` macro can create composed lenses by chaining field accesses:

```scala
case class Address(street: String, city: String)
case class Person(name: String, address: Address)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  
  // Lens directly to the nested street field
  val street: Lens[Person, String] = optic(_.address.street)
}
```

Now you can use the `street` lens to access the nested `street` field directly and update it:

```scala
val person = Person("John", Address("123 Main St", "Springfield"))
val street = Person.street.get(person)
val updatedPerson = Person.street.replace(person, "456 Elm St")
// => Person("John", Address("456 Elm St", "Springfield"))
```

## Prism

A **`Prism[S, A <: S]`** focuses on a specific case within a sum type (sealed trait/enum). It may fail if the value is a different case.

The key operations of a prism are summarized in the table below:

| Method          | Signature                         | Description                                                    |
|-----------------|-----------------------------------|----------------------------------------------------------------|
| `getOption`     | `(S)    => Option[A]`             | Extracts if value matches subtype `A`, otherwise `None`        |
| `getOrFail`     | `(S)    => Either[OpticCheck, A]` | Extracts with detailed error info on type mismatch             |
| `reverseGet`    | `(A)    => S`                     | Upcasts subtype `A` back to supertype `S`                      |
| `replace`       | `(S, A) => S`                     | Replaces if value matches subtype, otherwise returns unchanged |
| `replaceOption` | `(S, A) => Option[S]`             | Replaces if value matches subtype, otherwise `None`            |
| `replaceOrFail` | `(S, A) => Either[OpticCheck, S]` | Replaces with detailed error info on type mismatch             |

Before discussing these operations, let's explore how we can create prisms. Similar to lenses, we have two approaches: manual construction and automatic macro-based derivation.

### Manual Prism Construction

To create a prism for a case, you can use the following constructor:

```scala
object Prism {
  def apply[S, A <: S](
    source   : Reflect.Variant.Bound[S],
    focusTerm: Term.Bound[S, A]
  ): Prism[S, A] = ???
}
```

It takes a `Reflect.Variant.Bound[S]` representing the schema of the sum type `S`, and a `Term.Bound[S, A]` representing the specific case within `S` that the prism will focus on. The result is a prism from `S` to `A`.

Assume you have a `Notification` sealed trait representing different notification types:

```scala

sealed trait Notification

object Notification {
  case class Email(to: String, subject: String, body: String)                    extends Notification
  case class SMS(phoneNumber: String, message: String)                           extends Notification
  case class Push(deviceId: String, title: String, payload: Map[String, String]) extends Notification

  implicit val schema: Schema[Notification] = Schema.derived
}
```

First, we need to define schemas for each case class and then write a prism for each case (here we define a prism for the `Email` case):

```scala

sealed trait Notification

object Notification {
  case class Email(to: String, subject: String, body: String) extends Notification
  object Email {
    implicit val schema: Schema[Email] = Schema.derived
  }
  
  case class SMS(phoneNumber: String, message: String) extends Notification
  object SMS {
    implicit val schema: Schema[SMS] = Schema.derived
  }
  
  case class Push(deviceId: String, title: String, payload: Map[String, String]) extends Notification
  object Push {
    implicit val schema: Schema[Push] = Schema.derived
  }

  implicit val schema: Schema[Notification] = Schema.derived

  val email: Prism[Notification, Email] =
    Prism[Notification, Email](
      Schema[Notification].reflect.asVariant.get,
      Schema[Email].reflect.asTerm("Email")
    )
  
  val sms: Prism[Notification, SMS] =
    Prism[Notification, SMS](
      Schema[Notification].reflect.asVariant.get,
      Schema[SMS].reflect.asTerm("SMS")
    )
  
  val push: Prism[Notification, Push] =
    Prism[Notification, Push](
      Schema[Notification].reflect.asVariant.get,
      Schema[Push].reflect.asTerm("Push")
    )
}
```

### Automatic Macro-Based Prism Derivation

The `optic` macro inside the `CompanionOptics` trait creates a prism using intuitive selector syntax with the `when[CaseType]` method:

```scala

sealed trait Notification

object Notification extends CompanionOptics[Notification] {
  case class Email(to: String, subject: String, body: String)                    extends Notification
  case class SMS(phoneNumber: String, message: String)                           extends Notification
  case class Push(deviceId: String, title: String, payload: Map[String, String]) extends Notification

  implicit val schema: Schema[Notification] = Schema.derived

  // Macro-derived prisms using when[CaseType] syntax
  val email: Prism[Notification, Email] = optic(_.when[Email])
  val sms  : Prism[Notification, SMS]   = optic(_.when[SMS])
  val push : Prism[Notification, Push]  = optic(_.when[Push])
}
```

The macro inspects the selector expression `_.when[Email]`, validates that `Email` is a valid case of the sum type `Notification`, and generates the appropriate prism. This approach is type-safe—the compiler verifies that the case type exists and is a subtype of the sum type.

For nested structures, the `optic` macro can compose prisms with lenses by chaining case selection with field access:

```scala
sealed trait Response
object Response extends CompanionOptics[Response] {
  case class Success(data: Data)    extends Response
  case class Failure(error: String) extends Response
  
  case class Data(id: Int, value: String)
  
  implicit val schema: Schema[Response] = Schema.derived
  
  // Prism to the Success case
  val success: Prism[Response, Success] = optic(_.when[Success])
  
  // Composed optic: Prism + Lens = Optional
  // Focus on the 'value' field inside the Success case
  val successValue: Optional[Response, String] = optic(_.when[Success].data.value)
}
```

When you compose a prism with a lens using the `optic` macro, the result is an `Optional`—an optic that may fail to focus (because the sum type might be a different case) but if it succeeds, always finds the field:

```scala
val response = Response.Success(Response.Data(1, "hello"))

Response.successValue.getOption(response)
// => Some("hello")

Response.successValue.getOption(Response.Failure("error"))
// => None (response is Failure, not Success)
```

### Operations

Now let's explore prism operations. Assume we have some sample notifications:

```scala
// Sample notifications
val emailNotif: Notification = Notification.Email("user@example.com", "Hello", "Welcome!")
// emailNotif: Notification = Email(
//   to = "user@example.com",
//   subject = "Hello",
//   body = "Welcome!"
// )
val smsNotif: Notification   = Notification.SMS("+1234567890", "Your code is 1234")
// smsNotif: Notification = SMS(
//   phoneNumber = "+1234567890",
//   message = "Your code is 1234"
// )
val pushNotif: Notification  = Notification.Push("device-abc", "Alert", Map("action" -> "open"))
// pushNotif: Notification = Push(
//   deviceId = "device-abc",
//   title = "Alert",
//   payload = Map("action" -> "open")
// )
```

1. **`Prism#getOption`** — Extract the case if it matches, otherwise None

```scala
// getOption: Extract the case if it matches, otherwise None
Notification.email.getOption(emailNotif)
// => Some(Email(to = "user@example.com", subject = "Hello", body = "Welcome!"))

Notification.email.getOption(smsNotif)
// => None (smsNotif is an SMS, not an Email)

Notification.sms.getOption(smsNotif)
// Option[SMS] = Some(SMS(phoneNumber = "+1234567890", message = "Your code is 1234"))
```

2. **`Prism#getOrFail`** — Extract with detailed error information on failure:

```text
trait Prism[S, A <: S] {
  def getOrFail(s: S): Either[OpticCheck, A]
}
```

It returns `Right(a)` if the source `s` matches the expected case `A`, or `Left(opticCheck)` containing detailed error information if it does not:

```scala
Notification.email.getOrFail(emailNotif)
// => Right(Email(to = "user@example.com", subject = "Hello", body = "Welcome!"))

Notification.email.getOrFail(pushNotif)
// res5: Either[OpticCheck, Email] = Left(
//   OpticCheck(
//     List(
//       UnexpectedCase(
//         expectedCase = "Email",
//         actualCase = "Push",
//         full = DynamicOptic(ArraySeq(Case("Email"))),
//         prefix = DynamicOptic(ArraySeq(Case("Email"))),
//         actualValue = Push(
//           deviceId = "device-abc",
//           title = "Alert",
//           payload = Map("action" -> "open")
//         )
//       )
//     )
//   )
// )
```

3. **`Prism#reverseGet`** — Upcast a specific case back to the parent sum type:

```text
trait Prism[S, A <: S] {
  def reverseGet(a: A): S
}
```

An example of upcasting an `Email` back to `Notification`:

```scala
val email = Notification.Email("alice@example.com", "Meeting", "See you at 3pm")
val notification: Notification = Notification.email.reverseGet(email)
// => Email("alice@example.com", "Meeting", "See you at 3pm") as Notification
```

4. **`Prism#replace`** — Replace the value if it matches the case:

```scala
trait Prism[S, A <: S] {
  def replace(s: S, a: A): S
}
```

In the following example, we are replacing an existing `Email` notification with a new one:

```scala
val newEmail = Notification.Email("new@example.com", "Updated", "New content")

Notification.email.replace(emailNotif, newEmail)
// => Email("new@example.com", "Updated", "New content")
```

If the original value is NOT the expected case, replace returns unchanged:
```scala
Notification.email.replace(smsNotif, newEmail)
// res7: Notification = SMS(
//   phoneNumber = "+1234567890",
//   message = "Your code is 1234"
// )
// => SMS("+1234567890", "Your code is 1234") (unchanged, it was an SMS)
```

5. **`Prism#replaceOption`** — Replace returning Some on success, None on mismatch:

```scala
trait Prism[S, A <: S] {
  def replaceOption(s: S, a: A): Option[S]
}
```

The following example shows replacing an `Email` notification with a new one, returning `Some` on success and `None` if the original value is not an `Email`:

```scala
Notification.email.replaceOption(emailNotif, newEmail)
// => Some(Email("new@example.com", "Updated", "New content"))

Notification.email.replaceOption(smsNotif, newEmail)
// => None (cannot replace, smsNotif is not an Email)
```

6. **`Prism#replaceOrFail`** — Replace with detailed error on mismatch:

```scala
trait Prism[S, A <: S] {
  def replaceOrFail(s: S, a: A): Either[OpticCheck, S]
}
```

The following example shows replacing an `Email` notification with a new one, returning `Right(newValue)` on success and `Left(opticCheck)` with detailed error information if the original value is not an `Email`:

```scala
Notification.email.replaceOrFail(emailNotif, newEmail)
// => Right(Email("new@example.com", "Updated", "New content"))

Notification.email.replaceOrFail(pushNotif, newEmail)
// res11: Either[OpticCheck, Notification] = Left(
//   OpticCheck(
//     List(
//       UnexpectedCase(
//         expectedCase = "Email",
//         actualCase = "Push",
//         full = DynamicOptic(ArraySeq(Case("Email"))),
//         prefix = DynamicOptic(ArraySeq(Case("Email"))),
//         actualValue = Push(
//           deviceId = "device-abc",
//           title = "Alert",
//           payload = Map("action" -> "open")
//         )
//       )
//     )
//   )
// )
```

## Optional

You can think of **`Optional[S, A]`** as the composition of lenses and prisms — it focuses on a value that may or may not exist. Used for accessing fields through variant types, optional fields, or elements at specific indices.

```scala
sealed trait Optional[S, A] extends Optic[S, A] {
  def getOption(s: S): Option[A]
  def getOrFail(s: S): Either[OpticCheck, A]
  def replace(s: S, a: A): S
  def replaceOption(s: S, a: A): Option[S]
  def replaceOrFail(s: S, a: A): Either[OpticCheck, S]
}
```

| Method          | Signature                         | Description                                                  |
|-----------------|-----------------------------------|--------------------------------------------------------------|
| `getOption`     | `(S)    => Option[A]`             | Extracts the focused value if accessible, otherwise `None`   |
| `getOrFail`     | `(S)    => Either[OpticCheck, A]` | Extracts with detailed error info on failure                 |
| `replace`       | `(S, A) => S`                     | Replaces if accessible, otherwise returns original unchanged |
| `replaceOption` | `(S, A) => Option[S]`             | Replaces if accessible, otherwise `None`                     |
| `replaceOrFail` | `(S, A) => Either[OpticCheck, S]` | Replaces with detailed error info on failure                 |

### Manual Optional Construction

Unlike `Lens` and `Prism` which have direct constructors, `Optional` is primarily constructed through **composition** of other optics, mainly from `Lens` and `Prism`:

```scala
object Optional {
  // Compose Lens with Prism (yields Optional)
  def apply[S, T, A <: T](first: Lens[S, T], second: Prism[T, A]): Optional[S, A]

  // Compose Prism with Lens (yields Optional)
  def apply[S, T <: S, A](first: Prism[S, T], second: Lens[T, A]): Optional[S, A]
}
```

| First         | Second        | Output           |
|---------------|---------------|------------------|
| `Lens[S, T]`  | `Prism[T, A]` | `Optional[S, A]` |
| `Prism[S, T]` | `Lens[T, A]`  | `Optional[S, A]` |

These two basic compositions allow you to build `Optional` optics by combining lenses and prisms in either order. 

In addition, `Optional` can also be composed with other optics to create more complex optionals, all these composition methods are supported via the `Optional.apply` method overloads:

| First              | Second             | Output           |
|--------------------|--------------------|------------------|
| `Lens[S, T]`       | `Optional[T, A]`   | `Optional[S, A]` |
| `Optional[S, T]`   | `Lens[T, A]`       | `Optional[S, A]` |
| ------------------ | ------------------ | ---------------- |
| `Prism[S, T]`      | `Optional[T, A]`   | `Optional[S, A]` |
| `Optional[S, T]`   | `Prism[T, A]`      | `Optional[S, A]` |
| --------------     | ---------------    | ---------------- |
| `Optional[S, T]`   | `Optional[T, A]`   | `Optional[S, A]` |

Beside the above composition methods, `Optional` provides specialized constructors for common patterns such as index-based access in sequences, key-based access in maps, and accessing wrapped types:

| Constructor        | Signature                                                  | Description                             |
|--------------------|------------------------------------------------------------|-----------------------------------------|
| `Optional.at`      | `(Reflect.Sequence.Bound[A, C], Int) => Optional[C[A], A]` | Accesses element at index in a sequence |
| `Optional.atKey`   | `(Reflect.Map.Bound[K, V, M], K) => Optional[M[K, V], V]`  | Accesses value at key in a map          |
| `Optional.wrapped` | `(Reflect.Wrapper.Bound[A, B]) => Optional[A, B]`          | Accesses inner value of a wrapper type  |

#### Example: Composing Lens and Prism

The most common way to manually construct an `Optional` is by composing a `Lens` with a `Prism`. Assume you have a `PaymentMethod` sum type which has multiple cases, and we have written a prism for one of its cases, e.g., `CreditCard`:

```scala

sealed trait PaymentMethod
object PaymentMethod extends CompanionOptics[PaymentMethod] {
  case class CreditCard(number: String, expiry: String) extends PaymentMethod
  case class Cryptocurrency(walletAddress: String) extends PaymentMethod
  
  implicit val schema: Schema[PaymentMethod] = Schema.derived
  
  val creditCard     : Prism[PaymentMethod, CreditCard]     = optic(_.when[CreditCard])
  val cryptocurrency : Prism[PaymentMethod, Cryptocurrency] = optic(_.when[Cryptocurrency])
}
```

And also assume we have a `Customer` record that has a `payment` field of type `PaymentMethod` and we have written a lens for that field:

```scala
case class Customer(name: String, payment: PaymentMethod)
object Customer extends CompanionOptics[Customer] {
  implicit val schema: Schema[Customer] = Schema.derived
  
  val name   : Lens[Customer, String       ] = optic(_.name)
  val payment: Lens[Customer, PaymentMethod] = optic(_.payment)
}
```

We can now compose the `payment` lens with the `creditCard` prism to create an `Optional` that focuses on the `CreditCard` details within a `Customer`:

```scala
// Compose lens and prism manually to get Optional
val creditCard: Optional[Customer, PaymentMethod.CreditCard] =
  Optional(Customer.payment, PaymentMethod.creditCard)
```

#### Example: Index-Based Access

For accessing elements at specific indices in sequences:

```scala

case class Order(id: String, items: List[String])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived[Order]
  
  val items: Lens[Order, List[String]] = optic(_.items)
  
  // Manual Optional for first item
  val firstItem: Optional[Order, String] = {
    val atFirst = Optional.at(
      Schema[List[String]].reflect.asSequence.get,
      index = 0
    )
    Optional(items, atFirst)
  }
}
```

### Automatic Macro-Based Optional Derivation

The `optic` macro inside the `CompanionOptics` trait creates optionals automatically through several syntactic patterns. The macro intelligently determines when an `Optional` is needed based on the path expression.

#### Accessing Inner Values of ADTs

By combining the `.when[Case]` (prism) and `.field-name` (lens) syntax of optic macro, you can create optionals that focus on the inner values of ADTs:

```scala

case class ApiResponse(
  requestId: String,
  timestamp: Option[Long],
  result: Either[String, Int] // Left = error message, Right = status code
)
object ApiResponse extends CompanionOptics[ApiResponse] {
  implicit val schema: Schema[ApiResponse] = Schema.derived[ApiResponse]

  // Optional to the inner Long value (may not exist if None)
  val timestamp: Optional[ApiResponse, Long] = optic(_.timestamp.when[Some[Long]].value)

  // Optional to the Left value (may not exist if Right)
  val errorMessage: Optional[ApiResponse, String] = optic(_.result.when[Left[String, Int]].value)

  // Optional to the Right value (may not exist if Left)
  val statusCode: Optional[ApiResponse, Int] = optic(_.result.when[Right[String, Int]].value)
}
```

Here is another example focusing on sum types:

```scala

sealed trait Response
object Response extends CompanionOptics[Response] {
  case class Success(data: String, code: Int) extends Response
  case class Failure(error: String)           extends Response
  
  implicit val schema: Schema[Response] = Schema.derived
  
  // Prism to the Success case
  val success: Prism[Response, Success] = optic(_.when[Success])
  
  // Optional: Prism + Lens = Optional
  // Focuses on the 'data' field inside Success case
  val successData : Optional[Response, String] = optic(_.when[Success].data)
  val successCode : Optional[Response, Int   ] = optic(_.when[Success].code)
  val failureError: Optional[Response, String] = optic(_.when[Failure].error)
}
```

#### Index-based Access with `.at(index)` Syntax

We can access elements at specific indices in sequences using `.at(index)` syntax in `optic` macro:

```scala

case class OrderItem(sku: String, quantity: Int)
object OrderItem {
  implicit val schema: Schema[OrderItem] = Schema.derived
}

case class Order(id: String, items: List[OrderItem])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived
  
  // Optional for specific indices
  val firstItem : Optional[Order, OrderItem] = optic(_.items.at(0))
  val secondItem: Optional[Order, OrderItem] = optic(_.items.at(1))
  
  // Chain with field access
  val firstItemSku     : Optional[Order, String] = optic(_.items.at(0).sku)
  val firstItemQuantity: Optional[Order, Int   ] = optic(_.items.at(0).quantity)
}
```

Usage:

```scala
val order = Order("ord-1", List(
  OrderItem("SKU-A", 2),
  OrderItem("SKU-B", 1)
))
val emptyOrder = Order("ord-2", List.empty)

Order.firstItem.getOption(order)       // => Some(OrderItem("SKU-A", 2))
Order.secondItem.getOption(order)      // => Some(OrderItem("SKU-B", 1))
Order.firstItem.getOption(emptyOrder)  // => None (empty list)

Order.firstItemSku.getOption(order)    // => Some("SKU-A")
```

#### Key-based Access with `.atKey(key)`

To access values at a specific key, we can use `.atKey(key)` syntax in `optic` macro:

```scala

case class Config(settings: Map[String, String])
object Config extends CompanionOptics[Config] {
  implicit val schema: Schema[Config] = Schema.derived
  
  // Optional for specific keys
  def setting(key: String): Optional[Config, String] = optic(_.settings.atKey(key))
  
  // Pre-defined optionals for common keys
  val hostSetting: Optional[Config, String] = optic(_.settings.atKey("host"))
  val portSetting: Optional[Config, String] = optic(_.settings.atKey("port"))
}
```

Here is how you can use these optionals:

```scala
val config = Config(Map("host" -> "localhost", "port" -> "8080"))

Config.hostSetting.getOption(config)        // => Some("localhost")
Config.setting("timeout").getOption(config) // => None (key not present)
```

#### Wrapper Type Access with `.wrapped[T]` Syntax

To access the inner value of wrapper types (newtypes, opaque types) you can use the `.wrapped[T]` syntax in `optic` macro:

```scala

// Assume Email is a wrapper around String with validation
case class Email private (value: String)
object Email {
  implicit val schema: Schema[Email] = Schema.derived
  
  def apply(s: String): Either[String, Email] =
    if (s.contains("@")) Right(new Email(s)) else Left("Invalid email")
}

case class Contact(name: String, email: Email)
object Contact extends CompanionOptics[Contact] {
  implicit val schema: Schema[Contact] = Schema.derived
  
  // Optional to access the wrapped String inside Email
  val emailString: Optional[Contact, String] = optic(_.email.wrapped[String])
}
```

## Traversal

A **`Traversal[S, A]`** focuses on zero or more elements within a collection (List, Vector, Set, Map, etc.). Unlike `Lens` (exactly one element) or `Optional` (zero or one element), a `Traversal` can target any number of elements simultaneously.

The key methods of `Traversal` are:

| Method         | Signature                                   | Description                                      |
|----------------|---------------------------------------------|--------------------------------------------------|
| `fold`         | `(S)(Z, (Z, A) => Z) => Z`                  | Aggregates all focused values                    |
| `reduceOrFail` | `(S)((A, A) => A) => Either[OpticCheck, A]` | Reduces with error handling                      |
| `modify`       | `(S, A => A)      => S`                     | Applies function to all focused values           |
| `modifyOption` | `(S, A => A)      => Option[S]`             | Modifies if any elements exist, otherwise `None` |
| `modifyOrFail` | `(S, A => A)      => Either[OpticCheck, S]` | Modifies with detailed error info on failure     |

### Manual Traversal Construction

Traversals are primarily constructed through the `Traversal` companion object which provides specialized constructors for different collection types:

#### Sequence Traversals

The two basic constructors for creating traversals over sequences are `seqValues` and `atIndices`:

```scala
object Traversal {
  // Traverse all elements in a sequence (List, Vector, Set, ArraySeq, etc.)
  def seqValues[A, C[_]](seq: Reflect.Sequence.Bound[A, C]): Traversal[C[A], A]

  // Traverse specific indices
  def atIndices[A, C[_]](seq: Reflect.Sequence.Bound[A, C], indices: Seq[Int]): Traversal[C[A], A]
}
```

The `seqValues` method creates a traversal that focuses on all elements in a sequence, while `atIndices` creates a traversal that focuses only on elements at the specified indices. 

There are also convenience methods for common collection types:

```scala
object Traversal {
  def listValues[A]    (reflect: Reflect.Bound[A]): Traversal[List[A],     A]
  def vectorValues[A]  (reflect: Reflect.Bound[A]): Traversal[Vector[A],   A]
  def setValues[A]     (reflect: Reflect.Bound[A]): Traversal[Set[A],      A]
  def arraySeqValues[A](reflect: Reflect.Bound[A]): Traversal[ArraySeq[A], A]
}
```

For example, to create a traversal over all items in a shopping cart represented as a list and quantities as a vector, you can do the following:

```scala

case class ShoppingCart(items: List[String], quantities: Vector[Int])
object ShoppingCart {
  implicit val schema: Schema[ShoppingCart] = Schema.derived

  // Manual traversal for all items in the cart
  val allItems: Traversal[List[String], String] =
    Traversal.listValues(Schema[String].reflect)

  // Manual traversal for all quantities
  val allQuantities: Traversal[Vector[Int], Int] =
    Traversal.vectorValues(Schema[Int].reflect)
}
```

#### Map Traversals

To create traversals over keys or values in a map you can use the following constructors:

```scala
object Traversal {
  // Traverse all keys in a map
  def mapKeys[K, V, M[_, _]](map: Reflect.Map.Bound[K, V, M]): Traversal[M[K, V], K]
  
  // Traverse all values in a map
  def mapValues[K, V, M[_, _]](map: Reflect.Map.Bound[K, V, M]): Traversal[M[K, V], V]
  
  // Traverse values at specific keys
  def atKeys[K, V, M[_, _]](map: Reflect.Map.Bound[K, V, M], keys: Seq[K]): Traversal[M[K, V], V]
}
```

Let's say you have an inventory represented as a map of product names to stock counts. You can create traversals for both keys and values as follows:

```scala

case class Inventory(stock: Map[String, Int])
object Inventory {
  implicit val schema: Schema[Inventory] = Schema.derived
  
  // Manual traversal for all product names (keys)
  val productNames: Traversal[Map[String, Int], String] =
    Traversal.mapKeys(Schema[Map[String, Int]].reflect.asMap.get)
  
  // Manual traversal for all stock counts (values)
  val stockCounts: Traversal[Map[String, Int], Int] =
    Traversal.mapValues(Schema[Map[String, Int]].reflect.asMap.get)
}
```

### Automatic Macro-Based Traversal Derivation

The `optic` macro inside the `CompanionOptics` trait creates traversals using intuitive selector syntax with the `.each`, `.eachKey`, and `.eachValue` methods.

1. Use `.each` to traverse all elements in a sequence (List, Vector, Set, ArraySeq):

```scala

case class Order(id: String, items: List[String], prices: Vector[Double])
object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived
  
  // Traversal over all items
  val allItems: Traversal[Order, String] = optic(_.items.each)
  
  // Traversal over all prices
  val allPrices: Traversal[Order, Double] = optic(_.prices.each)
}
```

2. Use `.eachKey` to traverse all keys and `.eachValue` to traverse all values in a map:

```scala

case class UserScores(scores: Map[String, Int])
object UserScores extends CompanionOptics[UserScores] {
  implicit val schema: Schema[UserScores] = Schema.derived
  
  // Traversal over all user names (keys)
  val allUserNames: Traversal[UserScores, String] = optic(_.scores.eachKey)
  
  // Traversal over all scores (values)
  val allScores: Traversal[UserScores, Int] = optic(_.scores.eachValue)
}
```

3. Use `.atIndices(indices)` to traverse elements at specific indices:

```scala

case class Matrix(rows: Vector[Vector[Int]])
object Matrix extends CompanionOptics[Matrix] {
  implicit val schema: Schema[Matrix] = Schema.derived
  
  // Traverse elements at indices 0, 2, and 4
  val selectedRows: Traversal[Matrix, Vector[Int]] = optic(_.rows.atIndices(0, 2, 4))
}
```

4. Use `.atKeys(keys)` to traverse values at specific keys:

```scala

case class Environment(variables: Map[String, String])
object Environment extends CompanionOptics[Environment] {
  implicit val schema: Schema[Environment] = Schema.derived
  
  // Traverse values for specific environment variables
  val criticalVars: Traversal[Environment, String] = optic(_.variables.atKeys("PATH", "HOME", "USER"))
}
```

Please note that the `optic` macro supports chaining traversals with field access for deep navigation:

```scala

case class LineItem(sku: String, price: Double, quantity: Int)
object LineItem {
  implicit val schema: Schema[LineItem] = Schema.derived
}

case class Invoice(id: String, items: List[LineItem])
object Invoice extends CompanionOptics[Invoice] {
  implicit val schema: Schema[Invoice] = Schema.derived
  
  // Traverse to get all SKUs from all items
  val allSkus: Traversal[Invoice, String] = optic(_.items.each.sku)
  
  // Traverse to get all prices from all items
  val allPrices: Traversal[Invoice, Double] = optic(_.items.each.price)
  
  // Traverse to get all quantities from all items
  val allQuantities: Traversal[Invoice, Int] = optic(_.items.each.quantity)
}
```

### Operations

Let's explore traversal operations with a practical example. Assume we have a `Team` case class with a list of members and a map of scores:

```scala

case class Team(name: String, members: List[String], scores: Map[String, Int])
object Team extends CompanionOptics[Team] {
  implicit val schema: Schema[Team] = Schema.derived[Team]
  
  val allMembers    : Traversal[Team, String] = optic(_.members.each)
  val allScores     : Traversal[Team, Int   ] = optic(_.scores.eachValue)
  val allPlayerNames: Traversal[Team, String] = optic(_.scores.eachKey)
}

val team = Team(
  "Alpha",
  List("Alice", "Bob", "Charlie"),
  Map("Alice" -> 100, "Bob" -> 85, "Charlie" -> 92)
)
```

1. `Traversal#fold` aggregates all focused values:

```scala
// Count all members
Team.allMembers.fold(team)(0, (count, _) => count + 1)
// => 3

// Sum all scores
Team.allScores.fold(team)(0, _ + _)
// => 277

// Concatenate all member names
Team.allMembers.fold(team)("", (acc, name) => if (acc.isEmpty) name else s"$acc, $name")
// => "Alice, Bob, Charlie"
```

2. `Traversal#reduceOrFail` reduces with error handling:

```scala
// Find the maximum score
Team.allScores.reduceOrFail(team)(math.max)
// => Right(100)

// Find the minimum score
Team.allScores.reduceOrFail(team)(math.min)
// => Right(85)

// Attempt to reduce an empty collection
val emptyTeam = Team("Empty", Nil, Map.empty)
Team.allMembers.reduceOrFail(emptyTeam)(_ + _)
// => Left(OpticCheck(...)) with EmptySequence error
```

3. `Traversal#modify` — Apply function to all focused values:

```scala
// Convert all member names to uppercase
Team.allMembers.modify(team, _.toUpperCase)
// => Team("Alpha", List("ALICE", "BOB", "CHARLIE"), Map(...))

// Double all scores
Team.allScores.modify(team, _ * 2)
// => Team("Alpha", List(...), Map("Alice" -> 200, "Bob" -> 170, "Charlie" -> 184))

// Add prefix to all player names (keys)
Team.allPlayerNames.modify(team, name => s"Player: $name")
// => Team("Alpha", List(...), Map("Player: Alice" -> 100, ...))
```

4. `Traversal#modifyOption` — Modify returning Option

```scala
// Modify non-empty collection
Team.allMembers.modifyOption(team, _.toUpperCase)
// => Some(Team("Alpha", List("ALICE", "BOB", "CHARLIE"), Map(...)))

// Modify empty collection
val emptyTeam = Team("Empty", Nil, Map.empty)
Team.allMembers.modifyOption(emptyTeam, _.toUpperCase)
// => None
```

5. `Traversal#modifyOrFail` — Modify with detailed error on failure

```scala
// Successful modification
Team.allMembers.modifyOrFail(team, _.toUpperCase)
// => Right(Team("Alpha", List("ALICE", "BOB", "CHARLIE"), Map(...)))

// Failed modification (empty collection)
val emptyTeam = Team("Empty", Nil, Map.empty)
Team.allMembers.modifyOrFail(emptyTeam, _.toUpperCase)
// => Left(OpticCheck(List(EmptySequence(...))))
```

## Debug-Friendly toString

All optic types (`Lens`, `Prism`, `Optional`, `Traversal`) have a custom `toString` that produces output matching the `optic` macro syntax. This makes debugging easier by showing exactly what path the optic represents:

```scala

case class Person(name: String, address: Address)
case class Address(street: String, city: String)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val street: Lens[Person, String] = optic(_.address.street)
}

println(Person.street)  // Output: Lens(_.address.street)
```

**Examples by optic type:**

| Optic | toString Output |
|-------|-----------------|
| `Lens` for field | `Lens(_.name)` |
| `Lens` for nested field | `Lens(_.address.street)` |
| `Prism` for variant case | `Prism(_.when[CreditCard])` |
| `Optional` combining prism + lens | `Optional(_.when[Success].data)` |
| `Traversal` over sequence | `Traversal(_.items.each)` |
| `Traversal` over map keys | `Traversal(_.metadata.eachKey)` |
| `Traversal` over map values | `Traversal(_.metadata.eachValue)` |

The output mirrors what you would write with the `optic` macro, making it easy to understand and reproduce the optic path.

## Composing Optics

All optics can be composed together to create more complex access paths. All optics that extend the base `Optic` trait support composition via the `apply` method, which takes another optic as an argument and returns a new optic representing the combined access path:
```scala
trait Optic[S, A] {
  def apply[B](that: Lens[A, B]): Optic[S, B]
  def apply[B <: A](that: Prism[A, B]): Optic[S, B]
  def apply[B](that: Optional[A, B]): Optic[S, B]
  def apply[B](that: Traversal[A, B]): Traversal[S, B]
}
```

All optics (`Lens`, `Prism`, `Optional`, `Traversal`) implement the above `apply` methods to support composition with other optics. Additionally, they have `apply` overloads on their companion objects that take two optics as arguments and return the composed optic.

The following table summarizes the composition rules for combining different optic types:

| `this` ↓ / `that` → | **`Lens`**  | **`Prism`** | **`Optional`** | **`Traversal`** |
|---------------------|-------------|-------------|----------------|-----------------|
| **`Lens`**          | `Lens`      | `Optional`  | `Optional`     | `Traversal`     |
| **`Prism`**         | `Optional`  | `Prism`     | `Optional`     | `Traversal`     |
| **`Optional`**      | `Optional`  | `Optional`  | `Optional`     | `Traversal`     |
| **`Traversal`**     | `Traversal` | `Traversal` | `Traversal`    | `Traversal`     |

Here are some important notes regarding optic composition:

1. As the table demonstrates, `Traversal` is the most general optic type, as composing with it always results in a `Traversal`. This behavior occurs because `Traversal` focuses on zero or more elements, and once you have that level of generality, you cannot return to a more specific optic type. This is the **absorption property** encoded in the type system.

2. Using `Optic#apply` instead of the `apply` methods on companion objects allows for a more natural DSL-like syntax for composing optics. For example, instead of writing `Lens(Person.address, Address.postalCode)`, we can write `Person.address(Address.postalCode)`, which reads naturally as "address's postal code".

3. `Optional` is an absorbing optic for `Lens` and `Prism`—once you have partiality in your access path, composing with total optics (`Lens`) or other partial optics (`Prism`, `Optional`) preserves that partiality. `Traversal` is the most general optic, as composing with it always results in a `Traversal`.

### Examples

In this section, we explore the composition of `Lens` with other optics. The composition of other optics follows similar patterns.

#### Composing a Lens with Another Lens

We can chain two lenses to focus deeper into nested structures using the following composition operators:

```scala
object Lens {
  def apply[S, T, A](
    first : Lens[S, T],   // S => T
    second: Lens[T, A]    // T => A
  ): Lens[S, A] = ???     // S => A
}
```

This method takes two lenses: the first from `S` to `T`, and the second from `T` to `A`, and returns a new lens from `S` to `A`.

For example, if we have a `Person` case class that contains an `Address` and we want to create a lens to access the `street` field of the `Address` within `Person`, we can do so as follows:

```scala

case class Address(street: String, city: String)
object Address extends CompanionOptics[Address] { 
  implicit val schema: Schema[Address] = Schema.derived 
  
  val street: Lens[Address, String] =
    Lens[Address, String](
      Schema[Address].reflect.asRecord.get,
      Schema[String].reflect.asTerm("street")
    )
    
  val city: Lens[Address, String] =
    Lens[Address, String](
      Schema[Address].reflect.asRecord.get,
      Schema[String].reflect.asTerm("city")
    )
}
case class Person(name: String, age: Int, address: Address)
object Person {

  implicit val schema: Schema[Person] = Schema.derived[Person]

  val address: Lens[Person, Address] =
    Lens[Person, Address](
      Schema[Person].reflect.asRecord.get,
      Schema[Address].reflect.asTerm("address")
    )
    
  val street: Lens[Person, String] = 
    Lens[Person, Address, String](
      Person.address, // Lens from Person to Address
      Address.street  // Lens from Address to String (street field)
    )
}
```

To make the DSL more convenient, we can use the `Lens#apply` method:

```scala
object Person {
  val street: Lens[Person, String] = Person.address(Address.street)
}
```

The `optic` macro (or its alias `$`) provides a more concise way to derive composed lenses. By extending `CompanionOptics[T]` and using selector syntax, you can derive the same lenses with significantly less boilerplate:

```scala

case class Address(street: String, city: String)
object Address extends CompanionOptics[Address] 
case class Person(name: String, age: Int, address: Address)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  // Simple field lens
  val address: Lens[Person, Address] = optic(_.address)
  
  // Composed lens - directly access nested field via path syntax
  val street: Lens[Person, String] = optic(_.address.street)
  val city  : Lens[Person, String] = optic(_.address.city)
}
```

The `optic` macro inspects the selector path and automatically composes the necessary lenses. The path `_.address.street` is expanded into a composition of `Person → Address` and `Address → String` lenses.

#### Composing a Lens with a Prism

When you compose a `Lens` with a `Prism`, the result is an `Optional`. This makes sense because a `Lens` always succeeds (fields always exist in records), whereas a `Prism` may fail (a value may not match the particular case being targeted). Therefore, the composition may fail, which is exactly what `Optional` represents:

```scala
sealed trait Lens[S, A] extends Optic[S, A] {
  def apply[B <: A](that: Prism[A, B]): Optional[S, B]
}
```

Consider a scenario where you have a record containing a field whose type is a sealed trait (sum type), and you want to focus on a specific case of that sum type.

For instance, suppose we have an `Employee` case class that contains a `ContactInfo` field, where `ContactInfo` is a sealed trait with different cases:

```scala

sealed trait ContactInfo

object ContactInfo {
  case class Email(address: String) extends ContactInfo

  object Email {
    implicit val schema: Schema[Email] = Schema.derived[Email]
    
    val address: Lens[Email, String] =
      Lens[Email, String](
        Schema[Email].reflect.asRecord.get,
        Schema[String].reflect.asTerm("address")
      )
  }

  case class Phone(number: String) extends ContactInfo

  object Phone {
    implicit val schema: Schema[Phone] = Schema.derived[Phone]
  }

  case object NoContact extends ContactInfo

  implicit val schema: Schema[ContactInfo] = Schema.derived[ContactInfo]

  // Prism to focus on the Email case
  lazy val email: Prism[ContactInfo, Email] =
    Prism[ContactInfo, Email](
      Schema[ContactInfo].reflect.asVariant.get,
      Schema[Email].reflect.asTerm("Email")
    )

  // Prism to focus on the Phone case
  lazy val phone: Prism[ContactInfo, Phone] =
    Prism[ContactInfo, Phone](
      Schema[ContactInfo].reflect.asVariant.get,
      Schema[Phone].reflect.asTerm("Phone")
    )
}

case class Employee(name: String, contact: ContactInfo)

object Employee {
  implicit val schema: Schema[Employee] = Schema.derived[Employee]

  // Lens to focus on the contact field
  val contact: Lens[Employee, ContactInfo] =
    Lens[Employee, ContactInfo](
      Schema[Employee].reflect.asRecord.get,
      Schema[ContactInfo].reflect.asTerm("contact")
    )

  // Compose Lens with Prism to get an Optional
  // This focuses on the email address, but only if contact is an Email
  val contactEmail: Optional[Employee, ContactInfo.Email] =
    Employee.contact(ContactInfo.email)
  
  val emailAddress: Optional[Employee, String] =
    Employee.contactEmail(ContactInfo.Email.address)
}
```

The `Employee.contactEmail` is an `Optional[Employee, ContactInfo.Email]` that allows you to access or modify the email contact of an employee, but only if their contact information is indeed an `Email`. If the contact information is a `Phone` or `NoContact`, the operations will fail gracefully, returning `None`:

```scala
val employee1 = Employee("Alice", ContactInfo.Email("alice@example.com"))
// getOption returns Some when the contact is an Email
Employee.contactEmail.getOption(employee1) // => Some(Email("alice@example.com"))

// replace only succeeds if the contact is already an Email
Employee.contactEmail.replaceOption(employee1, ContactInfo.Email("newalice@example.com"))
// => Some(Employee("Alice", Email("newalice@example.com")))

val employee2 = Employee("Bob", ContactInfo.Phone("555-1234"))
// getOption returns None when the contact is NOT an Email
Employee.contactEmail.getOption(employee2) // => None

Employee.contactEmail.replaceOption(employee2, ContactInfo.Email("bob@example.com"))
// => None (Bob's contact is a Phone, not an Email)
```

You can further compose this `Optional` to reach deeper into the structure:

```scala
// Now you can get/set the email address string directly
Employee.emailAddress.getOption(employee1)
// => Some("alice@example.com")

Employee.emailAddress.getOption(employee2)
// => None
```

The `optic` macro supports case selection using the `.when[T]` syntax, which makes composing lenses with prisms much more concise:

```scala

sealed trait ContactInfo

object ContactInfo extends CompanionOptics[ContactInfo] {
  case class  Email(address: String) extends ContactInfo
  case class  Phone(number: String)  extends ContactInfo
  case object NoContact              extends ContactInfo

  implicit val schema: Schema[ContactInfo] = Schema.derived

  // Derive prisms using the optic macro with .when[T] syntax
  val email: Prism[ContactInfo, Email] = optic(_.when[Email])
  val phone: Prism[ContactInfo, Phone] = optic(_.when[Phone])
}

case class Employee(name: String, contact: ContactInfo)

object Employee extends CompanionOptics[Employee] {
  implicit val schema: Schema[Employee] = Schema.derived

  // Simple field lens
  val contact: Lens[Employee, ContactInfo] = optic(_.contact)

  // Compose Lens with Prism using path syntax - result is an Optional
  val contactEmail: Optional[Employee, ContactInfo.Email] = 
    optic(_.contact.when[ContactInfo.Email])
  
  // Chain even deeper to get the email address string
  val emailAddress: Optional[Employee, String] = 
    optic(_.contact.when[ContactInfo.Email].address)
    
  // Similarly for phone
  val phoneNumber: Optional[Employee, String] = 
    optic(_.contact.when[ContactInfo.Phone].number)
}
```

The path `_.contact.when[Email].address` is automatically composed into a `Lens → Prism → Lens` chain, producing an `Optional[Employee, String]` optic.

#### Composing a Lens with an Optional

When you compose a `Lens` with an `Optional`, the result is also an `Optional`. This follows naturally: a `Lens` always succeeds, and an `Optional` may fail; therefore, the composition may fail.

Building on our previous example, suppose we have a `Company` that contains an `Employee`:

```scala
case class Company(name: String, ceo: Employee)
object Company {
  implicit val schema: Schema[Company] = Schema.derived[Company]

  // Lens to focus on the CEO
  val ceo: Lens[Company, Employee] =
    Lens[Company, Employee](
      Schema[Company].reflect.asRecord.get,
      Schema[Employee].reflect.asTerm("ceo")
    )

  // Compose Lens with Optional to get another Optional
  // This chains: Company -> ceo (Lens) -> contactEmail (Optional)
  val ceoEmailContact: Optional[Company, ContactInfo.Email] =
    Company.ceo(Employee.contactEmail)

  // Further composition to get the email address string
  val ceoEmailAddress: Optional[Company, String] =
    Company.ceo(Employee.emailAddress)
}
```

Now you can work with the CEO's email contact through the company:

```scala
val techCorp = Company("TechCorp", Employee("Alice", ContactInfo.Email("alice@tech.com")))
val retailCo = Company("RetailCo", Employee("Bob", ContactInfo.Phone("555-9999")))

// Get the CEO's email contact
Company.ceoEmailContact.getOption(techCorp)
// => Some(Email("alice@tech.com"))

Company.ceoEmailContact.getOption(retailCo)
// => None (Bob's contact is a Phone)

// Get the CEO's email address string
Company.ceoEmailAddress.getOption(techCorp)
// => Some("alice@tech.com")

// Update the CEO's email address
Company.ceoEmailAddress.replaceOption(techCorp, "ceo@tech.com")
// => Some(Company("TechCorp", Employee("Alice", Email("ceo@tech.com"))))

Company.ceoEmailAddress.replaceOption(retailCo, "bob@retail.com")
// => None (cannot replace because Bob does not have an Email contact)
```

This composition pattern is powerful for navigating through complex nested structures where some paths may not be valid for all values.

With the `optic` macro, you can express the entire path in a single selector expression:

```scala

case class Company(name: String, ceo: Employee)
object Company extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Schema.derived

  // Simple field lens
  val ceo: Lens[Company, Employee] = optic(_.ceo)

  // Compose through multiple levels using path syntax
  // Lens (ceo) + Lens (contact) + Prism (when[Email]) = Optional
  val ceoEmailContact: Optional[Company, ContactInfo.Email] =
    optic(_.ceo.contact.when[ContactInfo.Email])

  // Go even deeper to the email address string
  val ceoEmailAddress: Optional[Company, String] =
    optic(_.ceo.contact.when[ContactInfo.Email].address)
    
  // Similarly for phone number
  val ceoPhoneNumber: Optional[Company, String] =
    optic(_.ceo.contact.when[ContactInfo.Phone].number)
}
```

The macro automatically determines the correct output type based on the composition rules: since the path includes a `.when[Email]` prism selector, the result is an `Optional`.

#### Composing a Lens with a Traversal

When you compose a `Lens` with a `Traversal`, the result is a `Traversal`. This combination allows you to:

- First zoom into a specific field of a record (via the `Lens`)
- Then iterate over all elements in a collection at that field (via the `Traversal`)

If you have a record containing a collection field and you want to operate on all elements of that collection, you can use this composition.

For example, let's create a `Department` that has a list of employees, and we want to access all employee names:

```scala
case class Department(name: String, employees: List[Employee])
object Department {
  implicit val schema: Schema[Department] = Schema.derived[Department]

  // Lens to focus on the employees list
  val employees: Lens[Department, List[Employee]] =
    Lens[Department, List[Employee]](
      Schema[Department].reflect.asRecord.get,
      Schema[List[Employee]].reflect.asTerm("employees")
    )

  // Traversal to iterate over all elements in the list
  val eachEmployee: Traversal[List[Employee], Employee] =
    Traversal.listValues(Schema[Employee].reflect)

  // Compose Lens with Traversal
  // This focuses on all employees in the department
  val allEmployees: Traversal[Department, Employee] =
    Department.employees(Department.eachEmployee)
}
```

With this `Traversal`, you can fold over all employees or modify them:

```scala
val engineering = Department(
  "Engineering",
  List(
    Employee("Alice", ContactInfo.Email("alice@company.com")),
    Employee("Bob", ContactInfo.Phone("555-1234")),
    Employee("Charlie", ContactInfo.Email("charlie@company.com"))
  )
)

// Fold to collect all employee names
Department.allEmployees.fold(engineering)(List.empty[String], (acc, emp) => acc :+ emp.name)
// => List("Alice", "Bob", "Charlie")

// Modify all employees (e.g., update their contact to NoContact)
Department.allEmployees.modify(engineering, emp => emp.copy(contact = ContactInfo.NoContact))
// => Department("Engineering", List(
//      Employee("Alice", NoContact),
//      Employee("Bob", NoContact),
//      Employee("Charlie", NoContact)
//    ))
```

You can chain further to go even deeper. For example, to access all employee names in a department:

```scala
object Employee {
  val name: Lens[Employee, String] =
    Lens[Employee, String](
      Schema[Employee].reflect.asRecord.get,
      Schema[String].reflect.asTerm("name")
    )
}

object Department {
  // Chain: Department -> employees (Lens) -> each (Traversal) -> name (Lens)
  // Result type: Traversal[Department, String]
  val allEmployeeNames: Traversal[Department, String] =
    Department.allEmployees(Employee.name)
}

// Fold to get all names
Department.allEmployeeNames.fold(engineering)(List.empty[String], (acc, name) => acc :+ name)
// => List("Alice", "Bob", "Charlie")

// Modify all names (e.g., convert them to uppercase)
Department.allEmployeeNames.modify(engineering, _.toUpperCase)
// => Department("Engineering", List(
//      Employee("ALICE", ...),
//      Employee("BOB", ...),
//      Employee("CHARLIE", ...)
//    ))
```

You can even compose a `Traversal` with a `Prism` to filter elements. For example, to retrieve only the email addresses from employees who have email contacts:

```scala
object Department {
  // Chain: Department -> allEmployees (Traversal) -> contactEmail (Optional)
  // Traversal + Optional = Traversal
  val allEmailContacts: Traversal[Department, ContactInfo.Email] =
    Department.allEmployees(Employee.contactEmail)

  // Further chain to get email address strings
  val allEmailAddresses: Traversal[Department, String] =
    Department.allEmailContacts(ContactInfo.Email.address)
}

// This only folds over employees who have Email contacts
Department.allEmailAddresses.fold(engineering)(List.empty[String], (acc, addr) => acc :+ addr)
// => List("alice@company.com", "charlie@company.com")
// Note: Bob is skipped because he has a Phone contact
```

This example demonstrates the power of optics composition: you can build complex data access paths by combining simple, reusable building blocks, and the type system ensures the resulting optic has the correct semantics (always succeeding, potentially failing, or iterating over multiple values).

The `optic` macro supports collection traversal using the `.each` syntax. This allows you to express traversals over lists, vectors, and other sequences in a concise path expression:

```scala

case class Department(name: String, employees: List[Employee])
object Department extends CompanionOptics[Department] {
  implicit val schema: Schema[Department] = Schema.derived

  // Lens to the employees list
  val employees: Lens[Department, List[Employee]] = optic(_.employees)

  // Traversal over all employees using .each syntax
  val allEmployees: Traversal[Department, Employee] = optic(_.employees.each)

  // Chain deeper: Lens + Traversal + Lens = Traversal
  val allEmployeeNames: Traversal[Department, String] = optic(_.employees.each.name)
  
  // Lens + Traversal + Lens + Prism = Traversal
  val allEmailContacts: Traversal[Department, ContactInfo.Email] = 
    optic(_.employees.each.contact.when[ContactInfo.Email])

  // Go even deeper to get email address strings
  val allEmailAddresses: Traversal[Department, String] = 
    optic(_.employees.each.contact.when[ContactInfo.Email].address)
}
```

The path `_.employees.each.contact.when[Email].address` composes:
1. `employees` — a `Lens` to the `List`
2. `.each` — a `Traversal` over list elements
3. `contact` — a `Lens` to `ContactInfo`
4. `.when[Email]` — a `Prism` to the `Email` case
5. `address` — a `Lens` to the `String`

The result is a `Traversal[Department, String]` that focuses on all email addresses in the department.

For maps, the macro provides `.eachKey` and `.eachValue` for traversing keys and values, respectively:

```scala

case class Inventory(items: Map[String, Int])
object Inventory extends CompanionOptics[Inventory] {
  implicit val schema: Schema[Inventory] = Schema.derived

  // Lens to the items map
  val items: Lens[Inventory, Map[String, Int]] = optic(_.items)

  // Traversal over all keys
  val allItemNames: Traversal[Inventory, String] = optic(_.items.eachKey)
  
  // Traversal over all values
  val allQuantities: Traversal[Inventory, Int] = optic(_.items.eachValue)
}
```

---

## Patching

The Patching system in ZIO Blocks provides a type-safe, serializable way to describe and apply transformations to data structures. Unlike direct mutations or lens-based updates, patches are **first-class values** that can be serialized, transmitted over the network, stored for audit logs, and composed together.

## Overview

A `Patch[S]` represents a sequence of operations that transform a value of type `S`. Because patches use serializable operations and reflective optics for navigation, they enable powerful use cases:

- **Remote Patching** — Send patches over the network to update remote state without transmitting entire objects
- **Audit Logs** — Record patches as a log of changes for compliance, debugging, or undo functionality
- **CRDT-like Operations** — Use commutative operations like `increment` that can be safely applied in any order
- **Optimistic Updates** — Apply patches locally while syncing with a server
- **Schema Evolution** — Patches work with the schema system, adapting as data structures evolve

:::note
For **untyped JSON patching** without a schema, use [`JsonPatch`](./json-patch.md) instead. `JsonPatch` is optimized for diff-and-apply workflows on raw JSON values and provides compact delta representations without requiring typed optics.
:::

```scala

case class Person(name: String, age: Int)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name: Lens[Person, String] = optic(_.name)
  val age: Lens[Person, Int] = optic(_.age)
}

// Create patches using smart constructors
val patch1 = Patch.set(Person.name, "John")
val patch2 = Patch.increment(Person.age, 1)

// Compose patches
val combined = patch1 ++ patch2

// Apply the patch
val jane = Person("Jane", 25)
val result = combined(jane)  // Person("John", 26)
```

## Creating Patches

Patches are created using smart constructors on the `Patch` companion object. Each constructor takes an optic to specify the target location and the operation parameters.

### Setting Values

The `set` operation replaces a value at the specified location:

```scala
case class Address(street: String, city: String, zip: String)
object Address extends CompanionOptics[Address] {
  implicit val schema: Schema[Address] = Schema.derived
  val street: Lens[Address, String] = optic(_.street)
  val city: Lens[Address, String] = optic(_.city)
}

case class Person(name: String, address: Address)
object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived
  val name: Lens[Person, String] = optic(_.name)
  val address: Lens[Person, Address] = optic(_.address)
  val city: Lens[Person, String] = optic(_.address.city)
}

// Set a simple field
val setName = Patch.set(Person.name, "Alice")

// Set a nested field
val setCity = Patch.set(Person.city, "San Francisco")

// Set an entire nested record
val newAddress = Address("123 Main St", "NYC", "10001")
val setAddress = Patch.set(Person.address, newAddress)
```

The `set` operation works with all optic types:
- **Lens** — Sets a single field
- **Optional** — Sets a value if the path exists
- **Traversal** — Sets all matching elements to the same value
- **Prism** — Sets a variant case

### Numeric Increments

The `increment` operation adds a delta to numeric fields. This is a **commutative operation** — applying increments in any order produces the same result, making it ideal for distributed systems:

```scala
case class Counter(count: Int, total: Long, balance: BigDecimal)
object Counter extends CompanionOptics[Counter] {
  implicit val schema: Schema[Counter] = Schema.derived
  val count: Lens[Counter, Int] = optic(_.count)
  val total: Lens[Counter, Long] = optic(_.total)
  val balance: Lens[Counter, BigDecimal] = optic(_.balance)
}

// Increment various numeric types
val addOne = Patch.increment(Counter.count, 1)
val addTen = Patch.increment(Counter.count, 10)
val subtractFive = Patch.increment(Counter.count, -5)

// Works with all numeric types
val addToTotal = Patch.increment(Counter.total, 1000L)
val addToBalance = Patch.increment(Counter.balance, BigDecimal("99.99"))

// Compose multiple increments
val combined = addOne ++ addTen ++ subtractFive
val counter = Counter(0, 0L, BigDecimal(0))
combined(counter)  // Counter(6, 0, 0)
```

Supported numeric types:
- `Int`, `Long`, `Short`, `Byte`
- `Float`, `Double`
- `BigInt`, `BigDecimal`

### Temporal Operations

Patches support temporal arithmetic with `addDuration` and `addPeriod`:

```scala

case class Event(
  timestamp: Instant,
  scheduledDate: LocalDate,
  scheduledTime: LocalDateTime,
  duration: Duration
)
object Event extends CompanionOptics[Event] {
  implicit val schema: Schema[Event] = Schema.derived
  val timestamp: Lens[Event, Instant] = optic(_.timestamp)
  val scheduledDate: Lens[Event, LocalDate] = optic(_.scheduledDate)
  val scheduledTime: Lens[Event, LocalDateTime] = optic(_.scheduledTime)
  val duration: Lens[Event, Duration] = optic(_.duration)
}

// Add duration to an Instant
val postpone1Hour = Patch.addDuration(Event.timestamp, Duration.ofHours(1))

// Add period to a LocalDate
val postpone1Week = Patch.addPeriod(Event.scheduledDate, Period.ofWeeks(1))

// Add both period and duration to a LocalDateTime
val postpone = Patch.addPeriodAndDuration(
  Event.scheduledTime,
  Period.ofDays(1),
  Duration.ofHours(2)
)

// Add duration to a Duration field

val extendDuration = Patch.addDuration(Event.duration, Duration.ofMinutes(30))
```

### String Edits

The `editString` operation applies character-level edits to strings:

```scala
case class Document(content: String)
object Document extends CompanionOptics[Document] {
  implicit val schema: Schema[Document] = Schema.derived
  val content: Lens[Document, String] = optic(_.content)
}

// Insert text at position
val insertHello = Patch.editString(
  Document.content,
  Vector(Patch.StringOp.Insert(0, "Hello "))
)

// Delete characters
val deleteFirst5 = Patch.editString(
  Document.content,
  Vector(Patch.StringOp.Delete(0, 5))
)

// Append text
val appendBang = Patch.editString(
  Document.content,
  Vector(Patch.StringOp.Append("!"))
)

// Replace a range (delete then insert)
val replaceWorld = Patch.editString(
  Document.content,
  Vector(Patch.StringOp.Modify(6, 5, "Universe"))
)

// Combine multiple edits (applied in sequence)
val doc = Document("Hello World")
val edits = Patch.editString(
  Document.content,
  Vector(
    Patch.StringOp.Delete(5, 6),      // "Hello"
    Patch.StringOp.Append(" there!")  // "Hello there!"
  )
)
edits(doc)  // Document("Hello there!")
```

String edit operations:
- `Insert(index, text)` — Insert text at the given position
- `Delete(index, length)` — Delete characters starting at index
- `Append(text)` — Append text to the end
- `Modify(index, length, text)` — Replace a range with new text

## Working with Collections

### Appending Elements

The `append` operation adds elements to the end of a sequence:

```scala
case class TodoList(items: Vector[String])
object TodoList extends CompanionOptics[TodoList] {
  implicit val schema: Schema[TodoList] = Schema.derived
  val items: Lens[TodoList, Vector[String]] = optic(_.items)
}

val addItems = Patch.append(
  TodoList.items,
  Vector("Buy groceries", "Walk the dog")
)

val list = TodoList(Vector("Existing task"))
addItems(list)  // TodoList(Vector("Existing task", "Buy groceries", "Walk the dog"))
```

Works with `Vector`, `List`, `Seq`, `IndexedSeq`, and `LazyList`. Use the appropriate implicit:

```scala

```

### Inserting at Index

The `insertAt` operation inserts elements at a specific position:

```scala

val insertAtStart = Patch.insertAt(
  TodoList.items,
  0,
  Vector("First priority")
)

val insertInMiddle = Patch.insertAt(
  TodoList.items,
  1,
  Vector("Second item", "Third item")
)

val list = TodoList(Vector("A", "B", "C"))
insertInMiddle(list)  // TodoList(Vector("A", "Second item", "Third item", "B", "C"))
```

### Deleting Elements

The `deleteAt` operation removes elements starting at an index:

```scala

// Delete one element at index 1
val deleteOne = Patch.deleteAt(TodoList.items, 1, 1)

// Delete three elements starting at index 0
val deleteThree = Patch.deleteAt(TodoList.items, 0, 3)

val list = TodoList(Vector("A", "B", "C", "D"))
deleteOne(list)  // TodoList(Vector("A", "C", "D"))
```

### Modifying Elements

The `modifyAt` operation applies a nested patch to an element at a specific index:

```scala
case class Task(title: String, priority: Int)
object Task extends CompanionOptics[Task] {
  implicit val schema: Schema[Task] = Schema.derived
  val title: Lens[Task, String] = optic(_.title)
  val priority: Lens[Task, Int] = optic(_.priority)
}

case class Project(tasks: Vector[Task])
object Project extends CompanionOptics[Project] {
  implicit val schema: Schema[Project] = Schema.derived
  val tasks: Lens[Project, Vector[Task]] = optic(_.tasks)
}

// Create a patch for the nested Task type
val increasePriority = Patch.increment(Task.priority, 1)

// Modify the task at index 0
val modifyFirst = Patch.modifyAt(Project.tasks, 0, increasePriority)

val project = Project(Vector(
  Task("Build feature", 1),
  Task("Write tests", 2)
))
modifyFirst(project)
// Project(Vector(Task("Build feature", 2), Task("Write tests", 2)))
```

## Working with Maps

### Adding Keys

The `addKey` operation adds a new key-value pair to a map:

```scala
case class Config(settings: Map[String, Int])
object Config extends CompanionOptics[Config] {
  implicit val schema: Schema[Config] = Schema.derived
  val settings: Lens[Config, Map[String, Int]] = optic(_.settings)
}

val addTimeout = Patch.addKey(Config.settings, "timeout", 30)
val addRetries = Patch.addKey(Config.settings, "retries", 3)

val config = Config(Map("port" -> 8080))
val combined = addTimeout ++ addRetries
combined(config)  // Config(Map("port" -> 8080, "timeout" -> 30, "retries" -> 3))
```

### Removing Keys

The `removeKey` operation removes a key from a map:

```scala
val removePort = Patch.removeKey(Config.settings, "port")

val config = Config(Map("port" -> 8080, "timeout" -> 30))
removePort(config)  // Config(Map("timeout" -> 30))
```

### Modifying Values

The `modifyKey` operation applies a nested patch to the value at a specific key:

```scala
case class UserSettings(preferences: Map[String, Int])
object UserSettings extends CompanionOptics[UserSettings] {
  implicit val schema: Schema[UserSettings] = Schema.derived
  val preferences: Lens[UserSettings, Map[String, Int]] = optic(_.preferences)
}

// Create a patch that increments an Int
val incrementValue: Patch[Int] = {
  implicit val intSchema: Schema[Int] = Schema[Int]
  Patch(
    DynamicPatch.root(Patch.Operation.PrimitiveDelta(Patch.PrimitiveOp.IntDelta(10))),
    intSchema
  )
}

val increaseVolume = Patch.modifyKey(UserSettings.preferences, "volume", incrementValue)

val settings = UserSettings(Map("volume" -> 50, "brightness" -> 80))
increaseVolume(settings)  // UserSettings(Map("volume" -> 60, "brightness" -> 80))
```

## Composing Patches

Patches compose with the `++` operator. The result applies the first patch, then the second:

```scala
case class Account(name: String, balance: Int, active: Boolean)
object Account extends CompanionOptics[Account] {
  implicit val schema: Schema[Account] = Schema.derived
  val name: Lens[Account, String] = optic(_.name)
  val balance: Lens[Account, Int] = optic(_.balance)
  val active: Lens[Account, Boolean] = optic(_.active)
}

val rename = Patch.set(Account.name, "Premium Account")
val deposit = Patch.increment(Account.balance, 100)
val activate = Patch.set(Account.active, true)

// Compose all patches
val upgrade = rename ++ deposit ++ activate

val account = Account("Basic", 50, false)
upgrade(account)  // Account("Premium Account", 150, true)
```

Composition is associative: `(a ++ b) ++ c` equals `a ++ (b ++ c)`.

The empty patch acts as an identity element:

```scala
val empty = Patch.empty[Account]
val patch = Patch.increment(Account.balance, 100)

(patch ++ empty)(account) == patch(account)  // true
(empty ++ patch)(account) == patch(account)  // true
```

## Applying Patches

### Basic Application

The simplest way to apply a patch uses the `apply` method, which uses `Lenient` mode and returns the original value on failure:

```scala
val patch = Patch.increment(Account.balance, 100)
val account = Account("Test", 50, true)

val result: Account = patch(account)  // Account("Test", 150, true)
```

### Application Modes

For more control, use the overload that takes a `PatchMode`:

```scala
val result: Either[SchemaError, Account] = patch(account, PatchMode.Strict)
```

#### PatchMode.Strict

Fails immediately if any operation cannot be applied:

```scala
case class Data(items: Vector[Int])
object Data extends CompanionOptics[Data] {
  implicit val schema: Schema[Data] = Schema.derived
  val items: Lens[Data, Vector[Int]] = optic(_.items)
}

// Try to delete at an invalid index
val badDelete = Patch.deleteAt(Data.items, 10, 1)
val data = Data(Vector(1, 2, 3))

badDelete(data, PatchMode.Strict)
// Left(SchemaError(...index out of bounds...))

badDelete(data, PatchMode.Lenient)
// Right(Data(Vector(1, 2, 3)))  // Operation skipped
```

Use `Strict` when you need to know if every operation succeeded.

#### PatchMode.Lenient

Skips operations that fail preconditions and continues with the rest:

```scala
val patch1 = Patch.deleteAt(Data.items, 10, 1)  // Will fail
val patch2 = Patch.increment(Counter.count, 5)   // Will succeed

val combined = patch1 ++ patch2

combined(data, PatchMode.Lenient)
// Skips the invalid delete, applies the increment
```

Use `Lenient` for best-effort patching where partial success is acceptable.

#### PatchMode.Clobber

Attempts to force operations through, using fallback behaviors:

```scala
// Adding a key that already exists
val addExisting = Patch.addKey(Config.settings, "port", 9000)
val config = Config(Map("port" -> 8080))

addExisting(config, PatchMode.Strict)
// Left(SchemaError(...key already exists...))

addExisting(config, PatchMode.Clobber)
// Right(Config(Map("port" -> 9000)))  // Overwrites existing key
```

Use `Clobber` when you want to force updates regardless of preconditions.

### Option-Based Application

Use `applyOption` for a simpler API that returns `None` on any failure:

```scala
val patch = Patch.increment(Account.balance, 100)
val result: Option[Account] = patch.applyOption(account)
// Some(Account("Test", 150, true))
```

## Diffing Values

The `diff` method computes a minimal patch that transforms one value into another:

```scala
val old = Person("Alice", 25)
val new = Person("Alice", 26)

// Compute the difference
val dynamicOld = Schema[Person].toDynamicValue(old)
val dynamicNew = Schema[Person].toDynamicValue(new)
val patch = dynamicOld.diff(dynamicNew)

// Apply to transform old into new
patch(dynamicOld, PatchMode.Strict)
// Right(dynamicNew)
```

The differ produces minimal patches using type-appropriate operations:
- **Numeric fields** — Uses delta operations (`+1` instead of `set 26`)
- **Strings** — Uses edit operations when more compact than replacement
- **Records** — Only includes changed fields
- **Sequences** — Uses LCS algorithm to compute minimal insert/delete operations
- **Maps** — Produces add/remove/modify operations for changed entries

### Diff Example

```scala
case class User(
  name: String,
  scores: Vector[Int],
  metadata: Map[String, String]
)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val v1 = User(
  "Alice",
  Vector(10, 20, 30),
  Map("level" -> "1", "status" -> "active")
)

val v2 = User(
  "Alice",
  Vector(10, 25, 30, 40),
  Map("level" -> "2", "status" -> "active")
)

val d1 = Schema[User].toDynamicValue(v1)
val d2 = Schema[User].toDynamicValue(v2)
val patch = d1.diff(d2)

// The patch contains:
// - scores: delete at index 1, insert 25 at index 1, append 40
// - metadata["level"]: increment from "1" to "2" (or set if strings)
```

## Serializable Operations

All patch operations are serializable through the schema system. Each operation type has an implicit `Schema` instance:

```scala
// Patches can be converted to/from DynamicValue
val patch = Patch.increment(Account.balance, 100)
val dynamicPatch = Schema[DynamicPatch].toDynamicValue(patch.dynamicPatch)

// Serialize to JSON, Avro, MessagePack, etc.

val json = JsonEncoder.encode(patch.dynamicPatch)
```

This enables storing patches in databases, sending them over APIs, or logging them for audit purposes.

## Operation Reference

### Value Operations

| Operation | Description |
|-----------|-------------|
| `Patch.set(optic, value)` | Set a value at the optic path |
| `Patch.replace(optic, value)` | Alias for `set` |
| `Patch.empty[S]` | Empty patch (identity) |

### Numeric Operations

| Operation | Description |
|-----------|-------------|
| `Patch.increment(optic, delta)` | Add delta to numeric field |

### Temporal Operations

| Operation | Description |
|-----------|-------------|
| `Patch.addDuration(optic, duration)` | Add duration to `Instant` or `Duration` |
| `Patch.addPeriod(optic, period)` | Add period to `LocalDate` or `Period` |
| `Patch.addPeriodAndDuration(optic, period, duration)` | Add both to `LocalDateTime` |

### String Operations

| Operation | Description |
|-----------|-------------|
| `Patch.editString(optic, edits)` | Apply string edit operations |
| `StringOp.Insert(index, text)` | Insert text at position |
| `StringOp.Delete(index, length)` | Delete characters |
| `StringOp.Append(text)` | Append text |
| `StringOp.Modify(index, length, text)` | Replace range |

### Sequence Operations

| Operation | Description |
|-----------|-------------|
| `Patch.append(optic, elements)` | Append elements to end |
| `Patch.insertAt(optic, index, elements)` | Insert at index |
| `Patch.deleteAt(optic, index, count)` | Delete elements |
| `Patch.modifyAt(optic, index, patch)` | Apply nested patch at index |

### Map Operations

| Operation | Description |
|-----------|-------------|
| `Patch.addKey(optic, key, value)` | Add key-value pair |
| `Patch.removeKey(optic, key)` | Remove key |
| `Patch.modifyKey(optic, key, patch)` | Apply nested patch to value |

## Best Practices

1. **Use increment for counters** — Increment operations are commutative and merge-friendly
2. **Prefer fine-grained patches** — Instead of replacing entire records, patch individual fields
3. **Check isEmpty before applying** — Skip no-op patches for efficiency
4. **Use Strict mode for validation** — Catch errors early in development
5. **Use Lenient mode for resilience** — Handle partial updates gracefully in production
6. **Serialize patches for audit** — Store the patch representation, not just before/after snapshots

---

## Reflect

The `Reflect` data type is the foundational data structure underlying ZIO Blocks. While `Schema[A]` is the user-facing API that wraps a `Reflect`, the `Reflect` type itself contains the actual reflective metadata that describes the structure of Scala data types at runtime:

```scala
// Simplified definition of Schema and Reflect
final case class Schema[A](reflect: Reflect.Bound[A])

sealed trait Reflect[F[_, _], A]
object Reflect {
  case class Record   [F[_, _], A](???)             extends Reflect[F, A]
  case class Variant  [F[_, _], A](???)             extends Reflect[F, A]
  case class Sequence [F[_, _], A, C[_]](???)       extends Reflect[F, C[A]]
  case class Map      [F[_, _], K, V, M[_, _]](???) extends Reflect[F, M[K, V]]
  case class Dynamic  [F[_, _]](???)                extends Reflect[F, DynamicValue]
  case class Primitive[F[_, _], A](???)             extends Reflect[F, A]
  case class Wrapper  [F[_, _], A, B](???)          extends Reflect[F, A]
  case class Deferred [F[_, _], A](???)             extends Reflect[F, A]
}
```

The `Reflect` type has two type parameters:

1. **`F[_, _]`** The binding type constructor. When `F = Binding`, the Reflect is "bound" and contains runtime functions for construction/deconstruction. When `F = NoBinding`, the Reflect is "unbound" and contains only pure structural data. 
2. **`A`** The Scala type that this Reflect describes.                                                                         

## Bound and Unbound Reflects

Each of the eight case class nodes of `Reflect` corresponds to a different category of Scala types (records, variants, sequences, maps, primitives, ... types). They may contain a field of type `F[BindingType.X, A]` that holds the runtime binding information for that type. For example the `Record` variant contains a `F[BindingType.Record, A]` instance that holds the `Constructor[A]` and `Deconstructor[A]` functions:

```scala
case class Record[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeName: TypeName[A],
  recordBinding: F[BindingType.Record, A],  // Binding info
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

We have discussed the binding system in detail on [the Binding System page](./binding.md).

This dual-nature design enables schema serialization:

- **Bound Reflect (`Reflect.Bound[A]`)**: Contains runtime bindings (constructors, deconstructors, functions) that allow actual construction and deconstruction of values. This is what `Schema` wraps.
- **Unbound Reflect**: Contains only pure data with no functions or closures. Can be serialized and deserialized identically, enabling transmission of schemas across the wire.

```scala
type Reflect.Bound[A]   = Reflect[Binding, A]
type Reflect.Unbound[A] = Reflect[NoBinding, A]

// Schemas are always bound
final case class Schema[A](reflect: Reflect.Bound[A])
```

## Reflect Nodes

`Reflect` is a sealed trait with eight nodes, each representing a different category of Scala types.

### 1. Record

`Reflect.Record` represents case classes and other product types:

```scala
case class Record[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeName: TypeName[A],
  recordBinding: F[BindingType.Record, A],
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Record] = Nil
) extends Reflect[F, A]
```

**Key Components:**

- **fields**: An indexed sequence of `Term` objects, each describing a field with its name, type, and nested `Reflect`.
- **typeName**: The fully qualified type name including namespace.
- **recordBinding**: Contains the `Constructor[A]` and `Deconstructor[A]` for building and tearing apart values.
- **doc**: Optional documentation.
- **modifiers**: Metadata modifiers for customization.

The following example shows a `Person` case class represented as a `Reflect.Record`:

```scala

case class Person(
  name: String,
  email: String,
  age: Int,
  height: Double,
  weight: Double
)

object Person {
  implicit val schema: Schema[Person] =
    Schema {
      Reflect.Record[Binding, Person](
        fields = Vector(
          Term("name", Schema.string.reflect),
          Term("email", Schema.string.reflect),
          Term("age", Schema.int.reflect),
          Term("height", Schema.double.reflect),
          Term("weight", Schema.double.reflect)
        ),
        typeId = TypeId.of[Person],
        recordBinding = Binding.Record[Person](
          constructor = new Constructor[Person] {
            override def usedRegisters: RegisterOffset =
              RegisterOffset(objects = 2, ints = 1, doubles = 2)
            override def construct(in: Registers, offset: RegisterOffset): Person =
              Person(
                in.getObject(offset).asInstanceOf[String],
                in.getObject(offset + RegisterOffset(objects = 1)).asInstanceOf[String],
                in.getInt(offset + RegisterOffset.Zero),
                in.getDouble(offset + RegisterOffset(ints = 1)),
                in.getDouble(offset + RegisterOffset(ints = 1, doubles = 1))
              )
          },
          deconstructor = new Deconstructor[Person] {
            override def usedRegisters: RegisterOffset =
              RegisterOffset(objects = 2, ints = 1, doubles = 2)
            override def deconstruct(out: Registers, offset: RegisterOffset, in: Person): Unit = {
              out.setObject(offset, in.name)
              out.setObject(offset + RegisterOffset(objects = 1), in.email)
              out.setInt(offset, in.age)
              out.setDouble(offset + RegisterOffset(ints = 1), in.height)
              out.setDouble(offset + RegisterOffset(ints = 1, doubles = 1), in.weight)
            }
          }
        )
      )
    }
}
```

### 2. Variant

`Reflect.Variant` represents sealed traits, enums, and other sum types—data structures that can be one of several cases:

```scala
case class Variant[F[_, _], A](
  cases: IndexedSeq[Term[F, A, ?]],
  typeName: TypeName[A],
  variantBinding: F[BindingType.Variant, A],
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Variant] = Nil
) extends Reflect[F, A]
```

**Key Components:**

- **cases**: An indexed sequence of `Term` objects, each describing one of the possible cases with its name, type, and nested `Reflect`.
- **typeName**: The fully qualified type name including namespace.
- **variantBinding**: Contains the binding information for the variant, such as a discriminator and matcher functions.

The following example shows a `Shape` sealed trait represented as a `Reflect.Variant`. We assume the schema for the `Circle` and `Rectangle` case classes are defined elsewhere:

```scala
sealed trait Shape extends Product with Serializable

object Shape {
   case class Circle(radius: Double) extends Shape
   object Circle {
      implicit val schema: Schema[Circle] = ???
   }
  
   case class Rectangle(width: Double, height: Double) extends Shape
   object Rectangle {
      implicit val schema: Schema[Rectangle] = ???
   }
  
  implicit val schema: Schema[Shape] =
    Schema[Shape] {
      Reflect.Variant[Binding, Shape](
        cases = IndexedSeq(
          Term(
            name = "Circle",
            value = Circle.schema.reflect
          ),
          Term(
            name = "Rectangle",
            value = Rectangle.schema.reflect
          )
        ),
        typeName = TypeName(namespace = Namespace(Seq.empty), name = "Shape"),
        variantBinding = Binding.Variant[Shape](
          discriminator = new Discriminator[Shape] {
            override def discriminate(a: Shape): Int = a match {
              case Circle(_)       => 0
              case Rectangle(_, _) => 1
            }
          },
          matchers = Matchers(
            new Matcher[Circle] {
              override def downcastOrNull(any: Any): Circle =
                any match {
                  case c: Circle => c
                  case _         => null.asInstanceOf[Circle]
                }
            },
            new Matcher[Rectangle] {
              override def downcastOrNull(any: Any): Rectangle =
                any match {
                  case r: Rectangle => r
                  case _            => null.asInstanceOf[Rectangle]
                }
            }
          )
        ),
        doc = Doc("A geometric shape"),
        modifiers = Seq(Modifier.config("protobuf.field-id", "1"))
      )
    }
}
```

### 3. Sequence

`Reflect.Sequence` represents sequential collections like `List`, `Vector`, `Array`, `Set`, and other `Iterable` types:

```scala
case class Sequence[F[_, _], A, C[_]](
  element: Reflect[F, A],
  typeName: TypeName[C[A]],
  seqBinding: F[BindingType.Seq[C], C[A]],
  doc: Doc = Doc.Empty,
  modifiers: scala.Seq[Modifier.Reflect] = Nil
) extends Reflect[F, C[A]]
```

**Key components**:
- **element**: The `Reflect` describing the element type.
- **seqBinding**: Contains the corresponding sequence binding

### 4. Map

`Reflect.Map` represents key-value collections like `Map` etc:

```scala
case class Map[F[_, _], K, V, M[_, _]](
  key: Reflect[F, K],
  value: Reflect[F, V],
  typeName: TypeName[M[K, V]],
  mapBinding: F[BindingType.Map[M], M[K, V]],
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, M[K, V]]
```

**Key Components:**

- **key**: The `Reflect` describing the key type.
- **value**: The `Reflect` describing the value type.
- **mapBinding**: Contains the corresponding map binding.

### 5. Dynamic

`Reflect.Dynamic` represents values whose types are not known at compile time. This is essential for handling JSON payloads, schemaless data, or any scenario where the structure is determined at runtime.

```scala
case class Dynamic[F[_, _]](
  dynamicBinding: F[BindingType.Dynamic, DynamicValue],
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Dynamic] = Nil
) extends Reflect[F, DynamicValue]
```

### 6. Primitive

`Reflect.Primitive` represents primitive and scalar types. This includes numeric types (`Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`), text types (`String`, `Char`), and `Boolean`. It also covers the full range of `java.time` temporal types: instants and dates (`Instant`, `LocalDate`, `LocalDateTime`, `LocalTime`, `OffsetDateTime`, `OffsetTime`, `ZonedDateTime`), durations and periods (`Duration`, `Period`), calendar components (`Year`, `YearMonth`, `Month`, `MonthDay`, `DayOfWeek`), and time zones (`ZoneId`, `ZoneOffset`). Additionally, it supports `Currency`, `UUID`, and `Unit`, and can be extended to support custom primitive types as needed.

```scala
case class Primitive[F[_, _], A](
  primitiveType: PrimitiveType[A],
  typeName: TypeName[A],
  primitiveBinding: F[BindingType.Primitive, A],
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

You can access all built-in primitive types inside companion object of `Reflect` data type, e.g. `Reflect.int`, `Reflect.string`, `Reflect.instant`, etc.

### 7. Wrapper

Modern Scala development often involves creating domain-specific types that add semantic meaning or validation to primitive types:

```scala
// Opaque type in Scala 3
opaque type Age = Int

// Newtype pattern
case class Email(value: String)

// ZIO Prelude newtypes
object UserId extends Newtype[String]
type UserId = UserId.Type
```

Each of these patterns shares a common characteristic: they wrap an underlying type (`Int`, `String`) to create a new type with distinct semantics.

`Reflect.Wrapper` is a specialized node type that models the relationship between a wrapper type and its underlying representation. It provides a unified abstraction for opaque types, newtypes, wrapper classes, and similar patterns where one type wraps another with optional validation logic:

```
case class Wrapper[F[_, _], A, B](
  wrapped: Reflect[F, B],
  typeId: TypeId[A],
  wrapperBinding: F[BindingType.Wrapper[A, B], A],
  doc: Doc = Doc.Empty,
  modifiers: Seq[Modifier.Reflect] = Nil
) extends Reflect[F, A]
```

The `Wrapper` has three type parameters:

- **`F[_, _]`**: The binding type constructor (typically `Binding` for bound schemas)
- **`A`**: The wrapper type (e.g., `Age`, `Email`)
- **`B`**: The underlying/wrapped type (e.g., `Int`, `String`)

So we can say a wrapper of type `Wrapper[F[_, _], A, B]` wraps a type `B` (described by `wrapped: Reflect[F, B]`) to create a new type `A`. 

Assume we have a positive integer type `PosInt` that wraps an `Int` but enforces a validation rule that the value must be non-negative. We can define its schema using `Reflect.Wrapper` as follows:

```scala

case class PosInt private (value: Int) extends AnyVal

object PosInt {
  def apply(value: Int): Either[String, PosInt] =
    if (value >= 0) Right(new PosInt(value))
    else Left("Expected positive value")

  implicit val schema: Schema[PosInt] = Schema(
    Reflect.Wrapper(
      wrapped = Schema[Int].reflect,
      typeId = TypeId.of[PosInt],
      wrapperBinding = Binding.Wrapper[PosInt, Int](
        wrap = v => PosInt(v), 
        unwrap = _.value
      )
    )
  )
}
```

To create schemas for wrapper types, use `transform`:

```scala

case class PosInt private (value: Int) extends AnyVal

object PosInt {
  def unsafeApply(value: Int): PosInt =
    if (value >= 0) new PosInt(value)
    else throw SchemaError.validationFailed("Expected positive value")

  implicit val schema: Schema[PosInt] = 
    Schema[Int].transform(PosInt.unsafeApply, _.value)
}
```

### 8. Deferred

`Reflect.Deferred` introduces laziness to handle recursive and mutually recursive data types. Without deferred evaluation, recursive types would cause infinite loops:

```scala
case class Deferred[F[_, _], A](
  deferred: () => Reflect[F, A]
) extends Reflect[F, A]
```

For example, if we have a recursive data type like below:

```scala
case class Tree(value: Int, children: List[Tree])
```

We can define its schema using `Reflect.Deferred` as follows:

```scala

// Recursive data type
case class Tree(value: Int, children: List[Tree])

object Tree {
  implicit val schema: Schema[Tree] = {
    lazy val treeReflect: Reflect.Bound[Tree] = Reflect.Record[Binding, Tree](
      fields = Vector(
        Schema[Int].reflect.asTerm("value"),
        Reflect.Deferred(() => Schema.list(new Schema(treeReflect)).reflect).asTerm("children")
      ),
      typeId = TypeId.of[Tree],
      recordBinding = Binding.Record(
        constructor = new Constructor[Tree] {
          def usedRegisters: RegisterOffset = RegisterOffset(ints = 1, objects = 1)

          def construct(in: Registers, offset: RegisterOffset): Tree =
            Tree(
              in.getInt(offset),
              in.getObject(offset).asInstanceOf[List[Tree]]
            )
        },
        deconstructor = new Deconstructor[Tree] {
          def usedRegisters: RegisterOffset = RegisterOffset(ints = 1, objects = 1)

          def deconstruct(out: Registers, offset: RegisterOffset, in: Tree): Unit = {
            out.setInt(offset, in.value)
            out.setObject(offset, in.children)
          }
        }
      )
    )

    new Schema(treeReflect)
  }
}
```

## Debug-Friendly toString

`Reflect` types have a custom `toString` that produces a human-readable SDL (Schema Definition Language) format. This makes debugging schemas much easier compared to the default case class output.

```scala

case class Person(name: String, age: Int, address: Address)
case class Address(street: String, city: String)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

println(Schema[Person].reflect)
// Output:
// record Person {
//   name: String
//   age: Int
//   address: record Address {
//     street: String
//     city: String
//   }
// }
```

**Format by Reflect type:**

| Type | Format |
|------|--------|
| `Primitive` | Type name (e.g., `String`, `Int`, `java.time.Instant`) |
| `Record` | `record Name { fields... }` |
| `Variant` | `variant Name { \| Case1 \| Case2... }` |
| `Sequence` | `sequence List[Element]` or multiline for complex elements |
| `Map` | `map Map[Key, Value]` or multiline for complex types |
| `Wrapper` | `wrapper Name(underlying)` |
| `Deferred` | `deferred => TypeName` (breaks recursive cycles) |
| `Dynamic` | `DynamicValue` |

**Variant example:**

```scala
sealed trait PaymentMethod
case object Cash extends PaymentMethod
case class CreditCard(number: String, cvv: String) extends PaymentMethod

// toString output:
// variant PaymentMethod {
//   | Cash
//   | CreditCard(
//       number: String,
//       cvv: String
//     )
// }
```

**Recursive type example:**

```scala
case class Tree(value: Int, children: List[Tree])

// toString output:
// record Tree {
//   value: Int
//   children: sequence List[
//     deferred => Tree
//   ]
// }
```

## Auto-Derivation

While you can manually construct `Reflect` instances as shown in the examples above, ZIO Blocks provides powerful auto-derivation capabilities that can automatically generate `Schema` instances (and thus `Reflect` instances) for most Scala types using macros and implicit resolution.

The auto-derivation mechanism inspects the structure of your data types at compile time and generates the appropriate `Reflect` representation, including nested types, collections, options, and more.

To leverage auto-derivation, simply define an implicit `Schema` for your type using `Schema.derived`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

The above will automatically generate a `Reflect.Record` for the `Person` case class, including fields for `name` and `age`, along with the necessary bindings for construction and deconstruction. The same applies to more complex types, including variants, collections, and recursive structures.

---

## Register System

The register system is one of the key innovations in ZIO Blocks (ZIO Schema 2) that enables **zero-allocation, box-free construction and deconstruction** of data types. 

## The Problem: Boxing/Unboxing Overheads

When building generic abstractions over data types (like serialization libraries), you need to describe all possible constructions and deconstructions uniformly. The traditional approach uses tuples and boxed primitives. For example assume we have a simple record data type:

```scala
case class Person(name: String, age: Int)
```

A traditional library might represent it as tuple when serializing/deserializing:

```scala
trait Tuple
case class Tuple2[A, B]( _1: A, _2: B)          extends Tuple
case class Tuple3[A, B, C](_1: A, _2: B, _3: C) extends Tuple
// ...
```

Tuple is generic data structure that can hold values of any type. So serializing `Person` would involve converting it to/from `Tuple2[String, Int]`:

```scala
case class Person(name: String, age: Int)
val person = Person("john", 42)

// Generic construction via tuples:
val tuple: (String, Int) = ("john", 42)  // Tuple2[String, Int]
```

The problem is that `Int` is a primitive type, and in order to fit it into a tuple, it must be **boxed** into `java.lang.Integer`. Why? Because tuples can only hold references to objects, not raw primitive values. They are generic containers that work uniformly for any type.

So the actual memory representation of the tuple looks like this:

```text
Stack:                          Heap:

                     
                     ┌────────────────────────┐      ┌─────────────────┐
┌─────────────┐      │ Tuple2 object          │      │ String "john"   │
│ tuple (ref) │─────▶│  _1: ────────────────────────▶│  char[]/byte[]: ───────▶[j][o][h][n]
└─────────────       │  _2: ──────┐           │      └─────────────────┘
                     └────────────────────────┘      
                                  │
                                  ▼
                          ┌─────────────────┐
                          │ Integer object  │
                          │  value: 42      │  ← 4 bytes for int
                          └─────────────────┘
```

This boxing creates significant runtime overhead because:

1. **Primitive boxing**: Values like `Int`, `Long`, `Double` must be wrapped in heap-allocated objects (`java.lang.Integer`, etc.)
2. **Tuple allocation**: All constructor arguments get wrapped in tuple objects. This creates extra allocations for every construction/deconstruction.
3. **Garbage collection pressure**: Each serialization/deserialization creates temporary objects

## The Solution: Register-Based Architecture

ZIO Blocks introduces a novel register-based design that completely eliminates tupling and boxing:

> "Zero allocation, zero boxing, construction and deconstruction of records. It doesn't get faster than this. You can't make generic code in Scala faster than what's been done here."
> — John De Goes, LambdaConf 2025

Instead of tuples, ZIO Blocks uses the `Registers` data structure, which contains:

1. A **byte array** for storing primitives (Int, Long, Double, Float, Boolean, Byte, Char, Short)
2. An **object array** for storing references to heap-allocated objects (AnyRef, which is the supertype of all reference types in Scala including String, custom classes, etc.)

This classification determines where values are stored in the `Registers` data structure. All primitive types use the same `bytes` register to store raw values, and all reference types use the same `objects` register to store references:

```scala
// Conceptual structure of Registers
class Registers {
  var bytes: Array[Byte]     = new Array[Byte](byteArrayLength)      // Stores all primitives efficiently
  var objects: Array[AnyRef] = new Array[AnyRef](objectArrayLength)  // Stores object references
}
```

The `Registers` class is a mutable data structure that serves as an intermediate buffer for **construction** and **deconstruction** operations. So it has methods to set and get values for each primitive type and for object references:

```scala
class Registers {
  private var bytes: Array[Byte]     = new Array[Byte](byteArrayLength)
  private var objects: Array[AnyRef] = new Array[AnyRef](objectArrayLength)

  // Methods to get/set primitive values in byte array (getInt/setInt, getBoolean/setBoolean, etc.)
  def getInt(offset: RegisterOffset): Int = ???
  def setInt(offset: RegisterOffset, value: Int): Unit = ???
  
  def setBoolean(offset: RegisterOffset, value: Boolean): Unit = ???
  def getBoolean(offset: RegisterOffset): Boolean = ???

  // Two methods to get/set object references in object array
  def getObject(offset: RegisterOffset): AnyRef = ???
  def setObject(offset: RegisterOffset, value: AnyRef): Unit = ???
}
```

This design allows primitives to be stored directly in their native binary representation without boxing, while objects are stored as simple references. So, when you use registers, the library adds zero overhead to construction and deconstruction — no tuples and no boxing of primitives.

```
┌─────────────────────────────────────────────────────────────────┐
│                        Registers                                │
├─────────────────────────────────────────────────────────────────┤
│  Byte Array (primitives)          │  Object Array (references)  │
│  ┌───┬───┬───┬───┬───┬───┬───┐    │  ┌───────┬───────┬───────┐  │
│  │ B │ B │ S │ S │ I │ I │...│    │  │ Obj0  │ Obj1  │ Obj2  │  │
│  └───┴───┴───┴───┴───┴───┴───┘    │  └───────┴───────┴───────┘  │
│  (raw bytes, no boxing)           │  (String, etc.)             │
└─────────────────────────────────────────────────────────────────┘
```

One powerful aspect of the register system is that you can reuse registers. If you reuse registers, then without any allocation you can construct and deconstruct things all day long. This is particularly valuable in high-throughput scenarios like:
- Deserializing streams of records
- Batch processing operations
- Real-time data pipelines

## RegisterOffset: Tracking Positions

`RegisterOffset` is a compact way to track positions within the `Registers` structure. It uses a single `Int` to encode two pieces of information:
1. Byte offset (for primitives) in the upper 16 bits
2. Object offset (for references) in the lower 16 bits

You can think of `RegisterOffset` as a cursor that tells you where to read/write the next primitive or object value. For example:
1. `RegisterOffset.Zero` represents the starting position with zero indexes for both primitives and objects.
2. `RegisterOffset(objects = 1)` indicates that the next object reference should be stored at index 1 in the object array, with zero byte offset for primitives.
3. `RegisterOffset(bytes = 10, objects = 3)` indicates that the next primitive value should be retrieved/stored starting at byte index 10 in the byte array, and the next object reference should be stored at index 3 in the object array.
4. `RegisterOffset(bytes = 4, ints = 2, objects = 3)` indicates that the next primitive value should be retrieved/stored starting at byte index 8 (4 bytes + 2 ints × 4 bytes each) in the byte array, and the next object reference should be stored at index 3 in the object array.

The byte offset is calculated by weighting each type by its size:

```
primitiveBytes = booleans + bytes 
               + (chars + shorts) × 2 
               + (floats + ints) × 4 
               + (doubles + longs) × 8
```

The object offset is simply the count of object references.

You don't need to do these calculations manually; the `RegisterOffset.getBytes` method computes the byte offset of the primitive register, and `RegisterOffset.getObjects` computes the offset of the object register.

```scala
RegisterOffset.getBytes(RegisterOffset(bytes = 4, ints = 2, objects = 3)) // output = (4 + 2*4) = 12
RegisterOffset.getObjects(RegisterOffset(bytes = 4, ints = 2, objects = 3)) // output = 3
```

## Creating Registers

`Registers` is a mutable container that holds values. It's created with an initial capacity:

```scala

// Create registers with space for 3 bytes, 1 ints, and 2 objects
val registers = Registers(RegisterOffset(bytes = 3, ints = 1, objects = 2))
```

While creating `Registers`, you specify how much space is needed for primitives and objects, using `RegisterOffset`. The library allocates the necessary arrays internally. After creation, when you set values, the library ensures that it has sufficient capacity, growing the arrays if necessary.

## Setting and Getting Values

The `Registers` data type has methods to set and get values for each primitive type and for object references. For example, the `setInt` method sets an `Int` value in the byte array, while `setObject` sets an object reference in the object array:

```scala
class Registers {
  private var bytes: Array[Byte]     = new Array[Byte](byteArrayLength)
  private var objects: Array[AnyRef] = new Array[AnyRef](objectArrayLength)

  // Methods to get/set primitive values in byte array (getInt/setInt, getBoolean/setBoolean, etc.)
  def getInt(offset: RegisterOffset): Int = ???
  def setInt(offset: RegisterOffset, value: Int): Unit = ???

  def setBoolean(offset: RegisterOffset, value: Boolean): Unit = ???
  def getBoolean(offset: RegisterOffset): Boolean = ???

  // Two methods to get/set object references in object array
  def getObject(offset: RegisterOffset): AnyRef = ???
  def setObject(offset: RegisterOffset, value: AnyRef): Unit = ???
}
```

Here are all the available methods:
- `setBoolean` / `getBoolean`
- `setByte` / `getByte`
- `setShort` / `getShort`
- `setInt` / `getInt`
- `setLong` / `getLong`
- `setFloat` / `getFloat`
- `setDouble` / `getDouble`
- `setChar` / `getChar`
- `setObject` / `getObject`

When setting or getting a value, you are required to provide one or two of the following parameters:

1. `offset`: The `RegisterOffset` indicating where the specific field starts in the `Registers`. It can be used to handle nested structures to point to where inner record starts, or for multiple records to point to each record's starting position, or for variant types to anchor the position of different cases.
2. `value`: The actual value to set. For the primitives, this value is a typed primitive (e.g. `Int`, `Double`); for objects, this is an `AnyRef`.

## Example: Encoding/Decoding a Record Data Type

Encoding data instances into registers involves mapping each field of a data type to its corresponding position in the two register arrays (byte array for primitives and object array for references). For example, assume we have a `Person` data type as below:

```scala
case class Person(
  name: String,
  email: String,
  age: Int,
  height: Double,
  weight: Double
)
```

We can encode it with registers, as follows:

```scala

// Person("John", "john@example.com", 42, 180.0, 67.0)
val registers = Registers(RegisterOffset(objects = 2, ints = 1, doubles = 2))

registers.setObject(
  RegisterOffset.Zero, // Object index: 0
  "John"
)

registers.setObject(
  RegisterOffset(objects = 1), // Object index: 1
  "john@example.com"
)

registers.setInt(
  RegisterOffset(objects = 2), // Byte index: 0
  42
)

registers.setDouble(
  RegisterOffset(objects = 2, ints = 1), // Byte index: (1 * 4) = 4
  180.0
)

registers.setDouble(
  RegisterOffset(objects = 2, ints = 1, doubles = 1), // Byte index: (1 * 4) + (1 * 8) = 12
  67.0
)
```

Conversely, to decode the `Person` data type from registers, you would read the values back from their respective positions:

```scala

// Decode Person from registers
val name = registers.getObject(
  RegisterOffset.Zero, // Object index: 0
).asInstanceOf[String]

val email = registers.getObject(
  RegisterOffset(objects = 1) // Object index: 1
).asInstanceOf[String]

val age = registers.getInt(
  RegisterOffset(objects = 2) // Object index: 2
)

val height = registers.getDouble(
  RegisterOffset(objects = 2, ints = 1) // Byte index: (1 * 4) = 4
)

val weight = registers.getDouble(
  RegisterOffset(objects = 2, ints = 1, doubles = 1) // Byte index: (1 * 4) + (1 * 8) = 12
)

val person = Person(name, email, age, height, weight)
```

---

## DeferHandle

`DeferHandle` is a handle returned by `Scope.defer` that allows cancelling a registered finalizer before the scope closes:

```scala
abstract class DeferHandle {
  def cancel(): Unit
}
```

When `Scope#defer(cleanup)` is called, the cleanup action is registered and a `DeferHandle` is returned. This handle can be used to remove that finalizer early, preventing it from running when the scope closes. This is useful when a resource is explicitly released before the scope ends, and running the finalizer again would be unnecessary or harmful.

## Construction

`DeferHandle` is not instantiated directly. Instead, it is created by calling `Scope#defer` with a cleanup action:

```scala

trait Scope {
  def defer(cleanup: => Unit): DeferHandle
}
```

The following example demonstrates creating a `DeferHandle`:

```scala

Scope.global.scoped { scope =>

  val handle = defer {
    println("This cleanup will run when scope closes, unless cancelled")
  }

  // The handle can now be used to cancel the cleanup
  handle.cancel()
}
```

## Core Operations

The `DeferHandle#cancel` method removes the registered finalizer so it will not run when the scope closes:

```scala
trait DeferHandle {
  def cancel(): Unit
}
```

This method is:

- **Thread-safe**: Can be called from any thread without synchronization
- **Idempotent**: Calling it multiple times has the same effect as calling once

If the scope has already closed (and the finalizer has already run or been discarded), calling `DeferHandle#cancel` is a no-op. In the following example, we register a cleanup action, then cancel it before the scope closes:

```scala

Scope.global.scoped { scope =>

  val buffer = allocate(ByteArrayOutputStream())
  val closeHandle = defer {
    println("Auto-closing buffer")
    $(buffer)(_.close())
  }

  // Manually close the buffer
  $(buffer)(_.close())

  // Cancel the automatic finalizer since we already closed it
  closeHandle.cancel()
}
```

## Use Cases

`DeferHandle` is useful in several common scenarios:

### Preventing Duplicate Cleanup

When a resource is explicitly released before the scope ends, cancel the automatic finalizer to avoid duplicate cleanup:

```scala

val result = Scope.global.scoped { scope =>

  val buffer = allocate(ByteArrayOutputStream())

  val finalizeHandle = defer {
    println(s"Finalizer running, buffer closing")
    $(buffer)(_.close())
  }

  // Explicit cleanup
  $(buffer) { buf =>
    buf.write("data".getBytes)
    println(s"Manual use: buffer has ${buf.size()} bytes")
    buf.close()
  }

  // Cancel the automatic finalizer
  finalizeHandle.cancel()
  
  "done"
}
```

### Conditional Cleanup

Cancel finalizers based on runtime conditions:

```scala

def acquireResource(shouldCleanup: Boolean) = Scope.global.scoped { scope =>

  val resource = "important resource"
  val handle = scope.defer {
    println("Cleaning up resource")
  }

  if (!shouldCleanup) {
    handle.cancel()
    println("Cleanup disabled")
  }

  resource
}

acquireResource(shouldCleanup = false)
acquireResource(shouldCleanup = true)
```

### Transferring Ownership

When transferring a resource to external management, cancel its finalizer so the external system can control cleanup:

```scala

val result = Scope.global.scoped { scope =>

  val stream = allocate(ByteArrayInputStream("data".getBytes))
  val handle = defer {
    println("Scope finalizer would close stream")
    $(stream)(_.close())
  }

  // Transfer ownership to external manager
  // (In real code, this might pass to a thread pool or async framework)
  handle.cancel()  // Let the manager handle cleanup
  
  "transferred"
}
```

## Noop Handle

When `defer()` is called on an already-closed scope, a no-op handle is returned:

```scala

Scope.global.scoped { scope =>

  val handle = scope.defer {
    println("This will run when scope closes")
  }

  // Subsequent calls to cancel() remove the finalizer
  handle.cancel()

  println("Finalizer has been cancelled")
}
```

## Thread Safety

`DeferHandle` is thread-safe. Multiple threads can call `cancel()` on the same handle without external synchronization:

```scala

val result = Scope.global.scoped { scope =>

  val handle = defer {
    println("Finalizer")
  }

  // Use a thread pool to simulate concurrent access
  val executor = Executors.newFixedThreadPool(5)
  val latch = new CountDownLatch(5)

  (1 to 5).foreach { i =>
    executor.submit(new Runnable {
      def run(): Unit = {
        handle.cancel()
        println(s"Thread $i cancelled")
        latch.countDown()
      }
    })
  }

  // Wait for all threads to finish
  latch.await()
  executor.shutdown()
  
  "completed"
}
```

## See Also

- [`Scope#defer`](./scope.md#registering-finalizers) — the method that returns a `DeferHandle`
- [`Finalizer`](./finalizer.md) — the trait defining `Finalizer#defer`
- [`Finalization`](./finalization.md) — the result of running all finalizers

## Integration

`DeferHandle` is part of ZIO Blocks' resource management system. It works directly with:

- **[`Scope`](./scope.md)** — The primary way to create a `DeferHandle` is via `Scope#defer`. A scope manages multiple finalizers and runs them all when the scope closes. `DeferHandle` allows selective cancellation of individual finalizers before that happens.

- **[`Finalizer`](./finalizer.md)** — `Finalizer` defines the `Finalizer#defer` operation that returns a `DeferHandle`. It abstracts the concept of registering cleanup actions.

- **[`Finalization`](./finalization.md)** — When a scope closes, it runs all registered finalizers. A cancelled `DeferHandle` removes its associated finalizer from this process.

Together, these types form the foundation of compile-time resource safety in ZIO Blocks, allowing you to manage resource lifecycles with certainty that cleanup will occur exactly when needed.

---

## Finalization

`Finalization` is the result of running all finalizers in a scope, collecting any errors that occurred during cleanup:

```scala

abstract class Finalization(val errors: Chunk[Throwable]) {
  def isEmpty: Boolean
  def nonEmpty: Boolean
  def orThrow(): Unit
  def suppress(initial: Throwable): Throwable
}
```

When a scope closes, each registered finalizer runs in LIFO order. If any finalizer throws an exception, that error is caught and collected into a `Finalization`. This type ensures that all finalizers run even if some fail, and allows the caller to decide how to handle accumulated errors.

`Finalization` collects errors from finalizers in a `Chunk[Throwable]`. The first error in the chunk corresponds to the head of the chunk (the first finalizer that failed in LIFO execution order).

## Core Methods

The following four methods allow you to inspect and handle errors from finalization:

### `Finalization#isEmpty`

Returns `true` if no finalizer errors were collected:

```scala

abstract class Finalization(val errors: Chunk[Throwable]) {
  def isEmpty: Boolean
}
```

Here's an example of checking if finalization succeeded:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>

    defer {
      println("Cleanup")
    }
    val fin = openScope.close()
    if (fin.isEmpty) println("No errors") else println("Errors occurred")
  }
}
```

### `Finalization#nonEmpty`

Returns `true` if at least one finalizer error was collected:

```scala

abstract class Finalization(val errors: Chunk[Throwable]) {
  def nonEmpty: Boolean
}
```

Here's an example of checking for errors:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>

    defer {
      throw new Exception("Cleanup failed")
    }
    val fin = openScope.close()
    if (fin.nonEmpty) {
      println(s"Errors occurred: ${fin.errors.length}")
    }
  }
}
```

### `Finalization#orThrow()`

Throws the first collected error with all remaining errors added as suppressed exceptions. Does nothing if there are no errors:

```scala

abstract class Finalization(val errors: Chunk[Throwable]) {
  def orThrow(): Unit
}
```

The first error corresponds to the head of the chunk (the first finalizer that failed in LIFO execution order). Remaining errors are attached as suppressed exceptions using `addSuppressed()`. Here's an example:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    defer { throw Exception("Error 2") }
    defer { throw Exception("Error 1") }
    val fin = openScope.close()
    
    try {
      fin.orThrow()
    } catch {
      case e: Exception =>
        println(s"Primary: ${e.getMessage}")
        e.getSuppressed.foreach(s => println(s"Suppressed: ${s.getMessage}"))
    }
  }
}
```

### `Finalization#suppress(initial)`

Adds all collected finalizer errors as suppressed exceptions to `initial` and returns it. If there are no errors, `initial` is returned unchanged:

```scala

abstract class Finalization(val errors: Chunk[Throwable]) {
  def suppress(initial: Throwable): Throwable
}
```

This is useful when you want to preserve the original error context while attaching cleanup errors. Here's an example:

```scala

Scope.global.scoped { scope =>

  val initialError = Exception("Original error")

  $(open()) { openScope =>
    defer { throw Exception("Cleanup error") }
    val fin = openScope.close()
    
    val combined = fin.suppress(initialError)
    println(s"Primary: ${combined.getMessage}")
    combined.getSuppressed.foreach(s => println(s"Suppressed: ${s.getMessage}"))
  }
}
```

## Error Ordering

Errors in the finalization are ordered by when finalizers ran (in LIFO sequence):

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    // Registered first, runs last (LIFO)
    defer { throw Exception("Error 1") }

    // Registered second, runs first
    defer { throw Exception("Error 2") }

    // Registered third, runs first
    defer { throw Exception("Error 3") }

    val fin = openScope.close()
    
    // Errors list order: [Error 3, Error 2, Error 1]
    println(s"First error: ${fin.errors.head.getMessage}")
    println(s"Total errors: ${fin.errors.length}")
  }
}
```

## Use Cases

Here are common scenarios where finalization handling is useful:

### Conditional Error Handling

Check if errors occurred and handle them appropriately:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    defer {
      println("Cleanup complete")
    }
    val fin = openScope.close()
    
    if (fin.nonEmpty) {
      fin.orThrow()
    } else {
      println("No errors during finalization")
    }
  }
}
```

### Combining Multiple Error Sources

Attach cleanup errors to an existing error:

```scala

def doWork(): Unit = {
  Scope.global.scoped { scope =>

    try {
      throw Exception("Work failed")
    } catch {
      case workError: Exception =>
        $(open()) { openScope =>
          defer { throw Exception("Cleanup failed") }
          val fin = openScope.close()
          val combined = fin.suppress(workError)
          throw combined
        }
    }
    ()  // Return unit on normal path
  }
}

try {
  doWork()
} catch {
  case e: Exception =>
    println(s"Work: ${e.getMessage}")
    e.getSuppressed.foreach(s => println(s"During cleanup: ${s.getMessage}"))
}
```

### Logging All Cleanup Errors

Inspect and log all errors without stopping execution:

```scala

Scope.global.scoped { scope =>

  $(open()) { openScope =>
    defer { throw Exception("Error 1") }
    defer { throw Exception("Error 2") }
    val fin = openScope.close()
    
    if (fin.nonEmpty) {
      println(s"Finalization collected ${fin.errors.length} errors:")
      fin.errors.foreach(e => println(s"  - ${e.getMessage}"))
    }
  }
}
```

## Relationship to Scope

`Finalization` is returned by:

- `Scope.open().close()` — when explicitly closing a scope
- `Scope.global.runFinalizers()` — when running global finalizers on shutdown

## See Also

- [`Scope#defer`](./scope.md) — registers finalizers that produce errors
- [`DeferHandle`](./defer-handle.md) — handle for cancelling finalizers
- [`Finalizer`](./finalizer.md) — the trait for registering cleanup actions

---

## Finalizer

`Finalizer` is a minimal capability interface for registering cleanup actions. It exposes only the `Finalizer#defer` method, preventing code from accessing scope internals like resource allocation or closing.

The structural definition:

```scala
trait Finalizer {
  def defer(f: => Unit): DeferHandle
}
```

This trait serves as a boundary between scope management internals and user code that only needs to register cleanup actions. By exposing only `defer`, code can safely request cleanup registration without requiring full scope access.

## Motivation / Use Case

`Finalizer` integrates with `Scope` to enable resource management patterns:

```scala

def openConnection(url: String)(implicit fin: Finalizer): String = {
  fin.defer {
    println(s"Closing connection to $url")
  }
  s"Connected to $url"
}

Scope.global.scoped { scope =>

  openConnection("https://example.com")
  // Connection closes when scope exits
  ()
}
```

By decoupling code that needs cleanup registration from code that manages the complete scope lifecycle, `Finalizer` allows functions to safely register finalizers without requiring full scope access.

## Construction / Creating Instances

`Finalizer` is not typically constructed directly. Instead, it is obtained through a scope:

### From a `Scope`

Any `Scope` instance can be used as a `Finalizer` since `Scope extends Finalizer`:

```scala

Scope.global.scoped { scope =>

  // scope is both a Scope and a Finalizer
  val handle = scope.defer {
    println("Cleanup")
  }
  ()  // Return unit
}
```

### As a Context Bound

Functions can request a `Finalizer` via `implicit` parameter, enabling decoupled cleanup registration:

```scala

def setupResource(name: String)(implicit fin: Finalizer): String = {
  fin.defer {
    println(s"Closing $name")
  }
  name
}
```

## Core Operations

The `Finalizer` interface provides a single core operation for registering cleanup handlers:

### `Finalizer#defer`

Registers a finalizer (cleanup action) to run when the scope closes. The cleanup action runs in LIFO order along with other finalizers registered on the same scope. Returns a `DeferHandle` that can cancel the registration before the scope closes:

```scala

Scope.global.scoped { scope =>

  val handle1 = scope.defer {
    println("Cleanup 1")
  }

  val handle2 = scope.defer {
    println("Cleanup 2")
  }

  // Finalizers run in LIFO: Cleanup 2, then Cleanup 1
  // Can cancel before scope closes
  handle1.cancel()
  // Now only Cleanup 2 runs
}
```

### Package-Level `defer` Helper

A package-level convenience function allows writing `defer { cleanup }` when a `Finalizer` is in scope. The signature is:

```scala
def defer(finalizer: => Unit)(implicit fin: Finalizer): DeferHandle
```

This removes the need to write `fin.defer { cleanup }`. Here's the convenience function in use:

```scala

def setupWithCleanup()(implicit fin: Finalizer) = {
  defer {
    println("Cleanup")
  }
}

Scope.global.scoped { scope =>

  setupWithCleanup()
  // Cleanup prints when scope closes
  ()  // Return unit (which is Unscoped)
}
```

## Integration

`Finalizer` is a supertrait of `Scope`. The structural definition shows this relationship:

```scala
sealed abstract class Scope extends Finalizer with ScopeVersionSpecific
```

This means any `Scope` instance can be used where a `Finalizer` is expected. However, the converse is not true—a `Finalizer` reference does not provide `Scope#allocate`, `Scope#$`, or other scope operations.

## Finalization Order

Finalizers registered with `Finalizer#defer` run in **LIFO order** (last registered runs first) when the scope closes. This ensures that resources acquired in order can be cleaned up in reverse order:

```scala

Scope.global.scoped { scope =>

  scope.defer { println("First registered, runs last") }
  scope.defer { println("Second registered, runs first") }
  // Output on scope close:
  // Second registered, runs first
  // First registered, runs last
  ()  // Return unit (which is Unscoped)
}
```

## See Also

- [`Scope`](./scope.md) — the full scope lifecycle management
- [`DeferHandle`](./defer-handle.md) — the handle returned by `Finalizer#defer` for cancellation
- [`Finalization`](./finalization.md) — the result of running all finalizers

---

## Resource Management & Dependency Injection

## Introduction

Resource management and dependency injection are fundamental to building reliable, maintainable applications. ZIO Blocks provides three complementary types that work together to eliminate common lifetime bugs while enabling powerful composition patterns: **Scope** provides compile-time safe resource boundaries, **Resource** encapsulates acquisition and cleanup with automatic finalization, and **Wire** describes dependency graphs with type-safe construction recipes. Together, they form a cohesive system for managing object lifecycles, preventing resource leaks, and building dependency-injected architectures.

## Overview

These three types solve the fundamental problem of managing resources and dependencies in concurrent, long-lived applications:

**[Scope](./scope.md)** is the foundation — it provides a compile-time safe boundary that prevents resources from escaping their intended lifetime. Using path-dependent types, Scope ensures that values allocated in one scope cannot accidentally be used in another scope, catching lifetime violations at compile time rather than causing runtime bugs.

**[Resource](./resource.md)** builds on Scope to describe how to acquire and finalize resources. Rather than executing immediately, a Resource is a lazy recipe that composes naturally with `map`, `flatMap`, and `zip`. When allocated within a scope, finalizers run automatically in LIFO order, ensuring cleanup happens even when errors occur.

**[Wire](./wire.md)** brings it all together by describing how to construct services and their dependencies. The Wire macro automatically handles dependency resolution, cycle detection, and AutoCloseable registration, letting you declaratively specify a dependency graph that the compiler validates.

### How They Work Together

The typical flow is:

1. **Define** dependencies using `Wire.shared[T]` or `Wire.unique[T]` — the macro inspects constructor parameters and generates a wire
2. **Compose** wires together using `Resource.from[App](wire1, wire2, ...)` — builds the dependency graph
3. **Allocate** within a scope using `scope.allocate(resource)` — acquires resources and registers finalizers
4. **Use** scoped values via the `$` accessor — compile-time ensures they can't escape
5. **Cleanup** happens automatically when the scope exits — finalizers run in reverse order (LIFO)

### Common Patterns

**Shared Singletons** — Use `Wire.shared[T]` for expensive resources (database connections, thread pools) that should be created once and reused across the application.

**Per-Request Instances** — Use `Wire.unique[T]` for request-scoped state that should be fresh for each request or operation.

**Manual Construction** — Use `Wire.Shared.fromFunction` or `Wire.Unique.fromFunction` when macro derivation doesn't fit your use case (custom initialization logic, special parameters).

**Resource Composition** — Use `Resource.map`, `Resource.flatMap`, and `Resource.zip` to build complex dependency chains where later resources depend on earlier ones.

### Integration Points

- **Wire** uses **Resource** to manage lifecycles of constructed services
- **Resource** uses **Scope** for finalization and scoped value boundaries
- Both **Wire** and **Resource** produce values that are usable only within a **Scope** context

---

## Resource

`Resource[A]` is a **lazy recipe for managing resource lifecycles**, encapsulating both acquisition and finalization tied to a `Scope`. Resources describe *what* to do, not *when* — creation only happens when a resource is passed to `scope.allocate()`. They compose naturally with `map`, `flatMap`, and `zip` to build complex dependency graphs with automatic cleanup in LIFO order.

```scala
sealed trait Resource[+A] {
  def map[B](f: A => B): Resource[B]
  def flatMap[B](f: A => Resource[B]): Resource[B]
  def zip[B](that: Resource[B]): Resource[(A, B)]
}
```

Key properties:
- **Lazy**: Resources don't acquire anything until allocated via `scope.allocate()`
- **Covariant**: `Resource[Dog]` is a subtype of `Resource[Animal]` when `Dog <: Animal`
- **Composable**: `map`, `flatMap`, and `zip` combine resources into larger structures
- **Two strategies**: Shared (memoized with reference counting) and Unique (fresh per allocation)
- **Auto-cleanup**: Finalizers run automatically in LIFO order when scopes close

## Motivation

Without Resources, managing complex initialization and cleanup is tedious and error-prone. Resources eliminate manual bookkeeping by tying value lifecycles to Scopes and registering finalizers automatically.

**Benefits:**
- Automatic cleanup even on exceptions
- LIFO finalization order (inner resources close before outer ones)
- Compositional: build complex dependency graphs declaratively
- Type-safe: compiler ensures you have dependencies available
- Works seamlessly with `Wire` for constructor-based dependency injection

## Installation

Add the ZIO Blocks Scope module to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.33"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-scope" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## Shared and Unique Resources

Resource offers two distinct types for managing instance lifecycles: **Shared** for singleton-like resources that are allocated once and reused across multiple scopes, and **Unique** for fresh resources created on each allocation. Understanding when to use each type is critical for building efficient and correct resource-management architectures.

### Comparison

| Aspect             | Shared                                             | Unique                                    |
|--------------------|----------------------------------------------------|-------------------------------------------|
| **Creation**       | `Resource.shared(f)`                               | `Resource.unique(f)` or `Resource(value)` |
| **Memoization**    | Yes, with reference counting                       | No, fresh per allocation                  |
| **When to use**    | Expensive resources (DB connections, thread pools) | Per-request state, isolated handlers      |
| **Instance reuse** | Same instance across all allocations               | New instance per allocation               |
| **Finalization**   | Runs when last reference released                  | Runs immediately when scope closes        |
| **Thread safety**  | Lock-free atomic reference counting                | Per-scope, no global coordination needed  |

### Shared Resources

**Shared resources are memoized**: the first allocation initializes the instance; subsequent allocations return the same reference with automatic reference counting. Finalizers run only when the last reference is released. Use shared resources for **expensive, globally singular components** — database connection pools, thread pools, logging systems, caches, and other heavyweight infrastructure that should exist exactly once for the lifetime of the application.

The canonical example is a **database connection pool**. Building a fresh pool for each service layer is wasteful and defeats pooling's purpose. Instead, wrap the pool in a shared resource: both the user service and order service allocate the same pool instance, with the system automatically tracking references and closing the pool only when the last service releases it.

Here's what shared acquisition looks like:

```scala
object Resource {
  def shared[A](f: Scope => A): Resource[A]
}
```

When you allocate a shared resource, reference counting ensures cleanup happens exactly once:

```scala

class DatabasePool extends AutoCloseable {
  def close(): Unit = println("Pool closed (after all services released)")
}

val poolResource = Resource.shared { scope =>
  println("Creating database pool (first allocation only)")
  new DatabasePool
}

// Allocate in ServiceA
Scope.global.scoped { scopeA =>

  val poolA: $[DatabasePool] = poolResource.allocate
  println("ServiceA allocated pool")

  // Allocate in ServiceB within a nested scope
  scopeA.scoped { scopeB =>

    val poolB: $[DatabasePool] = poolResource.allocate
    println("ServiceB allocated same pool instance (reference count incremented)")

    // Both services share the same underlying pool instance
    println("Both ServiceA and ServiceB are using the same pool instance")
  }
  println("ServiceB released, but pool stays open for ServiceA (ref count -= 1)")
}
println("All services released, pool closed (ref count == 0)")
```

### Unique Resources

**Unique resources create fresh instances each time**: every allocation produces a new value. Finalizers run when their owning scope closes, independent of other allocations. Use unique resources for **per-request state, isolated services, and stateful handlers** — anything that should never be shared because it encapsulates request-specific or scope-specific data.

A typical scenario is **per-request caches**: each API request gets its own cache instance to avoid one request polluting another's cached data. Similarly, stateful handlers (parser state machines, transaction contexts, event buffers) need isolation to prevent cross-contamination.

Here's what unique acquisition looks like:

```scala
object Resource {
  def unique[A](f: Scope => A): Resource[A]
}
```

When you allocate a unique resource, each allocation is independent:

```scala

class RequestCache extends AutoCloseable {
  def close(): Unit = println("Request cache closed")
}

val cacheResource = Resource.unique { scope =>
  println("Creating new request cache")
  new RequestCache
}

// Two allocations in the same scope produce different instances
Scope.global.scoped { scope =>

  println("Creating first cache...")
  val cache1: $[RequestCache] = cacheResource.allocate

  println("Creating second cache...")
  val cache2: $[RequestCache] = cacheResource.allocate

  println("Both caches are independent instances")
}
```

### Diamond Dependency Pattern

A classic architecture uses shared resources to solve **diamond dependencies**, where multiple services depend on the same expensive component. Consider an e-commerce application where both `ProductService` and `OrderService` depend on `Logger`:

```
       CachingApp
        /      \
       /        \
  ProductService  OrderService
       \        /
        \      /
         Logger
```

Without shared resources, each service would receive a different `Logger` instance. With `Resource.shared`, both services automatically receive the same singleton instance:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(s"LOG: $msg")
  def close(): Unit = println("Logger closed")
}

class ProductService(val logger: Logger) {
  def findProduct(id: String): String = {
    logger.log(s"Finding product $id")
    s"Product-$id"
  }
}

class OrderService(val logger: Logger) {
  def createOrder(productId: String): String = {
    logger.log(s"Creating order for $productId")
    s"ORD-123"
  }
}

class CachingApp(val productService: ProductService, val orderService: OrderService) extends AutoCloseable {
  def close(): Unit = println("App closed")
}

Scope.global.scoped { scope =>

  // Use Wire.shared[Logger] to ensure only one instance is created
  val app: $[CachingApp] = allocate(
    Resource.from[CachingApp](
      Wire.shared[Logger]  // Both services get the same Logger instance
    )
  )

  $(app) { a =>
    println(s"ProductService and OrderService share Logger? ${a.productService.logger eq a.orderService.logger}")
    a.productService.findProduct("P001")
    a.orderService.createOrder("P001")
  }
}
```

In this pattern, `Resource.shared` (via `Wire.shared`) guarantees a single `Logger` instance across all services, eliminating duplication while maintaining proper cleanup semantics.

### Multiple Shared Dependencies

Most applications need more than one shared resource. Here's a realistic example where `CachingApp` depends on both a shared `Logger` and a shared `MetricsCollector`. Both resources are singletons that all services share:

```
          CachingApp
           /      \
          /        \
         /          \
    ProductService  OrderService
     /  \            /  \
    /    \          /    \
   Logger Metrics  Logger Metrics
    \    /          \    /
     \  /            \  /
      \/              \/
   (shared singleton instances)
```

Here's the implementation:

```scala

class Logger extends AutoCloseable {
  def log(msg: String): Unit = println(s"[LOG] $msg")
  def close(): Unit = println("[Logger] Closed")
}

class MetricsCollector extends AutoCloseable {
  private var eventCount = 0
  def recordEvent(name: String): Unit = {
    eventCount += 1
    println(s"[METRICS] Event: $name (total: $eventCount)")
  }
  def close(): Unit = println(s"[MetricsCollector] Closed after $eventCount events")
}

class ProductService(val logger: Logger, val metrics: MetricsCollector) {
  def findProduct(id: String): String = {
    logger.log(s"Finding product $id")
    metrics.recordEvent("product.find")
    s"Product-$id"
  }
}

class OrderService(val logger: Logger, val metrics: MetricsCollector) {
  def createOrder(productId: String): String = {
    logger.log(s"Creating order for $productId")
    metrics.recordEvent("order.create")
    s"ORD-123"
  }
}

class CachingApp(
  val productService: ProductService,
  val orderService: OrderService
) extends AutoCloseable {
  def close(): Unit = println("[CachingApp] Closed")
}

Scope.global.scoped { scope =>

  // Both Logger and MetricsCollector are shared (singleton instances)
  val app: $[CachingApp] = allocate(
    Resource.from[CachingApp](
      Wire.shared[Logger],              // One Logger for all services
      Wire.shared[MetricsCollector]     // One MetricsCollector for all services
    )
  )

  $(app) { a =>
    println(s"ProductService and OrderService share Logger? ${a.productService.logger eq a.orderService.logger}")
    println(s"ProductService and OrderService share Metrics? ${a.productService.metrics eq a.orderService.metrics}")

    a.productService.findProduct("P001")
    a.orderService.createOrder("P001")
  }
}
```

When this scope closes, both the `Logger` and `MetricsCollector` are finalized exactly once, regardless of how many services reference them. The macro automatically builds a dependency graph, verifies all wires, and ensures LIFO cleanup order.

## Construction

Resources can be created in several ways: from values, from explicit acquire/release pairs, from `AutoCloseable` types, from custom functions, or derived automatically from a type's constructor using the `Resource.from` macros.

### `Resource.apply` — Wrap a Value

Wraps a by-name value as a resource. If the value implements `AutoCloseable`, its `close()` method is automatically registered as a finalizer:

```scala
object Resource {
  def apply[A](value: => A): Resource[A]
}
```

Here's how to use it:

```scala

case class Config(debug: Boolean)

class Database(val name: String) extends AutoCloseable {
  def close(): Unit = println(s"Closing database $name")
}

val configResource = Resource(Config(debug = true))
val dbResource = Resource(new Database("mydb"))
```

### `Resource.acquireRelease` — Explicit Lifecycle

Creates a resource with separate acquire and release functions. The acquire thunk runs during allocation; the release function is registered as a finalizer:

```scala
object Resource {
  def acquireRelease[A](acquire: => A)(release: A => Unit): Resource[A]
}
```

Here's an example with file handling:

```scala

val fileResource = Resource.acquireRelease {
  new FileInputStream("data.txt")
} { stream =>
  stream.close()
}
```

### `Resource.fromAutoCloseable` — Type-Safe Wrapping

Creates a resource specifically for `AutoCloseable` subtypes. This is a compile-time verified alternative to `Resource(value)` when you know the value is closeable:

```scala
object Resource {
  def fromAutoCloseable[A <: AutoCloseable](value: => A): Resource[A]
}
```

Here's an example with a stream:

```scala

val streamResource = Resource.fromAutoCloseable {
  new BufferedInputStream(new FileInputStream("data.bin"))
}
```

### `Resource.shared` — Memoized with Reference Counting

Creates a shared resource that memoizes its value across multiple allocations using reference counting. The first allocation initializes the value using an `OpenScope` parented to `Scope.global`; subsequent allocations increment the reference count and return the same instance. Each scope that receives the shared value registers a finalizer that decrements the count. When the count reaches zero, the shared scope closes automatically. This mechanism is **thread-safe and lock-free**, implemented using `AtomicReference` with atomic compare-and-swap operations to avoid contention.

Under the hood, a shared resource progresses through four states: (1) **Uninitialized** — the resource has never been allocated and the first call triggers initialization; (2) **Pending** — initialization is in progress and other threads wait via spin-yield for the value to become available; (3) **Created** — the value is fully initialized and ready, each allocation increments the reference count, and each scope registers a finalizer to decrement it; (4) **Destroyed** — the reference count reached zero, the resource was cleaned up, and further allocations will fail. This state machine ensures that no matter how many threads try to allocate simultaneously, the underlying value initializes exactly once, and cleanup happens exactly once when the last reference is released.

Use shared resources for **expensive, singleton-like components** (database connection pools, thread pools, logging systems, caches) that should exist exactly once for the lifetime of the application, even when multiple services depend on them.

Here's the signature:

```scala
object Resource {
  def shared[A](f: Scope => A): Resource[A]
}
```

Here's a realistic example showing reference counting and automatic cleanup:

```scala

class ExpensiveComponent extends AutoCloseable {
  println("ExpensiveComponent initialized (expensive operation)")
  def close(): Unit = println("ExpensiveComponent cleaned up (last reference released)")
}

val sharedResource = Resource.shared { scope =>
  new ExpensiveComponent()
}

// Multiple services allocating the same shared resource
Scope.global.scoped { scope =>

  // First allocation initializes the component
  println("ServiceA allocating...")
  val componentA: $[ExpensiveComponent] = sharedResource.allocate

  // ServiceB in a nested scope receives the same instance
  scope.scoped { innerScope =>

    println("ServiceB allocating (same instance, ref count += 1)...")
    val componentB: $[ExpensiveComponent] = sharedResource.allocate
    println("Both services have the same instance")
  }

  println("ServiceB released, but component stays alive (ref count -= 1)")
}

println("ServiceA released, component cleaned up (ref count == 0)")
```

### `Resource.unique` — Fresh Instances

Creates a unique resource that produces a fresh instance each time it's allocated. Use for per-request state or resources that should never be shared:

```scala
object Resource {
  def unique[A](f: Scope => A): Resource[A]
}
```

Here's an example showing fresh instances:

```scala

var counter = 0

val uniqueResource = Resource.unique[Int] { _ =>
  counter += 1
  counter
}
```

### `Resource.from[T]` — Derive from Constructor

Derives a `Resource[T]` from `T`'s primary constructor, requiring no external dependencies.
If `T` extends `AutoCloseable`, `Resource.from` automatically registers its `close()` method
as a finalizer:

```scala
object Resource {
  def from[T]: Resource[T]
}
```

To derive a resource for a type that has no constructor dependencies:

```scala

class MetricsCollector extends AutoCloseable {
  def record(event: String): Unit = println(s"Recording: $event")
  def close(): Unit               = println("MetricsCollector closed")
}

val metricsResource = Resource.from[MetricsCollector]
```

Internally, the macro inspects `T`'s constructor at compile time, verifies that no external
dependencies are needed, and synthesizes a `Resource.shared` that instantiates `T` and —
when `T` extends `AutoCloseable` — registers `close()` as a scope finalizer. Any attempt
to use `Resource.from[T]` on a type with unsatisfied constructor parameters results in a
compile-time error.

### `Resource.from[T](wires: Wire[?, ?]*)` — Derive with Dependency Overrides

Derives a `Resource[T]` from `T`'s constructor, with `Wire` values provided as dependency
overrides. Any dependency not covered by an explicit wire is auto-derived if the macro can
construct it; otherwise a compile-time error is produced:

```scala
object Resource {
  def from[T](wires: Wire[?, ?]*): Resource[T]
}
```

To derive a resource for a type whose dependencies are provided via wires:

```scala

case class Config(host: String, port: Int)

class Logger(config: Config) extends AutoCloseable {
  def log(msg: String): Unit = println(s"[$msg] ${config.host}:${config.port}")
  def close(): Unit          = log("Logger shutting down")
}

class Service(logger: Logger) extends AutoCloseable {
  def run(): Unit  = logger.log("Service running")
  def close(): Unit = logger.log("Service shutting down")
}

val serviceResource = Resource.from[Service](
  Wire(Config("localhost", 8080))
)
```

Internally, the macro builds a complete dependency graph at compile time: it inspects each
provided wire's input and output types, identifies all constructor parameters of `T`, and
auto-derives wires for any remaining dependencies. It then topologically sorts all wires to
determine acquisition order and composes them into a single `Resource` whose finalizers run
in LIFO order — inner dependencies close before outer ones.

## Core Operations

Resources support transformation and composition through `map`, `flatMap`, and `zip`.

### `Resource#map` — Transform the Value

Transforms the value produced by a resource without affecting finalization. The transformation function is applied after the resource is acquired:

```scala
trait Resource[+A] {
  def map[B](f: A => B): Resource[B]
}
```

Here's a usage example:

```scala

val portResource = Resource(8080)
val urlResource = portResource.map(port => s"http://localhost:$port")
```

### `Resource#flatMap` — Sequence Resources

Sequences two resources, using the result of the first to create the second. Both sets of finalizers are registered and run in LIFO order (inner before outer):

```scala
trait Resource[+A] {
  def flatMap[B](f: A => Resource[B]): Resource[B]
}
```

Here's an example with dependent resources:

```scala

case class DbConfig(url: String)

class Database(config: DbConfig) extends AutoCloseable {
  def query(sql: String): String = s"Result from ${config.url}: $sql"
  def close(): Unit = println("Database closed")
}

val configResource = Resource(DbConfig("jdbc:postgres://localhost"))
val dbResource = configResource.flatMap { config =>
  Resource.fromAutoCloseable(new Database(config))
}
```

### `Resource#zip` — Combine Resources

Combines two resources into a single resource that produces a tuple of both values. Both resources are acquired and both sets of finalizers are registered:

```scala
trait Resource[+A] {
  def zip[B](that: Resource[B]): Resource[(A, B)]
}
```

Here's an example combining multiple resources:

```scala

case class DbConfig(url: String)

class Database(config: DbConfig) extends AutoCloseable {
  def close(): Unit = println("Database closed")
}

class Cache extends AutoCloseable {
  def close(): Unit = println("Cache closed")
}

val dbResource = Resource.fromAutoCloseable(new Database(DbConfig("jdbc:postgres://localhost")))
val cacheResource = Resource.fromAutoCloseable(new Cache())
val combined = dbResource.zip(cacheResource)
```

### `Resource#allocate` — Acquire Within a Scope

Allocates a `Resource[A]` within the current `Scope`, returning a scoped `$[A]`. This is syntax sugar for `scope.allocate(resource)`, available via `import scope._` inside a `scoped` block:

```scala
implicit class ResourceOps[A](private val r: Resource[A]) {
  def allocate: $[A]
}
```

To allocate a resource and use its value inside a scope:

```scala

class Database extends AutoCloseable {
  def query(sql: String): String = s"Result: $sql"
  def close(): Unit              = println("Database closed")
}

Scope.global.scoped { implicit scope =>

  val db: $[Database] = Resource.fromAutoCloseable(new Database).allocate
  $(db)(_.query("SELECT 1"))
}
```

### `$[Resource[A]]#allocate` — Allocate a Scoped Resource

Allocates a `$[Resource[A]]` — a Resource that is itself a scoped value — returning `$[A]`. Use this when a method on a scoped object returns a Resource and you need to immediately acquire it while keeping the result scoped.

**What is a scoped resource?** A scoped value `$[A]` represents an `A` that is valid only while a scope is alive. A **scoped resource** `$[Resource[A]]` is a Resource object that exists *inside* that scope. When you call a method like `$(pool)(_.lease())` that returns a Resource, the result is typed as `$[Resource[Connection]]` — the Resource itself is scoped. The `.allocate` extension method unwraps this scoped resource and acquires it, returning the acquired value as a new scoped value `$[A]`.

**Motivation:** This pattern appears frequently in resource factories. A scoped object (like a database pool) has methods that produce new Resources. Without this extension, you'd need to unwrap the `$[Resource[A]]` from the `$` context, allocate it separately, and re-wrap the result. The `.allocate` method chains naturally, letting you write `$(pool)(_.lease()).allocate` instead of dealing with intermediate unwrapping.

The implicit class:

```scala
implicit class ScopedResourceOps[A](private val sr: $[Resource[A]]) {
  def allocate: $[A]
}
```

Here's a realistic example where a database pool (a scoped object) has a method that returns a Resource for individual connections:

```scala

class Connection extends AutoCloseable {
  def query(sql: String): String = s"Result: $sql"
  def close(): Unit              = println("Connection closed")
}

class Pool extends AutoCloseable {
  def lease(): Resource[Connection] = Resource.fromAutoCloseable(new Connection)
  def close(): Unit                 = println("Pool closed")
}

Scope.global.scoped { implicit scope =>

  val pool: $[Pool]           = Resource.fromAutoCloseable(new Pool).allocate
  val conn: $[Connection]     = $(pool)(_.lease()).allocate
  $(conn)(_.query("SELECT 1"))
}
```

## Integration

Resource is a core abstraction in ZIO Blocks' resource management ecosystem:

- **[`Scope`](./scope.md)** — Resources require a `Scope` for allocation. The `Scope` manages the lifetime of acquired resources and automatically runs finalizers when the scope closes.
- **[`Wire`](./wire.md)** — The `Resource.from[T](wires)` macro builds dependency graphs using `Wire` values, enabling constructor-based dependency injection with automatic resource management.
- **[`Finalizer`](./finalizer.md)** — Resources register their cleanup logic with `Finalizer` objects, which execute in LIFO order when scopes close.

For example, `Resource.from[T]` uses `Wire` to construct instances with their dependencies, automatically registering any `AutoCloseable` cleanup:

```scala
val serviceResource = Resource.from[Service](
  Wire(Config("localhost", 8080))
)
```

This builds a complete dependency graph: `Config` → `Logger` → `Service`, with all finalizers managed by the containing `Scope`.

## Running the Examples

All code from this guide is available as runnable examples in the `scope-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Lifecycle Management with Temporary Files

This example demonstrates creating and automatically cleaning up temporary files using Resource's lifecycle management. It shows how Resource ensures files are closed even if exceptions occur:

```scala title="scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `scope.defer(...)` for registering manual cleanup actions.
 *
 * This example shows how to create temporary files during processing and ensure
 * they are deleted when the scope exits—even if processing fails. Deferred
 * cleanup actions run in LIFO (last-in-first-out) order.
 */

/** Represents a temporary file with basic read/write operations. */
case class TempFile(path: String) {
  private var content: String = ""

  def write(data: String): Unit = content = data
  def read(): String            = content
  def delete(): Boolean         = { println(s"  Deleting: $path"); true }
}

/** Result of processing temporary files. */
case class ProcessingResult(processedCount: Int, totalBytes: Long, errors: List[String])

/** Processes a list of temporary files and aggregates results. */
object FileProcessor {
  def process(files: List[TempFile]): ProcessingResult = {
    val totalBytes = files.map(_.read().length.toLong).sum
    ProcessingResult(processedCount = files.size, totalBytes = totalBytes, errors = Nil)
  }
}

@main def tempFileHandlingExample(): Unit = {
  println("=== Temp File Handling Example ===\n")
  println("Demonstrating scope.defer() for manual cleanup registration.\n")

  val result = Scope.global.scoped { scope =>
    // Create temp files and register cleanup via defer.
    // Cleanup runs in LIFO order: file3, file2, file1.

    val file1 = createTempFile(scope, "/tmp/data-001.tmp", "First file content")
    val file2 = createTempFile(scope, "/tmp/data-002.tmp", "Second file - more data here")
    val file3 = createTempFile(scope, "/tmp/data-003.tmp", "Third file with the most content of all")

    println("\nProcessing files...")
    val processingResult = FileProcessor.process(List(file1, file2, file3))
    println(s"Processed ${processingResult.processedCount} files, ${processingResult.totalBytes} bytes\n")

    println("Exiting scope - cleanup runs in LIFO order:")
    processingResult
  }

  println(s"\nFinal result: $result")
}

/**
 * Creates a temporary file and registers its cleanup with the scope.
 *
 * The cleanup action is registered via `defer(...)`, ensuring the file is
 * deleted when the scope closes—regardless of whether processing succeeds.
 *
 * @param s
 *   the scope to register cleanup with
 * @param path
 *   the file path
 * @param content
 *   initial content to write
 * @return
 *   the created TempFile
 */
private def createTempFile(s: Scope, path: String, content: String): TempFile = {
  val file = TempFile(path)
  file.write(content)
  println(s"Created: $path (${content.length} bytes)")

  // Register cleanup - will run when scope exits, in LIFO order
  s.defer {
    file.delete()
  }

  file
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain scope.examples.tempFileHandlingExample"
```

### Acquiring and Releasing Database Connections

This example demonstrates the acquire-release pattern using Resource to manage database connections. It shows proper connection initialization and guaranteed cleanup:

```scala title="scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Configuration for database connection.
 *
 * @param host
 *   the database server hostname
 * @param port
 *   the database server port
 * @param database
 *   the database name to connect to
 */
final case class DbConfig(host: String, port: Int, database: String) {
  def connectionUrl: String = s"jdbc:postgresql://$host:$port/$database"
}

/**
 * Represents the result of a database query.
 *
 * @param rows
 *   the result set as a list of row maps
 */
final case class QueryResult(rows: List[Map[String, String]]) {
  def isEmpty: Boolean = rows.isEmpty
  def size: Int        = rows.size
}

/**
 * Simulates a database connection with lifecycle management.
 *
 * This class demonstrates how AutoCloseable resources integrate with ZIO Blocks
 * Scope. When allocated via `allocate(Resource(...))`, the `close()` method is
 * automatically registered as a finalizer.
 *
 * @param config
 *   the database configuration
 */
final class Database(config: DbConfig) extends AutoCloseable {
  private var connected = false

  def connect(): Unit = {
    println(s"[Database] Connecting to ${config.connectionUrl}...")
    connected = true
    println(s"[Database] Connected successfully")
  }

  def query(sql: String): QueryResult = {
    require(connected, "Database not connected")
    println(s"[Database] Executing: $sql")
    sql match {
      case s if s.contains("users") =>
        QueryResult(
          List(
            Map("id" -> "1", "name" -> "Alice"),
            Map("id" -> "2", "name" -> "Bob")
          )
        )
      case s if s.contains("orders") =>
        QueryResult(
          List(
            Map("order_id" -> "101", "user_id" -> "1", "total" -> "99.99"),
            Map("order_id" -> "102", "user_id" -> "2", "total" -> "149.50")
          )
        )
      case _ =>
        QueryResult(List(Map("result" -> "OK")))
    }
  }

  override def close(): Unit = {
    println(s"[Database] Closing connection to ${config.connectionUrl}")
    connected = false
  }
}

/**
 * Demonstrates basic resource lifecycle management with ZIO Blocks Scope.
 *
 * This example shows:
 *   - Allocating an AutoCloseable resource with automatic cleanup
 *   - Using `$(value)(f)` to access scoped values and execute queries
 *   - LIFO finalizer ordering (last allocated = first closed)
 *
 * When the scope exits, all registered finalizers run in reverse order,
 * ensuring proper cleanup even if exceptions occur.
 */
@main def runDatabaseExample(): Unit = {
  println("=== Database Connection Example ===\n")

  val config = DbConfig("localhost", 5432, "myapp")

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    // Allocate the database resource. Because Database extends AutoCloseable,
    // its close() method is automatically registered as a finalizer.
    val db: $[Database] = allocate(Resource {
      val database = new Database(config)
      database.connect()
      database
    })

    // Use $(value)(f) to access the scoped value and execute queries.
    $(db) { database =>
      val users = database.query("SELECT * FROM users")
      println(s"[Result] Found ${users.size} users: ${users.rows.map(_("name")).mkString(", ")}\n")

      val orders = database.query("SELECT * FROM orders WHERE status = 'pending'")
      println(s"[Result] Found ${orders.size} orders\n")

      val health = database.query("SELECT 1 AS health_check")
      println(s"[Result] Health check: ${health.rows.head("result")}\n")
    }

    println("[Scope] Exiting scoped region - finalizers will run in LIFO order")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain scope.examples.runDatabaseExample"
```

### Shared Resources with Memoization and Reference Counting

This example demonstrates Resource.shared to create a singleton logger instance that is automatically cleaned up only when the last service releases it. Shows reference counting in action:

```scala title="scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `Wire.shared` vs `Wire.unique` and diamond dependency patterns.
 *
 * Two services (ProductService, OrderService) share one Logger instance
 * (diamond pattern), but each gets its own unique Cache instance. This shows
 * how shared wires provide singleton behavior while unique wires create fresh
 * instances per injection site.
 *
 * Key concepts:
 *   - `Wire.shared[T]`: Single instance shared across all dependents (memoized)
 *   - `Wire.unique[T]`: Fresh instance created for each dependent
 *   - Diamond dependency: Multiple services depend on the same shared resource
 *   - Reference counting: Shared resources track usage and clean up when last
 *     user closes
 */
object CachingSharedLoggerExample {

  /** Tracks instantiation counts for demonstration purposes. */
  val loggerInstances = new AtomicInteger(0)
  val cacheInstances  = new AtomicInteger(0)

  /**
   * A shared logger that tracks instantiations and provides logging methods.
   * Implements AutoCloseable for proper resource cleanup.
   */
  class Logger extends AutoCloseable {
    val instanceId: Int = loggerInstances.incrementAndGet()
    println(s"  [Logger#$instanceId] Created")

    def info(msg: String): Unit  = println(s"  [Logger#$instanceId] INFO: $msg")
    def debug(msg: String): Unit = println(s"  [Logger#$instanceId] DEBUG: $msg")
    def close(): Unit            = println(s"  [Logger#$instanceId] Closed")
  }

  /**
   * A unique cache per service. Each service gets its own isolated cache
   * instance. Implements AutoCloseable for proper resource cleanup. Note: No
   * constructor params so it can be auto-wired with Wire.unique.
   */
  class Cache extends AutoCloseable {
    val instanceId: Int                    = cacheInstances.incrementAndGet()
    private var store: Map[String, String] = Map.empty
    println(s"  [Cache#$instanceId] Created")

    def get(key: String): Option[String]      = store.get(key)
    def put(key: String, value: String): Unit = store = store.updated(key, value)
    def close(): Unit                         = println(s"  [Cache#$instanceId] Closed")
  }

  /** Product service with its own cache but sharing the logger. */
  class ProductService(val logger: Logger, val cache: Cache) {
    println(s"  [ProductService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def findProduct(id: String): String =
      cache.get(id) match {
        case Some(product) =>
          logger.debug(s"Cache hit for product $id")
          product
        case None =>
          logger.info(s"Loading product $id from database")
          val product = s"Product-$id"
          cache.put(id, product)
          product
      }
  }

  /**
   * Order service with its own cache but sharing the same logger as
   * ProductService.
   */
  class OrderService(val logger: Logger, val cache: Cache) {
    println(s"  [OrderService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def createOrder(productId: String): String = {
      val orderId = s"ORD-${System.currentTimeMillis() % 10000}"
      cache.put(orderId, productId)
      logger.info(s"Created order $orderId for product $productId")
      orderId
    }
  }

  /** Top-level application combining both services. */
  class CachingApp(val productService: ProductService, val orderService: OrderService) extends AutoCloseable {
    def run(): Unit = {
      productService.logger.info("=== Application Started ===")
      val product = productService.findProduct("P001")
      orderService.createOrder(product)
      productService.findProduct("P001") // cache hit
    }
    def close(): Unit = println("  [CachingApp] Closed")
  }

  @main def runCachingExample(): Unit = {
    println("\n╔════════════════════════════════════════════════════════════════╗")
    println("║  Wire.shared vs Wire.unique - Diamond Dependency Example       ║")
    println("╚════════════════════════════════════════════════════════════════╝\n")

    println("Creating wires...")
    println("  - Logger: Wire.shared (singleton across all services)")
    println("  - Cache:  Wire.unique (fresh instance per service)\n")

    println("─── Resource Acquisition ───")
    Scope.global.scoped { scope =>

      val app: $[CachingApp] = allocate(
        Resource.from[CachingApp](
          Wire.shared[Logger],
          Wire.unique[Cache]
        )
      )

      println("\n─── Verification ───")
      println(s"  Logger instances created: ${loggerInstances.get()} (expected: 1)")
      println(s"  Cache instances created:  ${cacheInstances.get()} (expected: 2)")
      $(app) { a =>
        println(s"  ProductService.logger eq OrderService.logger: ${a.productService.logger eq a.orderService.logger}")
        println(s"  ProductService.cache  eq OrderService.cache:  ${a.productService.cache eq a.orderService.cache}")

        println("\n─── Running Application ───")
        a.run()
      }

      println("\n─── Scope Closing (LIFO cleanup) ───")
    }

    println("\n─── Summary ───")
    println(s"  Final Logger count: ${loggerInstances.get()} (shared = 1 instance)")
    println(s"  Final Cache count:  ${cacheInstances.get()} (unique = 2 instances)")
    println("\nDiamond pattern verified: both services received the same Logger instance.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain scope.examples.runCachingExample"
```

### Managing Shared Expensive Resources

This example demonstrates using Resource.shared for a database connection pool—an expensive resource that should exist exactly once. Shows how multiple services safely share the same pool instance with automatic cleanup:

```scala title="scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `Resource.Shared` with reference counting and nested resource
 * acquisition.
 *
 * This example shows a realistic connection pool pattern where:
 *   - The pool itself is a shared resource (created once, ref-counted)
 *   - Individual connections are resources that must be allocated in a scope
 *   - `pool.acquire` returns `Resource[PooledConnection]`, forcing proper
 *     scoping
 *
 * This pattern is common for database pools, HTTP client pools, and thread
 * pools.
 */

/** Configuration for the connection pool. */
final case class PoolConfig(maxConnections: Int, timeout: Long)

/**
 * A connection retrieved from the pool.
 *
 * Connections are resources - they must be released back to the pool when done.
 * This is enforced by making `acquire` return a `Resource[PooledConnection]`.
 */
final class PooledConnection(val id: Int, pool: ConnectionPool) extends AutoCloseable {
  println(s"    [Conn#$id] Acquired from pool")

  def execute(sql: String): String = {
    println(s"    [Conn#$id] Executing: $sql")
    s"Result from connection $id"
  }

  override def close(): Unit =
    pool.release(this)
}

/**
 * A connection pool that manages pooled connections.
 *
 * Key design: `acquire` returns `Resource[PooledConnection]`, not a raw
 * connection. This forces callers to allocate the connection in a scope,
 * ensuring proper release even if exceptions occur.
 */
final class ConnectionPool(config: PoolConfig) extends AutoCloseable {
  private val nextId  = new AtomicInteger(0)
  private val active  = new AtomicInteger(0)
  private val _closed = new AtomicInteger(0)

  println(s"  [Pool] Created with max ${config.maxConnections} connections")

  /**
   * Acquires a connection from the pool.
   *
   * Returns a `Resource[PooledConnection]` that must be allocated in a scope.
   * The connection is automatically released when the scope exits.
   */
  def acquire: Resource[PooledConnection] = Resource.acquireRelease {
    if (_closed.get() > 0) throw new IllegalStateException("Pool is closed")
    if (active.get() >= config.maxConnections)
      throw new IllegalStateException(s"Pool exhausted (max: ${config.maxConnections})")

    val id   = nextId.incrementAndGet()
    val conn = new PooledConnection(id, this)
    active.incrementAndGet()
    println(s"  [Pool] Active connections: ${active.get()}/${config.maxConnections}")
    conn
  } { conn =>
    conn.close()
  }

  private[examples] def release(conn: PooledConnection): Unit = {
    val count = active.decrementAndGet()
    println(s"    [Conn#${conn.id}] Released back to pool (active: $count)")
  }

  def activeConnections: Int = active.get()

  override def close(): Unit =
    if (_closed.compareAndSet(0, 1)) {
      println(s"  [Pool] *** POOL CLOSED *** (served ${nextId.get()} total connections)")
    }
}

@main def connectionPoolExample(): Unit = {
  println("=== Connection Pool with Resource-based Acquire ===\n")

  val poolConfig = PoolConfig(maxConnections = 3, timeout = 5000L)

  val poolResource: Resource[ConnectionPool] =
    Resource.fromAutoCloseable(new ConnectionPool(poolConfig))

  Scope.global.scoped { appScope =>

    println("[App] Allocating pool\n")
    val pool: $[ConnectionPool] = poolResource.allocate

    println("--- ServiceA doing work (connection scoped to this block) ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_a_table"))
      println(s"  [ServiceA] Got: $result")
    }
    println()

    println("--- ServiceB doing work ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_b_table"))
      println(s"  [ServiceB] Got: $result")
    }
    println()

    println("--- Multiple connections in same scope ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val a: $[PooledConnection] = $(p)(_.acquire).allocate
      val b: $[PooledConnection] = $(p)(_.acquire).allocate
      val aId                    = $(a)(_.id)
      val bId                    = $(b)(_.id)
      println(s"  [Parallel] Using connections $aId and $bId")
      $(a)(_.execute("UPDATE table_a SET x = 1"))
      $(b)(_.execute("UPDATE table_b SET y = 2"))
      ()
    }
    println()

    println("[App] All work complete, exiting app scope...")
  }

  println("\n=== Example Complete ===")
  println("\nKey insight: pool.acquire returns Resource[PooledConnection],")
  println("forcing proper scoped allocation and automatic release.")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain scope.examples.connectionPoolExample"
```

### Transactional Resource Management

This example demonstrates combining Resource with transaction boundaries. Shows how to manage resources (connections, transactions) that must be coordinated across scopes with proper rollback on failure:

```scala title="scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Transaction Boundary Example
 *
 * Demonstrates nested scopes and resource-returning methods for database
 * transaction management.
 *
 * Key patterns shown:
 *   - '''Resource-returning methods''': `beginTransaction` returns
 *     `Resource[DbTransaction]`
 *   - '''Nested scopes''': Transactions live in child scopes of the connection
 *   - '''Automatic cleanup''': Uncommitted transactions auto-rollback on scope
 *     exit
 *   - '''LIFO ordering''': Transaction closes before connection
 */
object TransactionBoundaryExample {

  /** Simulates a database connection that can create transactions. */
  class DbConnection(val id: String) extends AutoCloseable {
    println(s"  [DbConnection $id] Opened")

    /**
     * Begins a new transaction.
     *
     * Returns a `Resource[DbTransaction]` that must be allocated in a scope.
     * This ensures the transaction is always properly closed (with rollback if
     * not committed) when the scope exits.
     */
    def beginTransaction(txId: String): Resource[DbTransaction] =
      Resource.acquireRelease {
        new DbTransaction(this, txId)
      } { tx =>
        tx.close()
      }

    def close(): Unit =
      println(s"  [DbConnection $id] Closed")
  }

  /** Simulates an active database transaction. */
  class DbTransaction(val conn: DbConnection, val id: String) extends AutoCloseable {
    private var committed  = false
    private var rolledBack = false
    println(s"    [Tx $id] Started on connection ${conn.id}")

    def execute(sql: String): Int = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      println(s"    [Tx $id] Execute: $sql")
      sql.hashCode.abs % 100 + 1
    }

    def commit(): Unit = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      committed = true
      println(s"    [Tx $id] Committed")
    }

    def rollback(): Unit =
      if (!committed && !rolledBack) {
        rolledBack = true
        println(s"    [Tx $id] Rolled back")
      }

    def close(): Unit = {
      if (!committed && !rolledBack) {
        println(s"    [Tx $id] Auto-rollback (not committed)")
        rollback()
      }
      println(s"    [Tx $id] Closed")
    }
  }

  /** Result of transaction operations. */
  case class TxResult(success: Boolean, affectedRows: Int) derives Unscoped

  @main def runTransactionBoundaryExample(): Unit = {
    println("=== Transaction Boundary Example ===\n")
    println("Demonstrating Resource-returning beginTransaction method\n")

    Scope.global.scoped { connScope =>

      // Allocate the connection in the outer scope
      val conn: $[DbConnection] = Resource.fromAutoCloseable(new DbConnection("db-001")).allocate
      println()

      // Transaction 1: Successful insert
      println("--- Transaction 1: Insert user ---")
      val result1: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-001")).allocate
          val rows                 = $(tx)(_.execute("INSERT INTO users VALUES (1, 'Alice')"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows)
        }
      println(s"  Result: $result1\n")

      // Transaction 2: Transfer funds (multiple operations)
      println("--- Transaction 2: Transfer funds ---")
      val result2: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-002")).allocate
          val rows1                = $(tx)(_.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1"))
          val rows2                = $(tx)(_.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows1 + rows2)
        }
      println(s"  Result: $result2\n")

      // Transaction 3: Demonstrates auto-rollback on scope exit without commit
      println("--- Transaction 3: Auto-rollback (no explicit commit) ---")
      val result3: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-003")).allocate
          $(tx)(_.execute("DELETE FROM audit_log"))
          println("    [App] Not committing - scope exit will trigger auto-rollback...")
          TxResult(success = false, affectedRows = 0)
        }
      println(s"  Result: $result3\n")

      println("--- All transactions complete, connection still open ---")
      println("--- Exiting connection scope ---")
    }

    println("\n=== Example complete ===")
    println("\nKey insight: beginTransaction() returns Resource[DbTransaction],")
    println("forcing proper scoped allocation and automatic cleanup.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain scope.examples.runTransactionBoundaryExample"
```

### Multi-Layer Service Construction

This example demonstrates Resource.from macro to automatically build a complex dependency graph with multiple services. Shows automatic wiring of constructor dependencies and cleanup in correct LIFO order:

```scala title="scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates auto-wiring a layered web service using
 * `Resource.from[T](wires*)`.
 *
 * The macro automatically derives wires for concrete classes (Database,
 * UserRepository, UserController) while requiring only the leaf config value to
 * be provided explicitly. Resources are cleaned up in LIFO order when the scope
 * closes.
 *
 * Layer hierarchy:
 * {{{
 *   AppConfig (leaf value via Wire)
 *       ↓
 *   Database (auto-wired, AutoCloseable)
 *       ↓
 *   UserRepository (auto-wired)
 *       ↓
 *   UserController (auto-wired, AutoCloseable)
 * }}}
 */

/** Application configuration - the leaf dependency provided via Wire(value). */
case class WebAppConfig(dbUrl: String, serverPort: Int)

/** Domain model for users. */
case class User(id: Long, name: String, email: String)

/** Database layer - acquires a connection and releases it on close. */
class WebDatabase(config: WebAppConfig) extends AutoCloseable {
  println(s"  [WebDatabase] Connecting to ${config.dbUrl}")

  def execute(sql: String): Int = {
    println(s"  [WebDatabase] Executing: $sql")
    1
  }

  def close(): Unit = println("  [WebDatabase] Connection closed")
}

/** Repository layer - provides data access using the database. */
class UserRepository(db: WebDatabase) {
  println("  [UserRepository] Initialized")

  private var nextId = 1L

  def findById(id: Long): Option[User] = {
    db.execute(s"SELECT * FROM users WHERE id = $id")
    if (id > 0) Some(User(id, "Alice", "alice@example.com")) else None
  }

  def save(user: User): Long = {
    db.execute(s"INSERT INTO users VALUES (${user.id}, '${user.name}', '${user.email}')")
    val id = nextId
    nextId += 1
    id
  }
}

/** Controller layer - handles HTTP requests using the repository. */
class UserController(repo: UserRepository) extends AutoCloseable {
  println("  [UserController] Ready to serve requests")

  def getUser(id: Long): String =
    repo.findById(id).map(u => s"User(${u.id}, ${u.name})").getOrElse("Not found")

  def createUser(name: String, email: String): String = {
    val id = repo.save(User(0, name, email))
    s"Created user with id=$id"
  }

  def close(): Unit = println("  [UserController] Shutting down")
}

/**
 * Entry point demonstrating the auto-wiring feature.
 *
 * Only `Wire(config)` is provided; the macro derives wires for Database,
 * UserRepository, and UserController from their constructors.
 */
@main def layeredWebServiceExample(): Unit = {
  val config = WebAppConfig(dbUrl = "jdbc:postgresql://localhost:5432/mydb", serverPort = 8080)

  println("=== Constructing layers (order: config → database → repository → controller) ===")

  // Resource.from auto-wires the entire dependency graph
  val controllerResource: Resource[UserController] = Resource.from[UserController](
    Wire(config)
  )

  // Allocate within a scoped block; cleanup runs on scope exit
  Scope.global.scoped { scope =>

    val controller: $[UserController] = allocate(controllerResource)

    println("\n=== Handling requests ===")
    println(s"  GET /users/1  → ${$(controller)(_.getUser(1))}")
    println(s"  POST /users   → ${$(controller)(_.createUser("Bob", "bob@example.com"))}")

    println("\n=== Scope closing (LIFO cleanup: controller → database) ===")
  }

  println("=== Done ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.layeredWebServiceExample"
```

---

## Scope(Resource-management)

`Scope` is a **compile-time safe resource lifecycle manager** that tags allocated values with a scope-specific type, preventing use-after-close at compile time. Each scope instance has a distinct `$[A]` type that is unique to that scope, making values from different scopes structurally incompatible. The `$` operator macro and `Unscoped` typeclass create multiple layers of compile-time protection, eliminating an entire class of lifetime bugs without runtime overhead.

`Scope`:
- Prevents resource leaks and use-after-close via compile-time type checking
- Allocates resources eagerly and runs finalizers deterministically in LIFO order
- Is purely synchronous with zero runtime overhead (scoped values erase to underlying types)

Here's the interface definition:

```scala
trait Scope {
  type $[+A]

  def scoped[A](f: Scope => A): A
  def allocate[A](resource: Resource[A]): $[A]
  def allocate(value: => AutoCloseable): $[AutoCloseable]
  def open(): $[OpenScope]
  def defer(f: => Unit): DeferHandle
  def lower[A](value: parent.$[A]): $[A]
  def isClosed: Boolean
  def isOwner: Boolean
}
```

## Motivation

Most resource bugs in Scala are "escape" bugs—scenarios where a resource is used outside of its intended lifetime, leading to undefined behavior, crashes, or data corruption:

- **Storing in fields:** You open a database connection and store it in a field, intending to close it in a finalizer. But if the finalizer runs before you're truly done with the connection, or if you forget to close it, the connection is silently used after closure.
- **Capturing in closures:** You create a file handle and pass it to an async framework via a callback. The callback might be invoked long after your scope has closed and the file has been released, causing the program to crash or silently read/write corrupted data.
- **Passing to untrusted code:** You pass a resource to a library function that might store a reference and use it later, outside your scope. You have no way to know when it's safe to close.
- **Mixing lifetimes:** In large codebases, it becomes unclear which scope owns which resource. A developer might use a resource in the wrong scope, or two scopes might try to close the same resource.

Scope addresses these with a *tight* design. Each design choice solves a specific problem and works together with the others:

1. **Compile-time leak prevention via type tagging** — Every scope has its own `$[A]` type, combined with the `$` macro that restricts how you can use values and the `Unscoped` typeclass that marks safe return types. Together, these prevent returning resources from their scope at compile time. No runtime wrapper objects needed.

2. **Zero runtime overhead** — Scoped values erase to the underlying type `A` at runtime (via casts). There's no boxing, no extra objects, no GC pressure. The compile-time safety is "free."

3. **Eager allocation** — Resources are acquired immediately when you call `allocate`, not deferred to some later point. This makes lifetimes predictable and your code matches your mental model.

4. **Deterministic, LIFO finalization** — Finalizers are guaranteed to run in reverse order of allocation when a scope closes. If acquisition order implies dependencies (common in resource hierarchies), cleanup order is automatically correct. Exceptions in finalizers are collected rather than stopping cleanup.

5. **Structured scopes with parent-child relationships** — Scopes form a hierarchy; children always close before parents. The `lower` operator lets you safely use parent-scoped values in children, since parent will outlive child.

If you've used `try/finally`, `Using`, or ZIO's `Scope`, this is the same problem space—but optimized for **synchronous code** with **compile-time boundaries**.

## Installation

Add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.33"
```

Supported Scala versions: **2.13.x** and **3.x**.

## Quickstart

Here's a minimal example showing resource allocation, usage, and cleanup. This example introduces a canonical `Database` stub that we'll reuse throughout this guide:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

val out: String =
  Scope.global.scoped { scope =>

    val db: $[Database] =
      Resource.fromAutoCloseable(new Database).allocate

    // Safe access: the lambda parameter can only be used as a receiver
    $(db)(_.query("SELECT 1"))
  }

println(out)
```

What's happening in this code:

**Allocating resources in a scope.** When you call `Resource.fromAutoCloseable(new Database).allocate`, you're acquiring a database connection. The `allocate` method returns a **scoped value** of type `scope.$[Database]`—notice the `$` wrapper. This type is unique to the `scope` instance. You can import the scope to use the short form `$[Database]`.

**The `$` operator restricts access.** You cannot call `db.query(...)` directly on `$[Database]` because the methods are hidden at the type level. Instead, you use the `$` access operator: `$(db)(f)`, which takes a lambda. The lambda's parameter must be used only as a receiver (for method/field access), preventing accidental capture or escape.

**Safe return from scoped.** The `scoped` block returns a plain `String` (the result of `_.query("SELECT 1")`). This is safe because `String` is marked as `Unscoped`—a typeclass that says "this type is pure data, safe to leave a scope." If you tried to return `db` instead, the compiler would error.

**LIFO cleanup.** When the `scoped` block exits (normally or via exception), all finalizers run in reverse order. The database's `close()` method is registered automatically because `Database` extends `AutoCloseable`. So cleanup happens at the right time, in the right order, even if an exception occurred.

## Safety Model

Scope's compile-time safety comes from *three reinforcing layers* that work together to prevent resource leaks.

1. **Type identity per scope.** Every scope has a distinct `$[A]` type. This makes values from different scopes **structurally incompatible** at compile time, so you cannot accidentally use a resource in the wrong scope without an explicit conversion (`lower` for parent → child). For example, `scope1.$[Database]` and `scope2.$[Database]` are different types—the compiler refuses to mix them:

```scala
// does not compile:
Scope.global.scoped { scope1 =>

  val db1 = allocate(new Database)

  scope1.scoped { scope2 =>

    val x: scope2.$[Database] = db1  // Error: type mismatch
    // scope2.$[Database] is not compatible with scope1.$[Database]
  }
}
```

To safely use a parent scope's resource in a child scope, use `lower`:

```scala
Scope.global.scoped { outer =>

  val db = allocate(new Database)

  outer.scoped { inner =>

    val dbInChild = inner.lower(db)  // ✓ Correct: retags for child scope
    $(dbInChild)(_.query("SELECT 1"))
  }
  // db is still alive here after child closes
}
```

2. **Controlled access via the `$` macro.** The `$` operator only allows using an unwrapped value as a **method/field receiver**. This prevents returning the resource, storing it in a local val/var, passing it as an argument to a function, or capturing it in a closure. The `$` macro also requires a **lambda literal** (not a method reference or variable):

```scala
// does not compile:
val f: Database => String = _.query("x")
(scope $ db)(f) // Error: "$ requires a lambda literal ..."
```

A lambda literal is an anonymous function written directly in code (e.g., `_.query("x")` or `x => x + 1`). The macro inspects the actual code you pass, so you must pass the lambda directly: `$(db)(_.query("x"))` compiles, but storing it in a variable first defeats this check. Without this restriction, you could smuggle the resource out indirectly via a stored function:

```scala
// hypothetical: if the macro didn't require a lambda literal
var leaked: Database = null

val f: Database => String = { db =>
  leaked = db  // Store the database somewhere the macro can't see
  db.query("x")
}

$(db)(f)  // Macro sees the call but can't detect the smuggling above

// After the scope closes, the resource is still accessible:
leaked.query("SELECT *")  // Use-after-close bug!
```

By requiring a lambda literal, the macro can analyze the actual code syntax. It rejects any attempt to store or capture the parameter, making smuggling impossible.

3. **Scope boundary enforcement via `Unscoped`.** A `scoped { ... }` block can only return values with an `Unscoped` instance (pure data). Resources and closures cannot escape the scope boundary at compile time. For example, trying to return a resource directly fails:

```scala
// does not compile:
Scope.global.scoped { scope =>

  val db = allocate(new Database)
  db  // Error: No given instance of Unscoped[$[Database]]
}
```

Closures over resources are also rejected:

```scala
// does not compile:
Scope.global.scoped { scope =>

  val db = allocate(new Database)
  () => db.query("SELECT 1")  // Error: No given instance of Unscoped[() => String]
  // (the closure captures db)
}
```

Only types with an `Unscoped` instance can cross the scope boundary—typically pure data:

```scala
Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(_.query("SELECT 1"))  // ✓ Correct: returns String, which is Unscoped
}
```

## Construction

### `Scope.global` — The Root Scope

`Scope.global` is the predefined root scope instance. It exists for the lifetime of your application and is the entry point for all scope-based resource management.

In `Scope.global`, the `$[A]` type is an identity type (i.e., `$[A] = A`). Finalizers registered in the global scope run on JVM shutdown via a shutdown hook. On Scala.js, global finalizers are not automatically invoked.

Use `Scope.global` to access the root scope:

```scala

val result: String = Scope.global.scoped { scope =>

  "no resources allocated"
}
```

### `Scope#scoped` — Create and Enter a Child Scope

`scoped` creates a new child scope with lexical lifetime. All resources allocated within the lambda are automatically cleaned up (LIFO) when the lambda exits, whether normally or via exception:

The lambda receives the child scope as a parameter. You can import its members to use the short form `$[A]` instead of `scope.$[A]`:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("database closed")
}

Scope.global.scoped { scope =>

  val db: $[Database] =
    Resource.fromAutoCloseable(new Database).allocate

  // Use the database within the scope
  val result = $(db)(_.query("SELECT 1"))
  result
  // db is automatically closed here (scope exits)
}
```

### `Scope#open` — Create an Unowned Child Scope

`open()` creates a child scope you explicitly close, returning an `OpenScope` handle. Unlike `scoped { }`, this allows non-lexical lifetime management:

The child scope is **unowned** (usable from any thread) but remains **linked to the parent** (if the parent closes, the child's finalizers also run). You must call `close()` to detach and finalize immediately.

This is useful for resource pools, lazy initialization, or service factories where you need to decouple resource acquisition from cleanup. Unlike `scoped { }`, which ties lifetime to a lexical block, `open()` lets you keep resources alive across function boundaries and explicit time boundaries.

Here's a practical application initialization pattern:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

// Application initialization: open resources early, return handle for later cleanup
val appResources = Scope.global.open()
val db = appResources.scope.allocate(Resource.fromAutoCloseable(new Database))

try {
  // Use database from anywhere in the application
  val result = appResources.scope.scoped { scope =>

    // Can create child scopes and use parent resources with lower()
    val dbInChild = scope.lower(db)
    $(dbInChild)(_.query("SELECT 1"))
  }

  println(s"Query result: $result")

  // ... rest of application code ...

} finally {
  // Application shutdown: explicit cleanup (decoupled from creation)
  appResources.close().orThrow()
}
```

## Core Operations

### `Scope#allocate` — Acquire a Resource

Allocates a `Resource[A]` in this scope, acquiring the underlying value immediately and registering its finalizer:

```scala
trait Scope {
  def allocate[A](resource: Resource[A]): $[A]
  def allocate[A <: AutoCloseable](value: => A): $[A]
}
```

The first overload accepts any `Resource`. The second is a convenience for `AutoCloseable` values—their `close()` method is automatically registered as a finalizer.

If the scope is already closed, `allocate` throws `IllegalStateException`. Otherwise, the resource is acquired eagerly and its finalizer is registered to run LIFO when the scope closes:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  // Using Resource factory
  val db1: $[Database] =
    Resource.fromAutoCloseable(new Database).allocate

  // Using AutoCloseable overload (convenience)
  val db2: $[Database] = allocate(new Database)

  // Both are equivalent; use whichever is more readable
  ()
}
```

### `$` — Access a Scoped Value

The `$` operator safely accesses a scoped value by enforcing it is only used as a method/field receiver, preventing accidental capture or escape:

**Single value:** Use infix or unqualified syntax:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(new Database)

  // Infix syntax
  val result1 = (scope $ db)(_.query("SELECT 1"))

  // Unqualified after `import scope._`
  val result2 = $(db)(_.query("SELECT 2"))

  result1 + result2
}
```

**Multiple values:** Use unqualified syntax only:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

final class Cache extends AutoCloseable {
  def key(): String = "cache_key"
  def close(): Unit = ()
}

Scope.global.scoped { scope =>

  val db: $[Database] = allocate(new Database)
  val cache: $[Cache] = allocate(new Cache)

  // Multiple values: each parameter may only be a receiver
  val result = $(db, cache)((d, c) => d.query(c.key()))
  result
}
```

The `$` macro enforces receiver-only rules at compile time:
- ✓ Allowed: `d.method()`, `d.method(c.key())` (method calls, field access)
- ✗ Rejected: `store(d)`, `() => d.method()`, `d` (returned), `{ val x = d; 1 }` (binding)

If a result type is `Unscoped[B]` (pure data), `$` auto-unwraps it to `B`. Otherwise, it returns `scope.$[B]`.

### `Scope#lower` — Use a Parent Value in a Child Scope

`lower` retagges a parent-scoped value into a child scope. This is safe because a parent scope always outlives its children:

This is useful when a child scope needs access to resources allocated in its parent:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { outer =>

  val db: $[Database] = allocate(new Database)

  // Create an inner scope that needs the database
  outer.scoped { inner =>

    // Retag the parent's database into the child
    val dbInChild: inner.$[Database] = inner.lower(db)

    // Now use it in the child
    $(dbInChild)(_.query("child query"))
  }
  // When inner exits, its finalizers run
  // When outer exits, db's finalizers run (still alive for the outer scope)
}
```

### `Finalizer#defer` — Register a Manual Finalizer

`defer` registers a cleanup action to run when the scope closes. It returns a `DeferHandle` that can cancel the registration:

```scala
trait Finalizer {
  def defer(f: => Unit): DeferHandle
}
```

`defer` is useful for resources that are not wrapped in `Resource`, or when you need explicit control over finalization. Here's a practical example—managing a temporary file and a logger that don't implement `AutoCloseable`:

```scala

// A logger that needs manual cleanup but doesn't implement AutoCloseable
class Logger {
  def log(msg: String): Unit = println(s"[LOG] $msg")
  def close(): Unit = println("Logger closed")
}

Scope.global.scoped { scope =>

  // Create a temporary file (not AutoCloseable from standard library)
  val tempFile = Files.createTempFile("app", ".tmp")
  defer {
    Files.deleteIfExists(tempFile)
    println(s"Temp file deleted: $tempFile")
  }

  // Create a logger (not AutoCloseable)
  val logger = new Logger
  val loggerHandle = defer(logger.close())

  // Use both resources
  logger.log("Processing file: " + tempFile)
  Files.write(tempFile, "data".getBytes())

  // If needed, cancel the finalizer and clean up manually
  val data = Files.readAllBytes(tempFile)
  logger.log(s"Read ${data.length} bytes")

  // loggerHandle.cancel()  // Would prevent auto-cleanup
}
// When the scope exits: logger closes, then temp file is deleted (LIFO order)
```

If the scope is already closed, `defer` is silently ignored (no-op). The finalizer is guaranteed to run in LIFO order with other finalizers when the scope closes.

### `Scope#isClosed` — Check If Closed

Returns whether this scope's finalizers have already run:

```scala
trait Scope {
  def isClosed: Boolean
}
```

Once `isClosed` returns `true`, subsequent calls to `allocate`, `open`, or `$` throw `IllegalStateException`. This is checked to prevent use-after-close bugs.

Here's a practical example—a resource manager that guards against using a closed scope:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

// A service that holds and manages a scope
class DatabaseService {
  private val serviceScope = Scope.global.open()

  // Initialize database once at startup
  private val db = {
    try {
      serviceScope.scope.allocate(Resource.fromAutoCloseable(new Database))
    } catch {
      case e: IllegalStateException =>
        serviceScope.close().orThrow()
        throw e
    }
  }

  def isAvailable: Boolean = !serviceScope.scope.isClosed

  def execute(query: String): Either[String, String] = {
    if (serviceScope.scope.isClosed) {
      Left("Database service has been shut down")
    } else {
      try {
        Right(serviceScope.scope.scoped { scope =>

          val dbInChild = scope.lower(db)
          $(dbInChild)(_.query(query))
        })
      } catch {
        case e: IllegalStateException => Left(s"Service error: ${e.getMessage}")
      }
    }
  }

  def shutdown(): Unit = {
    if (!serviceScope.scope.isClosed) {
      serviceScope.close().orThrow()
      println("Service shutdown complete")
    }
  }
}

// Usage
val service = new DatabaseService
println(s"Service available: ${service.isAvailable}")

val result1 = service.execute("SELECT 1")
println(s"Query result: $result1")

service.shutdown()

// Attempting to use after shutdown is now safe
val result2 = service.execute("SELECT 2")
println(s"Query after shutdown: $result2")
```

`Scope.global` returns `false` until JVM shutdown. Child scopes created with `scoped { }` are closed when the block exits, while those created with `open()` remain open until you call `close()`.

### `Scope#isOwner` — Check Thread Ownership

Returns whether the calling thread is the owner of this scope:

```scala
trait Scope {
  def isOwner: Boolean
}
```

Ownership is used to detect cross-thread scope misuse. Thread ownership rules:
- `Scope.global`: always returns `true` (any thread may use it)
- Child scopes created via `scoped { }`: returns `true` only on the thread that entered the block
- Child scopes created via `open()`: always returns `true` (unowned, usable cross-thread)

Calling `scoped { }` on a scope you don't own throws `IllegalStateException` at runtime:

```scala

Scope.global.scoped { scope =>
  // On the thread that entered scoped, isOwner is true
  assert(scope.isOwner)

  // On a different thread, isOwner returns false
  val thread = new Thread {
    override def run(): Unit = {
      assert(!scope.isOwner)
    }
  }
  thread.start()
  thread.join()
}
```

**Example 1: Thread-owned scope (scoped) — fails on worker thread**

Thread-owned scopes cannot be used to create child scopes from a different thread:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

val executor = Executors.newFixedThreadPool(1)

try {
  Scope.global.scoped { scope =>

    val db = allocate(new Database)

    // Try to create a child scope from a different thread
    val future = executor.submit { () =>
      try {
        scope.scoped { childScope =>

          val dbInChild = childScope.lower(db)
          $(dbInChild)(_.query("SELECT 1"))
        }
      } catch {
        case e: IllegalStateException => s"Error: ${e.getMessage}"
      }
    }
    println(future.get())
  }
} finally {
  executor.shutdown()
}

// Example Output:
// Error: Cannot create child scope: current thread 'pool-1-thread-1' does not own this scope (owner: 'main')
// db closed
```

**Example 2: Unowned scope (open) — works across threads**

Open scopes are unowned and usable from any thread:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

val executor = Executors.newFixedThreadPool(1)

try {
  val poolScope = Scope.global.open()
  val db = poolScope.scope.allocate(Resource.fromAutoCloseable(new Database))

  // Use the resource from a worker thread
  val future = executor.submit { () =>
    poolScope.scope.scoped { scope =>

      val dbInChild = scope.lower(db)
      $(dbInChild)(_.query("SELECT 1"))
    }
  }
  println(future.get())

  poolScope.close().orThrow()
} finally {
  executor.shutdown()
}

// Output:
// result: SELECT 1
// db closed
```

The key difference: `scoped { }` creates **owned** scopes (tied to the entering thread), while `open()` creates **unowned** scopes (usable from any thread). Choose based on whether your resources need to cross thread boundaries.

## Returning Unscoped Data from a Scope

A `scoped { }` block can only return values that have an `Unscoped` instance—that is, pure data types with no embedded resources or cleanup logic. This restriction prevents resource leaks: you cannot accidentally return a resource that would be cleaned up before you could use it.

For built-in types like `String`, `Int`, or `List[String]`, `Unscoped` instances exist automatically. However, when your custom type contains a field whose type has no predefined `Unscoped` instance (such as `java.util.Date`, a legacy Java type that is pure data but not automatically recognized), automatic derivation won't work. In such cases, you must provide an `Unscoped` instance explicitly, asserting that your type holds only pure data:

```scala

// java.util.Date has no predefined Unscoped instance, so Unscoped.derived
// won't work here — we must provide the instance explicitly
case class QueryResult(rows: List[String], count: Int, executedAt: Date)

object QueryResult {
  implicit val unscoped: Unscoped[QueryResult] = new Unscoped[QueryResult] {}
}

Scope.global.scoped { scope =>

  // ... acquire database ...
  QueryResult(List("a", "b"), 2, new Date())  // Returns safely
}
```

**Only add `Unscoped` for pure data types.** Never add it for types that hold resources (connections, streams, file handles). If you encounter the compile error [`No given instance of Unscoped[MyType]`](#no-given-instance-of-unscopedmytype--escaping-a-scope), see the compile errors section for how to fix it. For the complete API and examples, see the [Unscoped reference](./unscoped.md).

## Lexical vs Explicit Scopes

The Scope API provides two primary patterns for managing resource lifetimes. Choose `Scope#scoped` if you can write both the code that acquires and the code that releases the resource in the same expression; choose `Scope#open()` if the resource lifetime must outlive the function that creates it. Most user code should prefer `scoped` for automatic cleanup and thread safety—use `open()` only when you need manual lifetime control, such as in connection pools or DI containers.

### Lexical Scopes with `Scope#scoped`

Use `Scope#scoped` when the resource lifetime is lexically bounded. Lexical scopes are thread-owned by default, preventing accidental cross-thread access and providing automatic cleanup even on exception. This makes them safe and composable: you can nest `scoped` blocks to express hierarchical resource dependencies, and the code structure naturally matches the resource lifetime.

Here's a basic pattern showing how to acquire and use a resource within a single scope:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

Scope.global.scoped { scope =>

  val db = allocate(new Database)
  $(db)(_.query("SELECT * FROM users"))
  // db closes when scope exits
}
```

The only trade-off is that you must know the scope's lifetime upfront and cannot easily extend resource lifetime across function boundaries without returning resources themselves. For details on different allocation approaches (with `Resource.fromAutoCloseable()` or directly with `AutoCloseable`), see [Core Operations — allocate](#scopeallocate--acquire-a-resource).

#### Nesting for hierarchical resources

When resources depend on each other, nest `scoped` blocks to express the hierarchy. Parent scopes always outlive their children, so you can safely use parent resources in child scopes:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

final class Connection extends AutoCloseable {
  def close(): Unit = println("connection closed")
}

Scope.global.scoped { outerScope =>

  val db = allocate(new Database)

  // Child scope for connection
  outerScope.scoped { innerScope =>

    val conn = allocate(new Connection)
    // Use conn and db here
    // conn closes first (LIFO)
  }

  // Can still use db here
  // db closes when outerScope exits
}
```

### Explicit Scopes with `Scope#open`

Use `Scope#open()` when the resource lifetime is not lexically bounded. Open scopes are unowned (usable from any thread), which makes them suitable for patterns like connection pools, resource caches, and DI containers where resources must outlive the function that creates them. This pattern gives you full control over resource acquisition and release timing.

The trade-off is that you accept full responsibility for cleanup: forgetting to call `close()` leaves resources open, and any exception during cleanup must be explicitly handled. Here's the key pattern—returning an `OpenScope` handle from a function:

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

def acquireDatabase(): Scope.OpenScope = {
  val os = Scope.global.open()
  val _ = os.scope.allocate(Resource.fromAutoCloseable(new Database))
  os
}

val handle = acquireDatabase()
try {
  // Use handle.scope as needed
  ()
} finally {
  handle.close().orThrow()
}
```

## Dependency Injection

`Scope` integrates seamlessly with `Wire` and `Resource.from` for automatic dependency injection. `Wire` describes a recipe for constructing a service and its dependencies, while `Scope` manages the resource lifetime. Together they eliminate manual dependency passing and ensure proper cleanup in LIFO order.

Here's an example using `Wire` and `Resource.from` within a scope:

```scala

final class Config(val dbUrl: String)

final class Database(config: Config) extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println(s"db closed (${config.dbUrl})")
}

final class UserService(db: Database) {
  def getUser(id: Int): String = s"user $id from ${db.query("SELECT * FROM users")}"
}

final class App(service: UserService) {
  def run(): Unit = println(service.getUser(1))
}

// Wire describes the dependency graph: App -> UserService -> Database -> Config
// Resource.from uses the Wire to automatically construct the entire graph
Scope.global.scoped { scope =>

  val config = Config("jdbc:postgres://localhost/db")
  val app = allocate(Resource.from[App](
    Wire(config)
  ))
  $(app)(_.run())
  // All resources (Database, App) clean up automatically in reverse order
}
```

For more details on `Wire` sharing strategies, resource composition, and advanced DI patterns, see the [Wire reference](./wire.md) and [Resource reference](./resource.md).

## Best Practices

### Entry point pattern — use `Scope.global.scoped` at the top level

Wrap your entire application's resource acquisition in a single lexical scope:

```scala

object MyApp {
  def main(args: Array[String]): Unit = {
    Scope.global.scoped { scope =>

      // All resources acquired here
      // Automatic cleanup when main exits
    }
  }
}
```

This is your "outer boundary" for resource safety. Everything inside is protected.

### Composition — use `Resource` builders before allocation

Build resource acquisition/release logic outside the scope, then `Scope#allocate` once inside. This separates *construction* (how) from *allocation* (when), making code testable and reusable.

**Key combinators:**

- **`.map(f)`** — Transform a resource's value
- **`.flatMap(f)`** — Chain resources where the second depends on the first
- **`.zip(other)`** — Combine two independent resources

**Example: Using `.zip()` to combine independent resources:**

```scala

final class Database extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

final class Cache extends AutoCloseable {
  def get(key: String): Option[String] = None
  def close(): Unit = println("cache closed")
}

// Compose outside scope — reusable across multiple applications
val dbResource = Resource.fromAutoCloseable(new Database)
val cacheResource = Resource.fromAutoCloseable(new Cache)
val appResources = dbResource.zip(cacheResource)

// Allocate once inside scope
Scope.global.scoped { scope =>

  val (db, cache) = allocate(appResources)
  // Use both — cleanup happens in LIFO order (cache first, then db)
}
```

**Example: Using `.flatMap()` for dependent resources:**

```scala

final class Config(val host: String, val port: Int)

final class Database(val config: Config) extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = println("db closed")
}

// Config resource must be acquired first, then database
val configResource = Resource(new Config("localhost", 5432))
val dbResource = configResource.flatMap { cfg =>
  Resource.fromAutoCloseable(new Database(cfg))
}

// Allocate the dependent chain
Scope.global.scoped { scope =>

  val db = allocate(dbResource)
  // db was initialized with config; cleanup happens in reverse order
}
```

To learn more about building and composing resources, see the [Resource reference](./resource.md).

## Runtime Errors

Runtime errors occur when you violate scope rules at runtime—typically by accessing resources after the scope has already cleaned them up, or by mixing scopes across threads.

### `IllegalStateException` — accessing a closed scope

This error occurs when you attempt to acquire resources (via `allocate`, `open`, or the `$` operator) on a scope that has already closed. Every `scoped { }` block cleans up its resources as soon as the block exits, so any attempt to use the scope after that point fails.

The error message is typically:
```
Cannot acquire resource: scope has already been closed. ...
```

This usually happens when you:
- Store a scope in a field or closure and try to use it later after the enclosing `scoped` block has exited
- Accidentally pass a scope to an async operation that runs after cleanup

To avoid this, keep the scope's lifetime clear: allocate resources, use them, then let them clean up when the scope exits. If you need resources to survive longer, use `Scope.global.open()` to get a handle you can manage manually. You can also check `scope.isClosed` before attempting operations as a defensive check.

### `IllegalStateException` — cross-thread scope usage

Scopes are thread-owned by default. When you create a child scope using `scoped { }`, it's owned by the thread that created it. If you try to access that scope (allocate resources, create child scopes) from a different thread, the scope will reject it.

The error message is typically:
```
Cannot create child scope: current thread '...' does not own this scope (owner: '...')
```

This happens when you try to:
- Pass a scope to another thread and use it there
- Share a scope across multiple threads that call `scoped { }` on it

To fix this, use thread-unowned scopes when you need to share across threads. Instead of `scope.scoped { }` (which creates a thread-owned child), use `Scope.global.open()` or the scope's `open()` method directly to get an `OpenScope` handle. These unowned scopes can be safely passed and used from any thread, though you're responsible for manual cleanup via the returned handle.

## Compile Errors

The following compile errors occur when `Scope` type rules are violated. All examples below use this scoping pattern (see [Quickstart](#quickstart) for full context):

```scala
Scope.global.scoped { scope =>

  val db: $[Database] = allocate(new Database)
  // ... usage or error ...
}
```

### `No given instance of Unscoped[MyType]` — escaping a scope

This error occurs when the type you return from a `scoped { }` block has no `Unscoped` instance. See [Scope boundary enforcement via `Unscoped`](#safety-model) for the full explanation of why this restriction exists.

If you write:
```scala
db  // ERROR: No given instance of Unscoped[$[Database]]
```

The compiler rejects this because `db` is a scoped resource (type `$[Database]`), not safe data. Even though you're inside the `scoped` block, the type system prevents you from returning it because it would be useless outside the scope (the resource would already be cleaned up).

To fix this, you have two options:

**Option 1: Extract data from the resource before returning**

Call a method on the resource to get pure data (strings, numbers, etc.) that are naturally `Unscoped`:

```scala
$(db)(_.query("data"))  // ✓ Correct: Returns String, which is Unscoped
```

The `String` returned by `query()` is pure data with no cleanup logic, so it can safely escape the scope.

**Option 2: Implement `Unscoped` for your custom types**

If you create custom types that hold only pure data, add an `Unscoped` instance so they can escape scopes. See the [Unscoped reference](./unscoped.md) for details and examples.

### `Scoped values may only be used as a method receiver` — macro violation

**What this means:** A scoped value (the parameter inside a `$(value)` lambda) can only be used as the receiver of a method call—the object you call `.method()` on. It cannot be passed to other functions, stored in variables, or captured in nested lambdas. This restriction prevents the resource from leaking out of its scope and being used after cleanup.

**When you hit this error:**

The macro detects several violations:

- **Passing as an argument:**
  ```scala
  $(db)(d => store(d))    // ERROR: cannot pass scoped value to a function
  $(db)(d => println(d))  // ERROR: cannot pass to println
  ```

- **Storing in a variable:**
  ```scala
  $(db)(d => {
    val conn = d           // ERROR: cannot bind to val/var
    conn.query()
  })
  ```

- **Returning the value itself:**
  ```scala
  $(db)(d => d)           // ERROR: must call a method, not return bare reference
  ```

- **Capturing in a nested lambda or closure:**
  ```scala
  $(db)(d =>
    () => d.query()       // ERROR: cannot capture in nested lambda
  )
  ```

**What works — calling methods on the parameter:**

```scala
$(db)(d => d.query("SELECT * FROM users"))      // ✓ Method call on receiver
$(db)(_.query("data"))                           // ✓ Using underscore shorthand
$(db)(d => d.execute(statement).rows)           // ✓ Chain method calls
```

If you need to transform or extract data from a resource before using it elsewhere, call a method to extract what you need:

```scala
$(db)(d => d.query("SELECT COUNT(*)"))  // ✓ Returns String (pure data)
// The returned String can now be passed to other functions
```

**Why this restriction exists:** Scoped values are bound to a specific cleanup phase. Allowing them to escape (via arguments or closures) would let them be used after cleanup, causing crashes or data corruption. By restricting usage to method calls only, the macro ensures the resource never leaves its scope.

## Integration

Scope integrates seamlessly with ZIO Blocks' other data types for building complex resource management systems.

### Resource

`Scope` manages the lifecycle of `Resource[A]` values through the `allocate` method. A `Resource` describes how to acquire and clean up a value; `Scope` executes that plan and tracks finalizers. For comprehensive information on constructing, composing, and sharing resources, see the [Resource reference](./resource.md).

Key integration points:
- Use `Resource[A].allocate` to acquire within a scope
- Compose resources with `flatMap`, `andThen`, and other combinators before allocating
- Use `Resource.shared` for multiple-use resources within a scope

### Finalizer

`Scope` extends `Finalizer`, the interface for registering cleanup actions. The `defer` method registers a finalizer that runs when the scope closes. For information on the `DeferHandle` and cancellation, see the [Finalizer reference](./finalizer.md).

Key integration points:
- `scope.defer(f)` registers a cleanup action
- `DeferHandle.cancel()` prevents a finalizer from running
- Finalizers run in LIFO order regardless of whether registered via `allocate` or `defer`

### Wire + Resource.from

For dependency injection patterns, Scope works naturally with [`Wire`](./wire.md) and [`Resource.from`](./resource.md) to build layered service architectures. Allocate resources in a parent scope, then use `lower` to pass them to child scopes as needed.

## Running the Examples

All code from this guide is available as runnable examples in the `scope-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Database Connection Lifecycle Management

This example demonstrates how to allocate a database connection within a scope, ensure proper cleanup, and handle the connection's lifecycle safely.

```scala title="scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Configuration for database connection.
 *
 * @param host
 *   the database server hostname
 * @param port
 *   the database server port
 * @param database
 *   the database name to connect to
 */
final case class DbConfig(host: String, port: Int, database: String) {
  def connectionUrl: String = s"jdbc:postgresql://$host:$port/$database"
}

/**
 * Represents the result of a database query.
 *
 * @param rows
 *   the result set as a list of row maps
 */
final case class QueryResult(rows: List[Map[String, String]]) {
  def isEmpty: Boolean = rows.isEmpty
  def size: Int        = rows.size
}

/**
 * Simulates a database connection with lifecycle management.
 *
 * This class demonstrates how AutoCloseable resources integrate with ZIO Blocks
 * Scope. When allocated via `allocate(Resource(...))`, the `close()` method is
 * automatically registered as a finalizer.
 *
 * @param config
 *   the database configuration
 */
final class Database(config: DbConfig) extends AutoCloseable {
  private var connected = false

  def connect(): Unit = {
    println(s"[Database] Connecting to ${config.connectionUrl}...")
    connected = true
    println(s"[Database] Connected successfully")
  }

  def query(sql: String): QueryResult = {
    require(connected, "Database not connected")
    println(s"[Database] Executing: $sql")
    sql match {
      case s if s.contains("users") =>
        QueryResult(
          List(
            Map("id" -> "1", "name" -> "Alice"),
            Map("id" -> "2", "name" -> "Bob")
          )
        )
      case s if s.contains("orders") =>
        QueryResult(
          List(
            Map("order_id" -> "101", "user_id" -> "1", "total" -> "99.99"),
            Map("order_id" -> "102", "user_id" -> "2", "total" -> "149.50")
          )
        )
      case _ =>
        QueryResult(List(Map("result" -> "OK")))
    }
  }

  override def close(): Unit = {
    println(s"[Database] Closing connection to ${config.connectionUrl}")
    connected = false
  }
}

/**
 * Demonstrates basic resource lifecycle management with ZIO Blocks Scope.
 *
 * This example shows:
 *   - Allocating an AutoCloseable resource with automatic cleanup
 *   - Using `$(value)(f)` to access scoped values and execute queries
 *   - LIFO finalizer ordering (last allocated = first closed)
 *
 * When the scope exits, all registered finalizers run in reverse order,
 * ensuring proper cleanup even if exceptions occur.
 */
@main def runDatabaseExample(): Unit = {
  println("=== Database Connection Example ===\n")

  val config = DbConfig("localhost", 5432, "myapp")

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    // Allocate the database resource. Because Database extends AutoCloseable,
    // its close() method is automatically registered as a finalizer.
    val db: $[Database] = allocate(Resource {
      val database = new Database(config)
      database.connect()
      database
    })

    // Use $(value)(f) to access the scoped value and execute queries.
    $(db) { database =>
      val users = database.query("SELECT * FROM users")
      println(s"[Result] Found ${users.size} users: ${users.rows.map(_("name")).mkString(", ")}\n")

      val orders = database.query("SELECT * FROM orders WHERE status = 'pending'")
      println(s"[Result] Found ${orders.size} orders\n")

      val health = database.query("SELECT 1 AS health_check")
      println(s"[Result] Health check: ${health.rows.head("result")}\n")
    }

    println("[Scope] Exiting scoped region - finalizers will run in LIFO order")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/DatabaseConnectionExample.scala))

```bash
sbt "scope-examples/runMain runDatabaseExample"
```

### Managing a Connection Pool with Multiple Allocations

This example demonstrates allocating multiple connections from a pool within the same scope and ensuring all are cleaned up correctly.

```scala title="scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `Resource.Shared` with reference counting and nested resource
 * acquisition.
 *
 * This example shows a realistic connection pool pattern where:
 *   - The pool itself is a shared resource (created once, ref-counted)
 *   - Individual connections are resources that must be allocated in a scope
 *   - `pool.acquire` returns `Resource[PooledConnection]`, forcing proper
 *     scoping
 *
 * This pattern is common for database pools, HTTP client pools, and thread
 * pools.
 */

/** Configuration for the connection pool. */
final case class PoolConfig(maxConnections: Int, timeout: Long)

/**
 * A connection retrieved from the pool.
 *
 * Connections are resources - they must be released back to the pool when done.
 * This is enforced by making `acquire` return a `Resource[PooledConnection]`.
 */
final class PooledConnection(val id: Int, pool: ConnectionPool) extends AutoCloseable {
  println(s"    [Conn#$id] Acquired from pool")

  def execute(sql: String): String = {
    println(s"    [Conn#$id] Executing: $sql")
    s"Result from connection $id"
  }

  override def close(): Unit =
    pool.release(this)
}

/**
 * A connection pool that manages pooled connections.
 *
 * Key design: `acquire` returns `Resource[PooledConnection]`, not a raw
 * connection. This forces callers to allocate the connection in a scope,
 * ensuring proper release even if exceptions occur.
 */
final class ConnectionPool(config: PoolConfig) extends AutoCloseable {
  private val nextId  = new AtomicInteger(0)
  private val active  = new AtomicInteger(0)
  private val _closed = new AtomicInteger(0)

  println(s"  [Pool] Created with max ${config.maxConnections} connections")

  /**
   * Acquires a connection from the pool.
   *
   * Returns a `Resource[PooledConnection]` that must be allocated in a scope.
   * The connection is automatically released when the scope exits.
   */
  def acquire: Resource[PooledConnection] = Resource.acquireRelease {
    if (_closed.get() > 0) throw new IllegalStateException("Pool is closed")
    if (active.get() >= config.maxConnections)
      throw new IllegalStateException(s"Pool exhausted (max: ${config.maxConnections})")

    val id   = nextId.incrementAndGet()
    val conn = new PooledConnection(id, this)
    active.incrementAndGet()
    println(s"  [Pool] Active connections: ${active.get()}/${config.maxConnections}")
    conn
  } { conn =>
    conn.close()
  }

  private[examples] def release(conn: PooledConnection): Unit = {
    val count = active.decrementAndGet()
    println(s"    [Conn#${conn.id}] Released back to pool (active: $count)")
  }

  def activeConnections: Int = active.get()

  override def close(): Unit =
    if (_closed.compareAndSet(0, 1)) {
      println(s"  [Pool] *** POOL CLOSED *** (served ${nextId.get()} total connections)")
    }
}

@main def connectionPoolExample(): Unit = {
  println("=== Connection Pool with Resource-based Acquire ===\n")

  val poolConfig = PoolConfig(maxConnections = 3, timeout = 5000L)

  val poolResource: Resource[ConnectionPool] =
    Resource.fromAutoCloseable(new ConnectionPool(poolConfig))

  Scope.global.scoped { appScope =>

    println("[App] Allocating pool\n")
    val pool: $[ConnectionPool] = poolResource.allocate

    println("--- ServiceA doing work (connection scoped to this block) ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_a_table"))
      println(s"  [ServiceA] Got: $result")
    }
    println()

    println("--- ServiceB doing work ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val c: $[PooledConnection] = $(p)(_.acquire).allocate
      val result                 = $(c)(_.execute("SELECT * FROM service_b_table"))
      println(s"  [ServiceB] Got: $result")
    }
    println()

    println("--- Multiple connections in same scope ---")
    appScope.scoped { workScope =>

      val p: $[ConnectionPool]   = lower(pool)
      val a: $[PooledConnection] = $(p)(_.acquire).allocate
      val b: $[PooledConnection] = $(p)(_.acquire).allocate
      val aId                    = $(a)(_.id)
      val bId                    = $(b)(_.id)
      println(s"  [Parallel] Using connections $aId and $bId")
      $(a)(_.execute("UPDATE table_a SET x = 1"))
      $(b)(_.execute("UPDATE table_b SET y = 2"))
      ()
    }
    println()

    println("[App] All work complete, exiting app scope...")
  }

  println("\n=== Example Complete ===")
  println("\nKey insight: pool.acquire returns Resource[PooledConnection],")
  println("forcing proper scoped allocation and automatic release.")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ConnectionPoolExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.connectionPoolExample"
```

### Handling Temporary File Resources with Automatic Cleanup

This example shows how to allocate temporary file resources and ensure they are automatically cleaned up when the scope closes, even if errors occur.

```scala title="scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `scope.defer(...)` for registering manual cleanup actions.
 *
 * This example shows how to create temporary files during processing and ensure
 * they are deleted when the scope exits—even if processing fails. Deferred
 * cleanup actions run in LIFO (last-in-first-out) order.
 */

/** Represents a temporary file with basic read/write operations. */
case class TempFile(path: String) {
  private var content: String = ""

  def write(data: String): Unit = content = data
  def read(): String            = content
  def delete(): Boolean         = { println(s"  Deleting: $path"); true }
}

/** Result of processing temporary files. */
case class ProcessingResult(processedCount: Int, totalBytes: Long, errors: List[String])

/** Processes a list of temporary files and aggregates results. */
object FileProcessor {
  def process(files: List[TempFile]): ProcessingResult = {
    val totalBytes = files.map(_.read().length.toLong).sum
    ProcessingResult(processedCount = files.size, totalBytes = totalBytes, errors = Nil)
  }
}

@main def tempFileHandlingExample(): Unit = {
  println("=== Temp File Handling Example ===\n")
  println("Demonstrating scope.defer() for manual cleanup registration.\n")

  val result = Scope.global.scoped { scope =>
    // Create temp files and register cleanup via defer.
    // Cleanup runs in LIFO order: file3, file2, file1.

    val file1 = createTempFile(scope, "/tmp/data-001.tmp", "First file content")
    val file2 = createTempFile(scope, "/tmp/data-002.tmp", "Second file - more data here")
    val file3 = createTempFile(scope, "/tmp/data-003.tmp", "Third file with the most content of all")

    println("\nProcessing files...")
    val processingResult = FileProcessor.process(List(file1, file2, file3))
    println(s"Processed ${processingResult.processedCount} files, ${processingResult.totalBytes} bytes\n")

    println("Exiting scope - cleanup runs in LIFO order:")
    processingResult
  }

  println(s"\nFinal result: $result")
}

/**
 * Creates a temporary file and registers its cleanup with the scope.
 *
 * The cleanup action is registered via `defer(...)`, ensuring the file is
 * deleted when the scope closes—regardless of whether processing succeeds.
 *
 * @param s
 *   the scope to register cleanup with
 * @param path
 *   the file path
 * @param content
 *   initial content to write
 * @return
 *   the created TempFile
 */
private def createTempFile(s: Scope, path: String, content: String): TempFile = {
  val file = TempFile(path)
  file.write(content)
  println(s"Created: $path (${content.length} bytes)")

  // Register cleanup - will run when scope exits, in LIFO order
  s.defer {
    file.delete()
  }

  file
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TempFileHandlingExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.tempFileHandlingExample"
```

### Managing Database Transactions with Commit/Rollback Semantics

This example demonstrates managing database transactions within a scope, showing how to handle commit and rollback operations correctly.

```scala title="scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Transaction Boundary Example
 *
 * Demonstrates nested scopes and resource-returning methods for database
 * transaction management.
 *
 * Key patterns shown:
 *   - '''Resource-returning methods''': `beginTransaction` returns
 *     `Resource[DbTransaction]`
 *   - '''Nested scopes''': Transactions live in child scopes of the connection
 *   - '''Automatic cleanup''': Uncommitted transactions auto-rollback on scope
 *     exit
 *   - '''LIFO ordering''': Transaction closes before connection
 */
object TransactionBoundaryExample {

  /** Simulates a database connection that can create transactions. */
  class DbConnection(val id: String) extends AutoCloseable {
    println(s"  [DbConnection $id] Opened")

    /**
     * Begins a new transaction.
     *
     * Returns a `Resource[DbTransaction]` that must be allocated in a scope.
     * This ensures the transaction is always properly closed (with rollback if
     * not committed) when the scope exits.
     */
    def beginTransaction(txId: String): Resource[DbTransaction] =
      Resource.acquireRelease {
        new DbTransaction(this, txId)
      } { tx =>
        tx.close()
      }

    def close(): Unit =
      println(s"  [DbConnection $id] Closed")
  }

  /** Simulates an active database transaction. */
  class DbTransaction(val conn: DbConnection, val id: String) extends AutoCloseable {
    private var committed  = false
    private var rolledBack = false
    println(s"    [Tx $id] Started on connection ${conn.id}")

    def execute(sql: String): Int = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      println(s"    [Tx $id] Execute: $sql")
      sql.hashCode.abs % 100 + 1
    }

    def commit(): Unit = {
      require(!committed && !rolledBack, s"Transaction $id already completed")
      committed = true
      println(s"    [Tx $id] Committed")
    }

    def rollback(): Unit =
      if (!committed && !rolledBack) {
        rolledBack = true
        println(s"    [Tx $id] Rolled back")
      }

    def close(): Unit = {
      if (!committed && !rolledBack) {
        println(s"    [Tx $id] Auto-rollback (not committed)")
        rollback()
      }
      println(s"    [Tx $id] Closed")
    }
  }

  /** Result of transaction operations. */
  case class TxResult(success: Boolean, affectedRows: Int) derives Unscoped

  @main def runTransactionBoundaryExample(): Unit = {
    println("=== Transaction Boundary Example ===\n")
    println("Demonstrating Resource-returning beginTransaction method\n")

    Scope.global.scoped { connScope =>

      // Allocate the connection in the outer scope
      val conn: $[DbConnection] = Resource.fromAutoCloseable(new DbConnection("db-001")).allocate
      println()

      // Transaction 1: Successful insert
      println("--- Transaction 1: Insert user ---")
      val result1: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-001")).allocate
          val rows                 = $(tx)(_.execute("INSERT INTO users VALUES (1, 'Alice')"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows)
        }
      println(s"  Result: $result1\n")

      // Transaction 2: Transfer funds (multiple operations)
      println("--- Transaction 2: Transfer funds ---")
      val result2: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-002")).allocate
          val rows1                = $(tx)(_.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1"))
          val rows2                = $(tx)(_.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2"))
          $(tx)(_.commit())
          TxResult(success = true, affectedRows = rows1 + rows2)
        }
      println(s"  Result: $result2\n")

      // Transaction 3: Demonstrates auto-rollback on scope exit without commit
      println("--- Transaction 3: Auto-rollback (no explicit commit) ---")
      val result3: TxResult =
        connScope.scoped { txScope =>

          val c: $[DbConnection]   = lower(conn)
          val tx: $[DbTransaction] = $(c)(_.beginTransaction("tx-003")).allocate
          $(tx)(_.execute("DELETE FROM audit_log"))
          println("    [App] Not committing - scope exit will trigger auto-rollback...")
          TxResult(success = false, affectedRows = 0)
        }
      println(s"  Result: $result3\n")

      println("--- All transactions complete, connection still open ---")
      println("--- Exiting connection scope ---")
    }

    println("\n=== Example complete ===")
    println("\nKey insight: beginTransaction() returns Resource[DbTransaction],")
    println("forcing proper scoped allocation and automatic cleanup.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/TransactionBoundaryExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.runTransactionBoundaryExample"
```

### Implementing an HTTP Client Pipeline with Request/Response Interceptors

This example shows how to build an HTTP client pipeline with interceptors for logging, authentication, and error handling, all managed within a scope.

```scala title="scope-examples/src/main/scala/scope/examples/HttpClientPipelineExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * HTTP Client Pipeline Example
 *
 * Demonstrates using scoped values with the `$` operator for safe resource
 * access. Operations are eager with the new opaque type API.
 */

/** API configuration containing base URL and authentication credentials. */
final case class ApiConfig(baseUrl: String, apiKey: String)

/** Parsed JSON data as a simple key-value store. */
final case class ParsedData(values: Map[String, String]) derives Unscoped

/** HTTP response containing status, body, and headers. */
final case class HttpResponse(statusCode: Int, body: String, headers: Map[String, String])

/** Stateless JSON parser that converts raw JSON strings to structured data. */
object JsonParser {
  def parse(json: String): ParsedData = {
    println(s"  [JsonParser] Parsing ${json.take(50)}...")
    val entries = json.stripPrefix("{").stripSuffix("}").split(",").map(_.trim).filter(_.nonEmpty)
    val values  = entries.flatMap { entry =>
      entry.split(":").map(_.trim.stripPrefix("\"").stripSuffix("\"")) match {
        case Array(k, v) => Some(k -> v)
        case _           => None
      }
    }.toMap
    ParsedData(values)
  }
}

/**
 * HTTP client that manages a connection to an API server.
 *
 * Implements `AutoCloseable` so the scope automatically registers cleanup.
 */
final class HttpClient(config: ApiConfig) extends AutoCloseable {
  println(s"  [HttpClient] Opening connection to ${config.baseUrl}")

  def get(path: String): HttpResponse = {
    println(s"  [HttpClient] GET $path")
    HttpResponse(200, s"""{"path":"$path","data":"sample"}""", Map("X-Api-Key" -> config.apiKey))
  }

  def post(path: String, body: String): HttpResponse = {
    println(s"  [HttpClient] POST $path with body: $body")
    HttpResponse(201, s"""{"created":true,"echo":"$body"}""", Map("Content-Type" -> "application/json"))
  }

  override def close(): Unit =
    println(s"  [HttpClient] Closing connection to ${config.baseUrl}")
}

/**
 * Demonstrates using scoped values with the `$` operator.
 *
 * Key concepts:
 *   - `allocate` returns `$[A]` (scoped value)
 *   - `$(scopedValue)(f)` applies a function to the underlying value
 *   - `$` auto-unwraps to pure data when the return type is `Unscoped`
 *   - Operations are eager (zero-cost wrapper)
 */
@main def httpClientPipelineExample(): Unit = {
  println("=== HTTP Client Pipeline Example ===\n")
  val config = ApiConfig("https://api.example.com", "secret-key-123")

  Scope.global.scoped { scope =>

    // Step 1: Allocate the HTTP client (automatically cleaned up when scope closes)
    val client: $[HttpClient] = allocate(Resource[HttpClient](new HttpClient(config)))

    // Step 2: Use the client to fetch and parse data
    println("Executing requests...\n")

    // Fetch and parse users
    println("--- Fetching: users ---")
    val users: ParsedData = $(client) { c =>
      val response = c.get("/users")
      JsonParser.parse(response.body)
    }

    // Fetch and parse orders
    println("\n--- Fetching: orders ---")
    val orders: ParsedData = $(client) { c =>
      val response = c.get("/orders")
      JsonParser.parse(response.body)
    }

    // Post analytics event
    println("\n--- Posting: analytics ---")
    val analytics: ParsedData = $(client) { c =>
      val response = c.post("/analytics", """{"event":"fetch_complete"}""")
      JsonParser.parse(response.body)
    }

    // Step 3: Access all results
    println(s"\n=== Users Result ===")
    println(s"Users data: ${users.values}")
    println(s"\n=== Orders Result ===")
    println(s"Orders data: ${orders.values}")
    println(s"\n=== Analytics Result ===")
    println(s"Analytics: ${analytics.values}")
  }

  println("\n[Scope closed - HttpClient was automatically cleaned up]")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/HttpClientPipelineExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.httpClientPipelineExample"
```

### Managing a Shared, Cached Logger Across Multiple Services

This example demonstrates allocating a logger once at the top level and sharing it across multiple services, ensuring it is properly closed when the application shuts down.

```scala title="scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `Wire.shared` vs `Wire.unique` and diamond dependency patterns.
 *
 * Two services (ProductService, OrderService) share one Logger instance
 * (diamond pattern), but each gets its own unique Cache instance. This shows
 * how shared wires provide singleton behavior while unique wires create fresh
 * instances per injection site.
 *
 * Key concepts:
 *   - `Wire.shared[T]`: Single instance shared across all dependents (memoized)
 *   - `Wire.unique[T]`: Fresh instance created for each dependent
 *   - Diamond dependency: Multiple services depend on the same shared resource
 *   - Reference counting: Shared resources track usage and clean up when last
 *     user closes
 */
object CachingSharedLoggerExample {

  /** Tracks instantiation counts for demonstration purposes. */
  val loggerInstances = new AtomicInteger(0)
  val cacheInstances  = new AtomicInteger(0)

  /**
   * A shared logger that tracks instantiations and provides logging methods.
   * Implements AutoCloseable for proper resource cleanup.
   */
  class Logger extends AutoCloseable {
    val instanceId: Int = loggerInstances.incrementAndGet()
    println(s"  [Logger#$instanceId] Created")

    def info(msg: String): Unit  = println(s"  [Logger#$instanceId] INFO: $msg")
    def debug(msg: String): Unit = println(s"  [Logger#$instanceId] DEBUG: $msg")
    def close(): Unit            = println(s"  [Logger#$instanceId] Closed")
  }

  /**
   * A unique cache per service. Each service gets its own isolated cache
   * instance. Implements AutoCloseable for proper resource cleanup. Note: No
   * constructor params so it can be auto-wired with Wire.unique.
   */
  class Cache extends AutoCloseable {
    val instanceId: Int                    = cacheInstances.incrementAndGet()
    private var store: Map[String, String] = Map.empty
    println(s"  [Cache#$instanceId] Created")

    def get(key: String): Option[String]      = store.get(key)
    def put(key: String, value: String): Unit = store = store.updated(key, value)
    def close(): Unit                         = println(s"  [Cache#$instanceId] Closed")
  }

  /** Product service with its own cache but sharing the logger. */
  class ProductService(val logger: Logger, val cache: Cache) {
    println(s"  [ProductService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def findProduct(id: String): String =
      cache.get(id) match {
        case Some(product) =>
          logger.debug(s"Cache hit for product $id")
          product
        case None =>
          logger.info(s"Loading product $id from database")
          val product = s"Product-$id"
          cache.put(id, product)
          product
      }
  }

  /**
   * Order service with its own cache but sharing the same logger as
   * ProductService.
   */
  class OrderService(val logger: Logger, val cache: Cache) {
    println(s"  [OrderService] Created with Logger#${logger.instanceId} and Cache#${cache.instanceId}")

    def createOrder(productId: String): String = {
      val orderId = s"ORD-${System.currentTimeMillis() % 10000}"
      cache.put(orderId, productId)
      logger.info(s"Created order $orderId for product $productId")
      orderId
    }
  }

  /** Top-level application combining both services. */
  class CachingApp(val productService: ProductService, val orderService: OrderService) extends AutoCloseable {
    def run(): Unit = {
      productService.logger.info("=== Application Started ===")
      val product = productService.findProduct("P001")
      orderService.createOrder(product)
      productService.findProduct("P001") // cache hit
    }
    def close(): Unit = println("  [CachingApp] Closed")
  }

  @main def runCachingExample(): Unit = {
    println("\n╔════════════════════════════════════════════════════════════════╗")
    println("║  Wire.shared vs Wire.unique - Diamond Dependency Example       ║")
    println("╚════════════════════════════════════════════════════════════════╝\n")

    println("Creating wires...")
    println("  - Logger: Wire.shared (singleton across all services)")
    println("  - Cache:  Wire.unique (fresh instance per service)\n")

    println("─── Resource Acquisition ───")
    Scope.global.scoped { scope =>

      val app: $[CachingApp] = allocate(
        Resource.from[CachingApp](
          Wire.shared[Logger],
          Wire.unique[Cache]
        )
      )

      println("\n─── Verification ───")
      println(s"  Logger instances created: ${loggerInstances.get()} (expected: 1)")
      println(s"  Cache instances created:  ${cacheInstances.get()} (expected: 2)")
      $(app) { a =>
        println(s"  ProductService.logger eq OrderService.logger: ${a.productService.logger eq a.orderService.logger}")
        println(s"  ProductService.cache  eq OrderService.cache:  ${a.productService.cache eq a.orderService.cache}")

        println("\n─── Running Application ───")
        a.run()
      }

      println("\n─── Scope Closing (LIFO cleanup) ───")
    }

    println("\n─── Summary ───")
    println(s"  Final Logger count: ${loggerInstances.get()} (shared = 1 instance)")
    println(s"  Final Cache count:  ${cacheInstances.get()} (unique = 2 instances)")
    println("\nDiamond pattern verified: both services received the same Logger instance.")
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/CachingSharedLoggerExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.runCachingExample"
```

### Building a Layered Web Service with Dependency Injection

This example shows how to build a multi-layered web service using Scope for dependency injection, allocating services at different layers and passing them down through child scopes.

```scala title="scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates auto-wiring a layered web service using
 * `Resource.from[T](wires*)`.
 *
 * The macro automatically derives wires for concrete classes (Database,
 * UserRepository, UserController) while requiring only the leaf config value to
 * be provided explicitly. Resources are cleaned up in LIFO order when the scope
 * closes.
 *
 * Layer hierarchy:
 * {{{
 *   AppConfig (leaf value via Wire)
 *       ↓
 *   Database (auto-wired, AutoCloseable)
 *       ↓
 *   UserRepository (auto-wired)
 *       ↓
 *   UserController (auto-wired, AutoCloseable)
 * }}}
 */

/** Application configuration - the leaf dependency provided via Wire(value). */
case class WebAppConfig(dbUrl: String, serverPort: Int)

/** Domain model for users. */
case class User(id: Long, name: String, email: String)

/** Database layer - acquires a connection and releases it on close. */
class WebDatabase(config: WebAppConfig) extends AutoCloseable {
  println(s"  [WebDatabase] Connecting to ${config.dbUrl}")

  def execute(sql: String): Int = {
    println(s"  [WebDatabase] Executing: $sql")
    1
  }

  def close(): Unit = println("  [WebDatabase] Connection closed")
}

/** Repository layer - provides data access using the database. */
class UserRepository(db: WebDatabase) {
  println("  [UserRepository] Initialized")

  private var nextId = 1L

  def findById(id: Long): Option[User] = {
    db.execute(s"SELECT * FROM users WHERE id = $id")
    if (id > 0) Some(User(id, "Alice", "alice@example.com")) else None
  }

  def save(user: User): Long = {
    db.execute(s"INSERT INTO users VALUES (${user.id}, '${user.name}', '${user.email}')")
    val id = nextId
    nextId += 1
    id
  }
}

/** Controller layer - handles HTTP requests using the repository. */
class UserController(repo: UserRepository) extends AutoCloseable {
  println("  [UserController] Ready to serve requests")

  def getUser(id: Long): String =
    repo.findById(id).map(u => s"User(${u.id}, ${u.name})").getOrElse("Not found")

  def createUser(name: String, email: String): String = {
    val id = repo.save(User(0, name, email))
    s"Created user with id=$id"
  }

  def close(): Unit = println("  [UserController] Shutting down")
}

/**
 * Entry point demonstrating the auto-wiring feature.
 *
 * Only `Wire(config)` is provided; the macro derives wires for Database,
 * UserRepository, and UserController from their constructors.
 */
@main def layeredWebServiceExample(): Unit = {
  val config = WebAppConfig(dbUrl = "jdbc:postgresql://localhost:5432/mydb", serverPort = 8080)

  println("=== Constructing layers (order: config → database → repository → controller) ===")

  // Resource.from auto-wires the entire dependency graph
  val controllerResource: Resource[UserController] = Resource.from[UserController](
    Wire(config)
  )

  // Allocate within a scoped block; cleanup runs on scope exit
  Scope.global.scoped { scope =>

    val controller: $[UserController] = allocate(controllerResource)

    println("\n=== Handling requests ===")
    println(s"  GET /users/1  → ${$(controller)(_.getUser(1))}")
    println(s"  POST /users   → ${$(controller)(_.createUser("Bob", "bob@example.com"))}")

    println("\n=== Scope closing (LIFO cleanup: controller → database) ===")
  }

  println("=== Done ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/LayeredWebServiceExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.layeredWebServiceExample"
```

### Reading Configuration from a File with Scope Management

This example demonstrates loading configuration from a file within a scope, ensuring the file handle is properly closed when no longer needed.

```scala title="scope-examples/src/main/scala/scope/examples/ConfigReaderExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates the `Unscoped` marker trait behavior.
 *
 * ==Key Concepts==
 *
 *   - '''`Unscoped`''' marks pure data types that can safely escape a scope
 *   - Pure data escapes freely; resources remain scope-bound
 *
 * ==Example Scenario==
 *
 * A configuration reader produces `ConfigData` (pure, escapable data), while a
 * secret store holds resources that must remain scoped to prevent leakage.
 */

// ---------------------------------------------------------------------------
// Domain Types
// ---------------------------------------------------------------------------

/**
 * Pure configuration data that can safely escape any scope.
 *
 * By deriving `Unscoped`, we declare this type contains no resources. When
 * returned from `scoped { ... }`, the raw `ConfigData` is returned.
 */
case class ConfigData(
  appName: String,
  version: String,
  settings: Map[String, String]
) derives Unscoped

/**
 * Reads configuration files from disk.
 *
 * This is a resource (holds file handles, caches) and must be closed.
 */
class ConfigReader extends AutoCloseable {
  private var closed = false

  def readConfig(@annotation.unused path: String): ConfigData = {
    require(!closed, "ConfigReader is closed")
    ConfigData(
      appName = "MyApplication",
      version = "1.0.0",
      settings = Map(
        "database.host" -> "localhost",
        "database.port" -> "5432",
        "log.level"     -> "INFO"
      )
    )
  }

  override def close(): Unit = {
    closed = true
    println("  [ConfigReader] Closed.")
  }
}

/**
 * Manages access to application secrets.
 *
 * This resource maintains connections and caches; it should NOT have an
 * `Unscoped` instance. It cannot escape the scope.
 */
class SecretStore extends AutoCloseable {
  private var closed = false

  def getSecret(key: String): String = {
    require(!closed, "SecretStore is closed")
    s"secret-value-for-$key"
  }

  override def close(): Unit = {
    closed = true
    println("  [SecretStore] Closed.")
  }
}

// ---------------------------------------------------------------------------
// Main Example
// ---------------------------------------------------------------------------

@main def runConfigReaderExample(): Unit = {
  println("=== Unscoped Example ===\n")

  // ConfigData is Unscoped, so it escapes the scope as raw ConfigData
  val escapedConfig: ConfigData = Scope.global.scoped { scope =>

    val reader: $[ConfigReader] = allocate(Resource(new ConfigReader))

    // $(reader)(f) auto-unwraps to ConfigData (Unscoped)
    $(reader)(_.readConfig("/etc/app/config.json"))
  }

  println("Escaped config (used outside scope):")
  println(s"  App: ${escapedConfig.appName} v${escapedConfig.version}")
  escapedConfig.settings.foreach { case (k, v) => println(s"    $k = $v") }
  println()

  println("SecretStore stays scoped:")
  Scope.global.scoped { scope =>

    val secrets: $[SecretStore] = allocate(Resource(new SecretStore))

    $(secrets) { s =>
      val dbPassword = s.getSecret("database.password")
      println(s"  Retrieved secret: $dbPassword")
    }
    ()
  }
  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ConfigReaderExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.runConfigReaderExample"
```

### Implementing a Plugin Architecture with Automatic Resource Discovery

This example shows how to build a plugin system that discovers and loads plugins dynamically, managing their lifecycle with scopes.

```scala title="scope-examples/src/main/scala/scope/examples/PluginArchitectureExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Plugin Architecture Example — Trait Injection via Subtype Wires
 *
 * Demonstrates how abstract trait dependencies are resolved via concrete
 * implementation wires using subtype resolution. This pattern enables:
 *   - Clean interface/implementation separation
 *   - Easy swapping of implementations (e.g., Stripe vs PayPal)
 *   - Compile-time verified dependency graphs
 */

/** Configuration for payment gateway connections. */
final case class GatewayConfig(apiKey: String, merchantId: String)

/** Result of a payment operation. */
final case class PaymentResult(transactionId: String, success: Boolean, message: String)

/**
 * Abstract payment gateway interface.
 *
 * Services depend on this trait, not concrete implementations.
 */
trait PaymentGateway {
  def charge(amount: BigDecimal, currency: String): PaymentResult
  def refund(transactionId: String): PaymentResult
}

/**
 * Stripe implementation of [[PaymentGateway]].
 *
 * When wired via `Wire.shared[StripeGateway]`, this satisfies any
 * `PaymentGateway` dependency through subtype resolution.
 */
final class StripeGateway(config: GatewayConfig) extends PaymentGateway with AutoCloseable {
  println(s"[Stripe] Connected with merchant ${config.merchantId}")

  def charge(amount: BigDecimal, currency: String): PaymentResult = {
    val txId = s"stripe_${System.nanoTime()}"
    PaymentResult(txId, success = true, s"Charged $currency $amount via Stripe")
  }

  def refund(transactionId: String): PaymentResult =
    PaymentResult(transactionId, success = true, s"Refunded $transactionId via Stripe")

  def close(): Unit = println("[Stripe] Connection closed")
}

/** PayPal implementation — demonstrates swappability. */
final class PayPalGateway(config: GatewayConfig) extends PaymentGateway with AutoCloseable {
  println(s"[PayPal] Connected with merchant ${config.merchantId}")

  def charge(amount: BigDecimal, currency: String): PaymentResult = {
    val txId = s"paypal_${System.nanoTime()}"
    PaymentResult(txId, success = true, s"Charged $currency $amount via PayPal")
  }

  def refund(transactionId: String): PaymentResult =
    PaymentResult(transactionId, success = true, s"Refunded $transactionId via PayPal")

  def close(): Unit = println("[PayPal] Connection closed")
}

/**
 * Checkout service that depends on the abstract [[PaymentGateway]] trait.
 *
 * This service is unaware of which gateway implementation is injected.
 */
final class CheckoutService(gateway: PaymentGateway) extends AutoCloseable {
  def processOrder(orderId: String, amount: BigDecimal): PaymentResult = {
    println(s"[Checkout] Processing order $orderId")
    gateway.charge(amount, "USD")
  }

  def close(): Unit = println("[Checkout] Service shutdown")
}

@main def pluginArchitectureExample(): Unit = {
  val config = GatewayConfig(apiKey = "sk_test_xxx", merchantId = "acme_corp")

  println("=== Using Stripe Gateway ===")
  val stripeResource: Resource[CheckoutService] = Resource.from[CheckoutService](
    Wire(config),
    Wire.shared[StripeGateway] // Satisfies PaymentGateway via subtyping
  )

  Scope.global.scoped { scope =>

    val checkout: $[CheckoutService] = allocate(stripeResource)
    $(checkout) { c =>
      val result = c.processOrder("ORD-001", BigDecimal("99.99"))
      println(s"Result: ${result.message}")
    }
    ()
  }

  println("\n=== Using PayPal Gateway ===")
  val paypalResource: Resource[CheckoutService] = Resource.from[CheckoutService](
    Wire(config),
    Wire.shared[PayPalGateway] // Swap to PayPal — no other changes needed
  )

  Scope.global.scoped { scope =>

    val checkout: $[CheckoutService] = allocate(paypalResource)
    $(checkout) { c =>
      val result = c.processOrder("ORD-002", BigDecimal("149.99"))
      println(s"Result: ${result.message}")
    }
    ()
  }
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/PluginArchitectureExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.pluginArchitectureExample"
```

### Demonstrating Thread Ownership Enforcement in Scope Hierarchies

This example demonstrates how Scope enforces thread ownership, preventing cross-thread scope misuse and illustrating the difference between owned and unowned scopes.

```scala title="scope-examples/src/main/scala/scope/examples/ThreadOwnershipExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Simulates a stateful resource that tracks which thread owns it.
 *
 * @param name
 *   the resource name
 */
final class ThreadAwareResource(val name: String) extends AutoCloseable {
  private val createdThread = Thread.currentThread()

  def getInfo: String = {
    val currentThread = Thread.currentThread()
    val owner         = createdThread.getName
    val current       = currentThread.getName
    if (createdThread eq currentThread) {
      s"[$name] Safe: owned by '$owner', accessed by '$current' (same thread)"
    } else {
      s"[$name] WARNING: owned by '$owner', accessed by '$current' (different thread!)"
    }
  }

  override def close(): Unit =
    println(s"[$name] Closing resource (was created by ${createdThread.getName})")
}

/**
 * Demonstrates thread ownership enforcement in ZIO Blocks Scope.
 *
 * This example shows:
 *   - Scope.global: `isOwner` always true; any thread can create children from
 *     it
 *   - Scope.Child: captures the creating thread; `isOwner` checks
 *     `Thread.currentThread() eq owner`
 *   - Scope.open(): creates an unowned child scope; `isOwner` always true from
 *     any thread
 *   - Calling `scoped` on a Scope.Child from a different thread throws
 *     IllegalStateException
 *
 * Thread ownership prevents accidentally passing a scope to another thread and
 * using it there, which would violate structured concurrency guarantees.
 */
@main def runThreadOwnershipExample(): Unit = {
  println("=== Thread Ownership Example ===\n")

  // === Part 1: Single-thread usage (CORRECT) ===
  println("--- Part 1: Single-thread usage (correct) ---\n")

  Scope.global.scoped { scope =>
    val currentThread = Thread.currentThread().getName
    println(s"[Main] Entered scope on thread: $currentThread\n")

    // Scope.Child is owned by the current thread (main)
    scope.scoped { child =>

      println(s"[Main] Created child scope on thread: $currentThread")
      println(s"[Main] Child scope isOwner: ${child.isOwner} (true only for the creating thread)")

      val res: $[ThreadAwareResource] =
        allocate(Resource(new ThreadAwareResource("SingleThreadResource")))

      $(res) { r =>
        println(s"[Main] ${r.getInfo}\n")
      }
    }

    println(s"[Main] Child scope closed, finalizers ran\n")
  }

  // === Part 2: Demonstrating Scope.open() for cross-thread usage ===
  println("--- Part 2: Unowned scope via open() (for cross-thread) ---\n")

  Scope.global.scoped { scope =>

    val mainThread = Thread.currentThread().getName
    println(s"[Main] On thread: $mainThread\n")

    // open() creates an unowned scope that any thread can use
    $(open()) { handle =>
      val childScope = handle.scope
      println(s"[Main] Created unowned scope via open()")
      println(s"[Main] Unowned scope isOwner: ${childScope.isOwner} (true from any thread)\n")

      // Now we can use this scope from a different thread
      val executor = Executors.newSingleThreadExecutor { r =>
        val t = new Thread(r)
        t.setName("worker-thread")
        t
      }

      try {
        val latch = new CountDownLatch(1)

        executor.execute { () =>
          try {
            val workerThread = Thread.currentThread().getName
            println(s"[Worker] On thread: $workerThread\n")

            // Using the unowned scope from a different thread - this works!
            childScope.scoped { workerChild =>

              println(s"[Worker] Created child of unowned scope")

              val res: $[ThreadAwareResource] =
                allocate(Resource(new ThreadAwareResource("CrossThreadResource")))

              $(res) { r =>
                println(s"[Worker] ${r.getInfo}\n")
              }

              println("[Worker] Worker scope closed")
            }
          } finally {
            latch.countDown()
          }
        }

        // Wait for worker thread to finish
        latch.await()
        println()
      } finally {
        executor.shutdown()
        // Clean up the open scope and propagate any finalizer failures
        handle.close().orThrow()
        println("[Main] Unowned scope closed\n")
      }
    }
  }

  // === Part 3: Explanation of ownership violation (what would fail) ===
  println("--- Part 3: Thread ownership violation (explanation) ---\n")
  println("""
If you tried to pass a Scope.Child to another thread and call scoped on it,
you would get an IllegalStateException:

  Scope.global.scoped { scope =>

    // This scope is owned by the main thread
    val executor = Executors.newSingleThreadExecutor()

    executor.execute { () =>
      // This would throw: Cannot create child scope: current thread does not own this scope.
      scope.scoped { child => ... }  // WRONG: scope is owned by main thread
    }
  }

Solution: Use scope.open() instead, which creates an unowned scope that
any thread can use. See Part 2 above for the correct pattern.
""")

  println("=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/ThreadOwnershipExample.scala))

```bash
sbt "scope-examples/runMain runThreadOwnershipExample"
```

### Detecting and Demonstrating Circular Dependency Scenarios

This example shows how to detect and handle circular dependencies in resource management, illustrating how scopes help prevent subtle bugs in complex dependency graphs.

```scala title="scope-examples/src/main/scala/scope/examples/CircularDependencyDemoExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates compile-time cycle detection in ZIO Blocks Scope.
 *
 * The `Resource.from[T]` macro analyzes the dependency graph at compile time
 * and rejects circular dependencies with a descriptive error message showing
 * the exact cycle path.
 *
 * ==The Problem==
 * Circular dependencies (A → B → A) cannot be resolved by constructor injection
 * because neither service can be instantiated without the other already
 * existing.
 *
 * ==The Solution==
 * Break the cycle by introducing an interface (trait) that one service depends
 * on, allowing the implementation to be provided separately. This is a standard
 * Dependency Inversion Principle pattern.
 */

// ─────────────────────────────────────────────────────────────────────────────
// PROBLEMATIC: Circular Dependency (would not compile)
// ─────────────────────────────────────────────────────────────────────────────

// These classes form a cycle: ServiceA → ServiceB → ServiceA
// Uncommenting the Resource.from call below would produce a compile error.

// class ServiceA(b: ServiceB) {
//   def greet(): String = s"A says hello, B says: ${b.respond()}"
// }
//
// class ServiceB(a: ServiceA) {
//   def respond(): String = s"B responds, A type: ${a.getClass.getSimpleName}"
// }
//
// Attempting to wire this would fail at compile time:
// val circularResource = Resource.from[ServiceA]()
//
// Expected compile error:
// ┌────────────────────────────┐
// │                            ▼
//     ServiceA ──► ServiceB
// ▲                            │
// └────────────────────────────┘
//
// Break the cycle by:
//   • Introducing an interface/trait
//   • Using lazy initialization
//   • Restructuring dependencies

// ─────────────────────────────────────────────────────────────────────────────
// SOLUTION: Break the cycle with an interface
// ─────────────────────────────────────────────────────────────────────────────

/** Interface that ServiceA depends on, breaking the compile-time cycle. */
trait ServiceBApi {
  def respond(): String
}

/** Concrete implementation of ServiceA that depends only on the interface. */
class ServiceAImpl(b: ServiceBApi) {
  def greet(): String = s"A says hello, B says: ${b.respond()}"
}

/** Concrete implementation of ServiceB without any dependency on A. */
class ServiceBImpl extends ServiceBApi {
  override def respond(): String = "B responds successfully"
}

/** Application that composes both services. */
class Application(a: ServiceAImpl, @annotation.unused b: ServiceBApi) {
  def run(): String = a.greet()
}

/**
 * Demonstrates the working (non-circular) pattern.
 *
 * The dependency graph is now: Application → ServiceAImpl → ServiceBApi ↘
 * ServiceBApi
 *
 * ServiceBImpl provides ServiceBApi, and there is no cycle.
 */
@main def circularDependencyDemoExample(): Unit = {
  println("=== Circular Dependency Demo ===\n")
  println("Demonstrating compile-time cycle detection and how to break cycles.\n")

  Scope.global.scoped { scope =>

    println("Creating application with proper dependency structure...")

    // Wire.shared[ServiceBImpl] provides both ServiceBImpl and ServiceBApi (via subtyping)
    val app: $[Application] = allocate(
      Resource.from[Application](
        Wire.shared[ServiceBImpl]
      )
    )

    println(s"Result: ${$(app)(_.run())}")
    println("\nThe dependency graph was validated at compile time.")
    println("No cycles detected - application wired successfully.")
  }

  println("\n─── Key Takeaways ───")
  println("• Resource.from[T] detects cycles at compile time")
  println("• Cycles produce clear ASCII diagrams showing the path")
  println("• Break cycles by introducing interfaces/traits")
  println("• The Dependency Inversion Principle resolves most cycles")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/CircularDependencyDemoExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.circularDependencyDemoExample"
```

### Using Scope with Legacy Libraries that Don't Support Managed Resources

This example demonstrates how to integrate Scope with legacy libraries that don't natively support resource management, using wrapper resources and the `leak` escape hatch when necessary.

```scala title="scope-examples/src/main/scala/scope/examples/LegacyLibraryInteropExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates `leak(scopedValue)` for third-party library interop.
 *
 * Sometimes you must pass scoped resources to legacy or third-party libraries
 * that require raw types. The `leak` escape hatch extracts the underlying
 * value, bypassing compile-time safety. Use sparingly—you assume responsibility
 * for ensuring the resource outlives its usage.
 */

// ---------------------------------------------------------------------------
// Fake classes simulating a legacy networking library
// ---------------------------------------------------------------------------

/** Configuration for establishing a socket connection. */
case class SocketConfig(host: String, port: Int)

/**
 * A managed socket that implements AutoCloseable.
 *
 * In a real scenario, this would wrap an actual network socket.
 */
class ManagedSocket(val config: SocketConfig) extends AutoCloseable {
  private var closed = false

  def send(data: Array[Byte]): Unit =
    if (closed) throw new IllegalStateException("Socket closed")
    else println(s"  [Socket] Sent ${data.length} bytes to ${config.host}:${config.port}")

  def receive(): Array[Byte] =
    if (closed) throw new IllegalStateException("Socket closed")
    else {
      println(s"  [Socket] Received response from ${config.host}:${config.port}")
      "ACK".getBytes
    }

  override def close(): Unit = {
    closed = true
    println(s"  [Socket] Connection to ${config.host}:${config.port} closed")
  }
}

/**
 * Simulates a third-party protocol handler that cannot be modified.
 *
 * This legacy library requires a raw `ManagedSocket` and does not understand
 * scoped types. This is the typical scenario where `leak` becomes necessary.
 */
object LegacyProtocolHandler {

  /**
   * Handles a connection using a proprietary protocol.
   *
   * @param socket
   *   the raw socket—must remain open for the duration of this call
   */
  def handleConnection(socket: ManagedSocket): Unit = {
    println("  [Legacy] Starting proprietary protocol handshake...")
    socket.send("HELLO".getBytes)
    val response = socket.receive()
    println(s"  [Legacy] Handshake complete: ${new String(response)}")
  }
}

// ---------------------------------------------------------------------------
// Example entry point
// ---------------------------------------------------------------------------

@main def legacyLibraryInteropExample(): Unit = {
  println("=== Legacy Library Interop Example ===\n")
  println("Demonstrating leak() for passing scoped resources to third-party code.\n")

  Scope.global.scoped { scope =>

    // Allocate the socket as a scoped resource.
    // The socket is tagged with the scope's type, preventing accidental escape.
    val scopedSocket: $[ManagedSocket] = allocate(
      Resource.fromAutoCloseable(new ManagedSocket(SocketConfig("api.example.com", 443)))
    )
    println("Allocated scoped socket.\n")

    // -------------------------------------------------------------------------
    // WARNING: leak() bypasses compile-time safety guarantees!
    //
    // After calling leak(), the compiler cannot prevent you from:
    //   - Storing the socket in a field that outlives the scope
    //   - Passing it to code that might cache or close it unexpectedly
    //   - Using it after the scope has closed
    //
    // Only use leak() when:
    //   1. The third-party API genuinely cannot accept scoped types
    //   2. You can guarantee the scope outlives all usage of the leaked value
    //   3. The third-party code won't cache or transfer ownership
    // -------------------------------------------------------------------------

    // WARNING: leak() bypasses compile-time safety — use only for third-party interop.
    // This intentionally escapes the scoped type and will emit a compiler warning.
    @nowarn("msg=.*leaked.*|.*leak.*")
    val rawSocket: ManagedSocket = leak(scopedSocket)

    println("Passing raw socket to legacy protocol handler:")
    LegacyProtocolHandler.handleConnection(rawSocket)

    println("\nScope exiting - socket will be closed automatically:")
  }

  println("\nExample complete. The socket was safely closed when the scope exited.")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/LegacyLibraryInteropExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.legacyLibraryInteropExample"
```

### Integration Testing with Automatic Setup and Teardown

This example shows how to use Scope to manage test fixtures and resources in integration tests, ensuring automatic cleanup between test runs and proper resource finalization.

```scala title="scope-examples/src/main/scala/scope/examples/IntegrationTestHarnessExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package scope.examples

/**
 * Demonstrates combining DI-wired services with manually-managed test fixtures.
 *
 * This example shows a realistic integration test setup where:
 *   - Application services are wired via [[Resource.from]] (DI approach)
 *   - Test fixtures are managed with [[Resource.acquireRelease]] (manual
 *     approach)
 *   - Both resource types are properly cleaned up in LIFO order
 */
object IntegrationTestHarnessExample {

  // --- Test Infrastructure ---

  /** Configuration for the test environment. */
  case class TestConfig(dbUrl: String, serverPort: Int)

  /** Test database with lifecycle hooks for setup/teardown and data seeding. */
  class TestDatabase(val config: TestConfig) extends AutoCloseable {
    private var data = Map.empty[String, Any]

    def setup(): Unit                         = println(s"  [DB] Initialized at ${config.dbUrl}")
    def teardown(): Unit                      = { data = Map.empty; println("  [DB] Data cleared") }
    def seed(newData: Map[String, Any]): Unit = {
      data = newData; println(s"  [DB] Seeded with ${newData.size} entries")
    }
    def query(key: String): Option[Any] = data.get(key)
    def close(): Unit                   = println("  [DB] Connection closed")
  }

  /** Test HTTP server that can be started and stopped. */
  class TestServer(val config: TestConfig) extends AutoCloseable {
    val baseUrl: String = s"http://localhost:${config.serverPort}"

    def start(): Unit = println(s"  [Server] Started at $baseUrl")
    def stop(): Unit  = println("  [Server] Stopped")
    def close(): Unit = println("  [Server] Resources released")
  }

  /** Aggregates test fixtures for convenient access during tests. */
  case class TestFixture(db: TestDatabase, server: TestServer)

  // --- Application Under Test ---

  /** The application being tested; requires a database connection. */
  class AppUnderTest(val db: TestDatabase) extends AutoCloseable {
    def handleRequest(req: String): String = db.query(req).map(_.toString).getOrElse("Not found")
    def close(): Unit                      = println("  [App] Shutdown complete")
  }

  // --- Resource Definitions ---

  /**
   * Creates a manually-managed test fixture using [[Resource.acquireRelease]].
   *
   * This approach gives explicit control over setup and teardown phases, which
   * is typical for test fixtures that need initialization beyond construction.
   */
  def testFixtureResource(config: TestConfig): Resource[TestFixture] =
    Resource.acquireRelease {
      println("  [Fixture] Acquiring test fixture...")
      val db     = new TestDatabase(config)
      val server = new TestServer(config)
      db.setup()
      server.start()
      TestFixture(db, server)
    } { fixture =>
      println("  [Fixture] Releasing test fixture...")
      fixture.server.stop()
      fixture.db.teardown()
      fixture.db.close()
      fixture.server.close()
    }

  /**
   * Runs the integration test harness example.
   *
   * Demonstrates:
   *   1. Manual fixture via [[Resource.acquireRelease]] for test infrastructure
   *   2. DI-wired application via [[Resource.from]] consuming the fixture
   *   3. Proper cleanup ordering: app closes before fixtures
   */
  def run(): Unit = {
    println("=== Integration Test Harness Example ===\n")

    val config = TestConfig("jdbc:h2:mem:test", 8080)

    // Combine manual fixtures with DI-wired application
    val testHarnessResource: Resource[(TestFixture, AppUnderTest)] =
      testFixtureResource(config).flatMap { fixture =>
        // Seed test data
        fixture.db.seed(Map("user:1" -> "Alice", "user:2" -> "Bob"))

        // Wire the app using DI, injecting the fixture's database
        val appWire     = Wire.shared[AppUnderTest]
        val dbWire      = Wire(fixture.db)
        val appResource = Resource.from[AppUnderTest](appWire, dbWire)

        appResource.map(app => (fixture, app))
      }

    // Run in a scoped block - all resources cleaned up on exit
    Scope.global.scoped { scope =>

      println("Allocating resources...")
      val harness: $[(TestFixture, AppUnderTest)] = allocate(testHarnessResource)
      println()

      // Run test scenarios - access the tuple via $
      println("Running test scenarios:")
      $(harness) { h =>
        println(s"  GET user:1 -> ${h._2.handleRequest("user:1")}")
        println(s"  GET user:2 -> ${h._2.handleRequest("user:2")}")
        println(s"  GET user:3 -> ${h._2.handleRequest("user:3")}")
        println(s"  Server URL: ${h._1.server.baseUrl}")
      }
      println()

      println("Scope closing, releasing resources in LIFO order...")
    }

    println("\n=== Example Complete ===")
  }
}

@main def runIntegrationTestHarness(): Unit = IntegrationTestHarnessExample.run()
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/scope/examples/IntegrationTestHarnessExample.scala))

```bash
sbt "scope-examples/runMain scope.examples.IntegrationTestHarnessExample"
```

---

## Unscoped

`Unscoped[A]` is a marker typeclass for types that can safely escape a scope without tracking. Types with an `Unscoped` instance are considered "safe data"—they don't hold resources and can be freely extracted from a scope. Here's the definition:

```scala
trait Unscoped[A]
```

The `Unscoped` typeclass distinguishes between two categories of types:

1. **Unscoped types** (have an instance): Primitives, strings, collections, value types, and pure data. These can leave a scope without risk.
2. **Scoped types** (no instance): Resources like streams, connections, handles. These must remain tracked within a scope.

When the `$` operator is used to access a scoped value, if the result type has an `Unscoped` instance, it returns the value directly (unwrapped). Otherwise, it returns the value still wrapped in `$`.

## Motivation / Use Case

The exact problem: A `scoped` block automatically closes all resources when it exits. If you accidentally returned a resource (like a database connection or file handle) from the block, it would be closed—but you might try to use it later, causing a **use-after-close** crash. Here's an example (this would fail without `Unscoped`):

```scala

final class Database {
  def query(sql: String) = s"result: $sql"
}

// Without Unscoped constraint, this compiles (BAD):
// val db: Database = Scope.global.scoped { scope =>
//   val db = allocate(Resource(new Database()))
//   db  // BUG: returns the resource itself, not data extracted from it
// }
// db.query("SELECT 1")  // CRASH: use-after-close (scope already closed it)
```

The solution: `Unscoped` makes this a **compile error** instead of a runtime bug. When a `scoped` block returns a value, that value's type must have an `Unscoped` instance—meaning the type checker verifies you're only extracting *computed results* (like `Int`, `String`, or aggregate data), not resources themselves.

You can still extract computed results by using the `$` operator to unwrap scoped values *within* the scope:

```scala

Scope.global.scoped { scope =>

  val intValue = allocate(Resource(42))
  // Extract the Int value (not the Resource), computed inside the scope
  val n: Int = $(intValue)(x => x + 1)

  val text = allocate(Resource("hello"))
  // Extract the String value (not the Resource), computed inside the scope
  val s: String = $(text)(x => x.toUpperCase)

  (n, s)  // Tuple of pure data: safe to return
}
```

## Returning Unscoped Data from Scopes

Extract computed results that don't hold resources:

```scala

case class ProcessingResult(count: Int, elapsed: FiniteDuration)

object ProcessingResult {
  implicit val unscoped: Unscoped[ProcessingResult] = new Unscoped[ProcessingResult] {}
}

def processData(): ProcessingResult = Scope.global.scoped { scope =>

  val startTime = java.time.Instant.now()
  val input = allocate(Resource(Seq(1, 2, 3, 4, 5)))
  val count = $(input)(_.length)

  val endTime = java.time.Instant.now()
  val elapsed = java.time.Duration.between(startTime, endTime).toNanos

  ProcessingResult(count, FiniteDuration(elapsed, java.util.concurrent.TimeUnit.NANOSECONDS))
}

val result = processData()
println(result)
```

Only create instances for **pure data types** that don't hold resources. Never create instances for types that contain connections, streams, handles, or any resource-like fields.

## Predefined Instances

All built-in instances follow a simple principle: **if a type cannot hold resources, it gets an `Unscoped` instance**. Collections inherit this property from their elements — `List[Int]` is unscoped because `Int` is unscoped.

**Primitive and atomic values** (cannot hold resources by nature):
- `Int`, `Long`, `Short`, `Byte`, `Char`, `Boolean`, `Float`, `Double`, `Unit`
- `String`, `BigInt`, `BigDecimal`
- `java.util.UUID`

**Collections with conditional instances** (safe when elements/entries are unscoped):
- Sequences: `Array[A]`, `List[A]`, `Vector[A]`, `Seq[A]`, `IndexedSeq[A]`, `Iterable[A]`
- Sets: `Set[A]`
- Maps: `Map[K, V]` (when both `K` and `V` are unscoped)
- Wrappers: `Option[A]`, `Either[A, B]`, `Tuple2[A, B]` through `Tuple4[A, B, C, D]`
- ZIO types: `zio.blocks.chunk.Chunk[A]`

**Standard library time types** (immutable, cannot hold resources):
- `java.time.Instant`, `LocalDate`, `LocalTime`, `LocalDateTime`, `ZonedDateTime`, `OffsetDateTime`
- `java.time.Duration`, `Period`, `ZoneId`, `ZoneOffset`
- `scala.concurrent.duration.Duration`, `FiniteDuration`

All other types (resources, handles, connections) must be manually defined if needed.

## Thread Safety

`Unscoped` instances themselves are immutable and thread-safe. However, the types they mark must be truly immutable for safe concurrent use. For example, `Array[Int]` is mutable—if shared across threads without synchronization, it could cause data races.

## Integration

- [`Scope.$`](./scope.md) — the operator that uses `Unscoped`
- [`Resource`](./resource.md) — types that provide `Unscoped` may be wrapped in resources
- [`Scope`](./scope.md) — manages the lifecycle of resources

---

## Wire

`Wire[-In, +Out]` is a **compile-time safe recipe for constructing a service and its dependencies**. Wires describe how to construct an `Out` value given access to its dependencies via a `Context[In]` and a `Scope` for finalization:

```scala
sealed trait Wire[-In, +Out] {
  def isShared: Boolean
  def isUnique: Boolean = !isShared

  def shared: Wire.Shared[In, Out]
  def unique: Wire.Unique[In, Out]

  def toResource(deps: Context[In]): Resource[Out]
}
```

The type parameters are:
- **`In` (contravariant)**: the dependencies required to construct the service
- **`Out` (covariant)**: the service produced

Wires are the building blocks of dependency injection in Scope. They form the foundation for constructor-based dependency injection via `Resource.from[T](wires*)`.

## Overview

A `Wire` is a **lazy recipe**, not an execution. It holds a construction function `(Scope, Context[In]) => Out` and a **sharing strategy** — either `Shared` (reference-counted via `Resource.shared`) or `Unique` (fresh instance per allocation). The Wire itself does nothing until you convert it to a `Resource` and allocate it within a scope.

Here's the typical flow:

```
Wire (recipe)
  ↓
Resource (lazy, composable)
  ↓
scope.allocate(...)
  ↓
$[T] (scoped value in scope)
```

## Motivation

Without Wires, building a multi-layer application requires manual dependency passing:

```scala
final case class Config(dbUrl: String)

final class Database(config: Config) extends AutoCloseable {
  def close(): Unit = println(s"closing connection to ${config.dbUrl}")
}

final class UserService(db: Database) {
  def getUser(id: Int): String = s"user $id"
}

final class App(service: UserService) {
  def run(): Unit = println(service.getUser(1))
}

// Manual wiring:
Scope.global.scoped { scope =>

  val config = Config("jdbc:postgres://localhost/db")
  val db = Resource.fromAutoCloseable(new Database(config)).allocate
  val service = new UserService($(db)(identity))
  val app = new App(service)
  app.run()
}
```

With `Wire` + `Resource.from`, the macro handles the dependency graph:

```scala
Scope.global.scoped { scope =>

  val app = Resource.from[App](
    Wire(Config("jdbc:postgres://localhost/db"))
  ).allocate
  $(app)(_.run())
}
```

**Benefits:**
- **Compile-time graph validation** — cycle detection, duplicate providers, missing dependencies
- **Automatic finalization** — `AutoCloseable` resources are finalized in LIFO order
- **Sharing control** — choose which services are singletons (shared) vs fresh per allocation (unique)
- **Type-safe construction** — no stringly-typed dependency resolution

## Installation

Add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-scope" % "0.0.33"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-scope" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## Construction

`Wire` provides multiple ways to create instances, ranging from automatic macro-based derivation to manual construction for custom logic. Here are the main construction methods:

### Wire.shared[T] — derive a shared wire

The `Wire.shared[T]` macro inspects `T`'s primary constructor and generates a shared wire that reuses the same instance across dependents:

```scala

final case class Config(debug: Boolean)

final class Database(config: Config) extends AutoCloseable {
  def query(sql: String): String = s"[db] $sql"
  def close(): Unit = println("database closed")
}

val wire: Wire.Shared[Config, Database] = Wire.shared[Database]

Scope.global.scoped { scope =>

  val config = Config(debug = true)
  val deps = Context[Config](config)
  val db = allocate(wire.toResource(deps))
  $(db)(_.query("SELECT 1"))
}
```

### Wire.unique[T] — derive a unique wire

Like `Wire.shared[T]`, but creates a fresh instance each time the wire is used. Use for request-scoped or per-call services:

```scala

final class RequestHandler {
  val id = scala.util.Random.nextInt()
}

val wire: Wire.Unique[Any, RequestHandler] = Wire.unique[RequestHandler]

Scope.global.scoped { scope =>

  val deps = Context.empty[Any]
  val resource = wire.toResource(deps)

  val h1 = allocate(resource)
  val h2 = allocate(resource)

  val ids: (Int, Int) = (
    $(h1)(_.id),
    $(h2)(_.id)
  )
  // ids._1 != ids._2 (different instances)
}
```

### Wire.apply[T] — lift a pre-existing value

Creates a shared wire that injects a value you already have. If the value is `AutoCloseable`, its `close()` method is automatically registered as a finalizer:

```scala

final case class Config(dbUrl: String)

val config = Config("jdbc:postgres://localhost/db")
val wire: Wire.Shared[Any, Config] = Wire(config)

Scope.global.scoped { scope =>

  val cfg = allocate(wire.toResource(Context.empty[Any]))
  $(cfg)(_.dbUrl)
}
```

### Wire.Shared.fromFunction — manual shared wire construction

Use this for custom construction logic when macro derivation doesn't fit:

```scala

final case class Config(timeout: Int)

final class Client(config: Config) {
  def call(): String = s"calling with timeout=${config.timeout}"
}

val wire: Wire.Shared[Config, Client] =
  Wire.Shared.fromFunction { (scope, ctx) =>
    val config = ctx.get[Config]
    new Client(config)
  }

Scope.global.scoped { scope =>

  val config = Config(30)
  val deps = Context[Config](config)
  val client = allocate(wire.toResource(deps))
  $(client)(_.call())
}
```

### Wire.Unique.fromFunction — manual unique wire construction

Like `Wire.Shared.fromFunction`, but for unique wires:

```scala

final class RequestContext {
  val id = scala.util.Random.nextInt()
}

val wire: Wire.Unique[Any, RequestContext] =
  Wire.Unique.fromFunction { (_, _) =>
    new RequestContext
  }

Scope.global.scoped { scope =>

  val deps = Context.empty[Any]
  val resource = wire.toResource(deps)

  val r1 = allocate(resource)
  val r2 = allocate(resource)

  val different: Boolean = $(r1)(_.id) != $(r2)(_.id)
  // different == true
}
```

## Shared vs Unique

The fundamental difference is **reuse semantics**:

| Aspect | Shared | Unique |
|--------|--------|--------|
| **Construction** | `Wire.shared[T]` macro or `Wire.Shared.fromFunction` | `Wire.unique[T]` macro or `Wire.Unique.fromFunction` |
| **Resource type** | `Resource.shared[T]` (reference-counted) | `Resource.unique[T]` (fresh per call) |
| **When to use** | Singletons, expensive resources (connections, thread pools) | Request scoped, stateful per-call (request handlers) |
| **Instance reuse** | Same instance across entire dependency graph | New instance per allocation |
| **Finalization** | Runs when last referencing scope closes | Runs when each scope closes |

In the diamond pattern (where `App` depends on both `UserService` and `OrderService`, both of which depend on `Database`), a shared wire ensures `Database` is constructed once and both services receive the same instance:

```scala

final class Database {
  val id = scala.util.Random.nextInt()
}

final class UserService(db: Database) {
  def getDbId(): Int = db.id
}

final class OrderService(db: Database) {
  def getDbId(): Int = db.id
}

final class App(userService: UserService, orderService: OrderService) {
  def check(): Boolean = {
    // With shared Database, these should be equal
    userService.getDbId() == orderService.getDbId()
  }
}

// Using shared wires for all dependencies
val resource = Resource.from[App](
  Wire.shared[Database],
  Wire.shared[UserService],
  Wire.shared[OrderService]
)

Scope.global.scoped { scope =>

  val app = resource.allocate
  $(app)(_.check())  // true: Database is shared
}
```

## Core Operations

The `Wire` interface provides a set of operations for inspecting and transforming wires:

### `Wire#isShared` and `Wire#isUnique`

Check the sharing strategy of a wire:

```scala
trait Wire[-In, +Out] {
  def isShared: Boolean
  def isUnique: Boolean = !isShared
}
```

Here's how to use these methods:

```scala

val sharedWire = Wire.shared[String]
val uniqueWire = Wire.unique[String]

println(s"sharedWire.isShared: ${sharedWire.isShared}")      // true
println(s"sharedWire.isUnique: ${sharedWire.isUnique}")      // false
println(s"uniqueWire.isShared: ${uniqueWire.isShared}")      // false
println(s"uniqueWire.isUnique: ${uniqueWire.isUnique}")      // true
```

### `Wire#shared` and `Wire#unique` — convert between strategies

Convert a wire to the opposite strategy:

```scala
trait Wire[-In, +Out] {
  def shared: Wire.Shared[In, Out]
  def unique: Wire.Unique[In, Out]
}
```

Here's how to convert between sharing strategies:

```scala

val original = Wire.shared[String]

val converted: Wire.Unique[Any, String] = original.unique
println(s"original.isShared: ${original.isShared}")          // true
println(s"converted.isUnique: ${converted.isUnique}")        // true

// Converting back returns a new shared wire
val backToShared: Wire.Shared[Any, String] = converted.shared
println(s"backToShared.isShared: ${backToShared.isShared}")  // true
```

Calling `shared` on an already-shared wire returns `this` (identity); likewise `unique` on a unique wire returns `this`.

### `Wire#toResource` — convert a wire to a Resource

Converts the wire to a lazy `Resource` by providing the dependency context:

```scala
trait Wire[-In, +Out] {
  def toResource(deps: Context[In]): Resource[Out]
}
```

Here's how to use this method:

```scala

final case class Config(value: String)

val wire = Wire.shared[Config]
val deps = Context[Config](Config("hello"))

val resource: Resource[Config] = wire.toResource(deps)

Scope.global.scoped { scope =>

  val cfg = allocate(resource)
  $(cfg)(_.value)
}
```

### `Wire#make` — construct directly from a wire

Directly construct a value without going through `Resource.toResource`. This is a low-level operation; prefer `allocate(wire.toResource(...))` for safety:

```scala
trait Wire.Shared[-In, +Out] {
  def make(scope: Scope, context: Context[In]): Out
}
```

Here's how to use this method:

```scala

final class Service {
  def getName: String = "service"
}

val wire = Wire.shared[Service]

Scope.global.scoped { scope =>

  val service = wire.asInstanceOf[Wire.Shared[Any, Service]].make(scope, Context.empty[Any])
  println(service.getName)
}
```

## Macro Derivation

When you call `Wire.shared[T]` or `Wire.unique[T]`, the macro performs these checks:

1. **Is `T` a class?** — traits and abstract classes are rejected; only concrete classes can be auto-wired.
2. **Does `T` have a primary constructor?** — the macro inspects constructor parameters to determine dependencies.
3. **Is each parameter either a dependency type or a special injected type?** — parameters of type `Scope` or `Finalizer` are recognized and injected automatically; others are looked up in the context.
4. **Does `T` extend `AutoCloseable`?** — if yes, `close()` is automatically registered as a finalizer in the scope.

Example with all three features:

```scala

final case class Config(dbUrl: String)

final class Logger(implicit finalizer: Finalizer) {
  def log(msg: String): Unit = println(msg)
}

final class Database(config: Config)(implicit scope: Scope) extends AutoCloseable {
  def connect(): Unit = {
    scope.defer(println("database connection closed"))
    println(s"connecting to ${config.dbUrl}")
  }

  def query(sql: String): String = s"result: $sql"

  def close(): Unit = println("database closed")
}

final class Service(db: Database, logger: Logger) {
  def run(): Unit = {
    logger.log(db.query("SELECT 1"))
  }
}

// Macro handles Finalizer injection, Scope injection, and AutoCloseable registration
val wire = Wire.shared[Service]
```

### What happens with subtype conflicts

If a constructor has dependencies of related types (e.g., both `FileInputStream` and `InputStream`), the macro rejects the wire because `Context` is type-indexed and cannot reliably disambiguate.

```scala
// This will NOT compile
final class App(input: InputStream, fileInput: FileInputStream)
val wire = Wire.shared[App]  // error: subtype conflict

// Fix: wrap one type to make it distinct
final case class FileInputWrapper(value: FileInputStream)
final class App(input: InputStream, fileInput: FileInputWrapper)
val wire = Wire.shared[App]  // ok
```

## Integration with Resource.from

`Wire` is designed for use with `Resource.from[T](wires*)`, which performs whole-graph dependency injection:

```scala

final case class AppConfig(dbUrl: String)

final class Database(config: AppConfig) extends AutoCloseable {
  def query(sql: String): String = s"result: $sql"
  def close(): Unit = ()
}

final class Repository(db: Database) {
  def query(): String = db.query("SELECT *")
}

final class Service(repo: Repository) extends AutoCloseable {
  def run(): String = repo.query()
  def close(): Unit = ()
}

final class App(service: Service) {
  def run(): String = service.run()
}

// Provide only the leaf dependency; Resource.from derives the rest
val appResource: Resource[App] = Resource.from[App](
  Wire(AppConfig("jdbc:postgres://localhost/db"))
)

Scope.global.scoped { scope =>

  val app = allocate(appResource)
  $(app)(_.run())
}
```

When `Resource.from` composes wires, it respects the sharing strategy:
- **Shared wires** → reference-counted (single instance in the graph)
- **Unique wires** → fresh per allocation

The macro detects cycles, duplicate providers, and missing dependencies at compile time.

## Comparison with Alternatives

| Feature | Wire | Manual Passing | Service Locator |
|---------|------|----------------|-----------------|
| **Type safety** | ✓ (compile-time validation) | ✓ (implicit) | ✗ (string keys) |
| **Cycle detection** | ✓ (compile time) | ✗ | ✗ (runtime) |
| **Sharing semantics** | ✓ (configurable) | Manual | ✓ (singleton pattern) |
| **Finalization** | ✓ (LIFO, automatic) | Manual | Manual |
| **Performance** | ~0 overhead (macro-generated) | ~0 overhead | ~1 allocation overhead |

## Running the Examples

All code from this guide is available as runnable examples in the `scope-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

Basic wire construction demonstrates how to create and use `Wire` for dependency injection. View the example source code:

```scala title="scope-examples/src/main/scala/wire/WireBasicExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package wire

/**
 * Demonstrates basic Wire construction patterns:
 *   - Wire.shared[T] macro derivation
 *   - Wire.apply(value) to lift a value
 *   - Converting wires to Resources
 *   - Using wires in a dependency graph
 */

final case class DbConfig(host: String, port: Int) {
  def url: String = s"jdbc:postgresql://$host:$port/db"
}

final class Database(config: DbConfig) extends AutoCloseable {
  println(s"[Database] Connecting to ${config.url}")

  def query(sql: String): String = {
    println(s"[Database] Executing: $sql")
    "result"
  }

  def close(): Unit =
    println("[Database] Connection closed")
}

final class UserService(db: Database) {
  println("[UserService] Initialized")

  def getUser(id: Int): String = {
    db.query(s"SELECT * FROM users WHERE id = $id")
    s"User(id=$id, name=Alice)"
  }
}

final class BasicApp(service: UserService) {
  def run(): Unit = {
    val user = service.getUser(1)
    println(s"[App] Got: $user")
  }
}

@main def wireBasicExample(): Unit = {
  println("=== Wire Basic Construction Example ===\n")

  // Create the dependency leaf (config) using Wire.apply
  val configWire: Wire.Shared[Any, DbConfig] =
    Wire(DbConfig("localhost", 5432))

  // Derive wires for Database and UserService using the macro
  val dbWire: Wire.Shared[DbConfig, Database] =
    Wire.shared[Database]

  val serviceWire: Wire.Shared[Database, UserService] =
    Wire.shared[UserService]

  val appWire: Wire.Shared[UserService, BasicApp] =
    Wire.shared[BasicApp]

  println("[Setup] Created all wires\n")

  // Use Resource.from to automatically compose the dependency graph
  val appResource: Resource[BasicApp] = Resource.from[BasicApp](
    configWire,
    dbWire,
    serviceWire,
    appWire
  )

  println("[Setup] Composed resource graph\n")

  // Allocate within a scope
  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app: $[BasicApp] = allocate(appResource)

    println("\n[App] Running application")
    $(app)(_.run())

    println("\n[Scope] Exiting scoped region - finalizers will run")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/wire/WireBasicExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain wire.wireBasicExample"
```

Comparing shared vs unique semantics shows how shared wires reuse the same instance across dependents, while unique wires create fresh instances. View the example:

```scala title="scope-examples/src/main/scala/wire/WireSharedUniqueExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package wire

/**
 * Demonstrates the semantic difference between shared and unique wires:
 *   - Shared wires: same instance across dependents (reference-counted)
 *   - Unique wires: fresh instance per allocation
 *
 * Uses a counter to track how many times each service is instantiated.
 */

final class Counter {
  private val count = new AtomicInteger(0)

  def next(): Int = count.incrementAndGet()

  def value: Int = count.get()
}

final class ServiceA(counter: Counter) {
  val id = counter.next()
  println(s"[ServiceA] Initialized with counter id=$id")
}

final class ServiceB(counter: Counter) {
  val id = counter.next()
  println(s"[ServiceB] Initialized with counter id=$id")
}

final class SharedDependencyApp(a: ServiceA, b: ServiceB) {
  def checkSharing(): Boolean = {
    // If Counter is shared, both services got the same instance
    val aCountId = a.id
    val bCountId = b.id
    println(s"[SharedDependencyApp] ServiceA counter id=$aCountId, ServiceB counter id=$bCountId")
    aCountId != bCountId && a.id < b.id // Sequential IDs from same Counter
  }
}

final class UniqueDependencyApp(a: ServiceA, b: ServiceB) {
  def checkUniqueness(): Boolean = {
    // If Counter is unique, services got different instances (different starting IDs)
    val aCountId = a.id
    val bCountId = b.id
    println(s"[UniqueDependencyApp] ServiceA counter id=$aCountId, ServiceB counter id=$bCountId")
    aCountId != bCountId // Different Counter instances
  }
}

@main def wireSharedUniqueExample(): Unit = {
  println("=== Wire Shared vs Unique Example ===\n")

  println("--- Test 1: Shared Counter (diamond pattern) ---\n")

  // Using a SHARED Counter: both ServiceA and ServiceB share the same Counter instance
  val sharedResource: Resource[SharedDependencyApp] = Resource.from[SharedDependencyApp](
    Wire.shared[Counter] // Shared: one instance across the graph
  )

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app      = allocate(sharedResource)
    val isShared = $(app)(_.checkSharing())

    println(s"\n[Result] Counter was shared: $isShared\n")
  }

  println("\n--- Test 2: Unique Counter ---\n")

  // Using a UNIQUE Counter: each dependency gets a fresh Counter instance
  val uniqueResource: Resource[UniqueDependencyApp] = Resource.from[UniqueDependencyApp](
    Wire.unique[Counter] // Unique: fresh instance per dependency
  )

  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app      = allocate(uniqueResource)
    val isUnique = $(app)(_.checkUniqueness())

    println(s"\n[Result] Counters were unique: $isUnique\n")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/wire/WireSharedUniqueExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain wire.wireSharedUniqueExample"
```

Manual wire construction demonstrates how to use `fromFunction` for custom construction logic. View the example:

```scala title="scope-examples/src/main/scala/wire/WireFromFunctionExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package wire

/**
 * Demonstrates manual wire construction using fromFunction:
 *   - Wire.Shared.fromFunction for custom shared wire logic
 *   - Wire.Unique.fromFunction for custom unique wire logic
 *   - Using manual wires when macro derivation doesn't fit
 *
 * This is useful for complex initialization, conditional logic, or when you
 * need control over which dependencies to use.
 */

final case class ApiKey(value: String)

final case class HttpConfig(baseUrl: String, timeout: Int)

/**
 * A custom HTTP client that uses an API key and timeout from config.
 * Demonstrates a scenario where we want custom construction logic rather than
 * simple parameter passing.
 */
final class HttpClient(config: HttpConfig, apiKey: ApiKey) extends AutoCloseable {
  println(
    s"[HttpClient] Created with baseUrl=${config.baseUrl}, timeout=${config.timeout}ms, apiKey=${apiKey.value}"
  )

  def get(path: String): String = {
    println(s"[HttpClient] GET $path with api key")
    "response"
  }

  def close(): Unit =
    println("[HttpClient] Connection pool closed")
}

/**
 * A custom authenticator that needs the API key. Demonstrates context
 * extraction in a manual wire.
 */
final class Authenticator(apiKey: ApiKey) {
  println(s"[Authenticator] Using API key: ${apiKey.value}")

  def authenticate(): Boolean = {
    println("[Authenticator] Validating API key...")
    true
  }
}

final class ManualWireApp(client: HttpClient, auth: Authenticator) {
  def run(): Unit =
    if (auth.authenticate()) {
      val response = client.get("/api/users")
      println(s"[App] Got response: $response")
    }
}

@main def wireFromFunctionExample(): Unit = {
  println("=== Wire fromFunction (Manual Construction) Example ===\n")

  // Manually create wires using fromFunction for custom logic
  // This gives full control when macro derivation isn't suitable

  val httpClientWire: Wire.Shared[HttpConfig & ApiKey, HttpClient] =
    Wire.Shared.fromFunction { (scope, ctx) =>
      // Extract both dependencies from context
      val config = ctx.get[HttpConfig]
      val apiKey = ctx.get[ApiKey]

      // Custom initialization logic
      println("[Manual] Custom HttpClient construction")
      val client = new HttpClient(config, apiKey)

      // Register custom cleanup if needed (in addition to AutoCloseable)
      scope.defer(println("[Manual] HttpClient cleanup deferred"))

      client
    }

  val authenticatorWire: Wire.Shared[ApiKey, Authenticator] =
    Wire.Shared.fromFunction { (scope, ctx) =>
      val apiKey = ctx.get[ApiKey]

      // Custom initialization logic
      println("[Manual] Custom Authenticator construction")
      val auth = new Authenticator(apiKey)

      scope.defer(println("[Manual] Authenticator cleanup deferred"))

      auth
    }

  // Provide leaf dependencies
  val configWire = Wire(HttpConfig("https://api.example.com", 30000))
  val apiKeyWire = Wire(ApiKey("secret-key-12345"))

  // Compose into the app resource
  val appResource: Resource[ManualWireApp] = Resource.from[ManualWireApp](
    configWire,
    apiKeyWire,
    httpClientWire,
    authenticatorWire
  )

  println("[Setup] Created manual wires\n")

  // Allocate and run
  Scope.global.scoped { scope =>

    println("[Scope] Entering scoped region\n")

    val app = allocate(appResource)

    println("\n[App] Running application")
    $(app)(_.run())

    println("\n[Scope] Exiting scoped region - finalizers will run")
  }

  println("\n=== Example Complete ===")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/scope-examples/src/main/scala/wire/WireFromFunctionExample.scala))

Run this example with:

```bash
sbt "scope-examples/runMain wire.wireFromFunctionExample"
```

---

## SchemaError

`SchemaError` is a **structured error type** for schema operations in ZIO Blocks. It represents one or more validation, conversion, or structural failures that occurred while decoding, encoding, or transforming data, each annotated with a [`DynamicOptic`](./dynamic-optic.md) path that pinpoints the failing location in the data structure.

```scala
final case class SchemaError(errors: ::[SchemaError.Single])
  extends Exception with NoStackTrace
```

Here is the full structure of `SchemaError`:

```
SchemaError
 └── errors: ::[Single]       (non-empty list — always at least one failure)
      │
      └── Single (sealed trait)
           │ ├── source: DynamicOptic   (path to the failing location)
           │ └── message: String        (human-readable description)
           │
           ├── ConversionFailed       (type or value conversion failed)
           ├── MissingField           (required field absent)
           ├── DuplicatedField        (same field key appears more than once)
           ├── ExpectationMismatch    (wrong DynamicValue variant encountered)
           ├── UnknownCase            (unrecognised sealed-trait discriminator)
           └── Message                (free-form message, optional path)
```

`SchemaError`:

- Aggregates multiple independent failures into a single error value
- Annotates every failure with a precise traversal path through the data
- Extends `Exception` so it can be thrown and caught with standard JVM mechanisms
- Suppresses stack traces via `NoStackTrace` — error location is conveyed through the path, not the JVM stack

## Motivation

When decoding a complex nested value, a single structural problem — a missing field, a type mismatch, an unknown case discriminator — must be reported together with the location where it occurred. In a large schema, multiple independent problems can coexist, and surfacing them all at once saves the caller round-trips.

Every `Schema#fromDynamicValue`, every `Codec#decode`, and every optic traversal that can fail returns `Either[SchemaError, A]`. The same type carries both structural errors (missing fields, wrong types) and domain validation errors (value out of range, blank string), so callers deal with a single error channel.

`SchemaError` was introduced to replace the earlier `JsonError` and `DynamicValueError` types that existed as separate error channels for each format. Those types used string concatenation (`"error1; error2"`) to combine failures via `++`, which silently discarded path information from the second error onward. `SchemaError` solves this by maintaining a non-empty list (`::`) of `Single` failures — each one independently annotated with its own `DynamicOptic` path — so no information is lost during aggregation.

Here is a quick taste of how `SchemaError` behaves:

```scala

// Create a simple message error
val err = SchemaError("Age must be positive")

// Annotate with the location in the data
val located = SchemaError.missingField(Nil, "email").atField("user")
println(located.message)   // Missing field 'email' at: .user

// Combine two independent failures
val combined = SchemaError("name is blank") ++ SchemaError("age is negative")
println(combined.errors.length)  // 2
```

## Construction / Creating Instances

### `SchemaError.apply`

The simplest constructor — creates a free-form `Message` error at the root path:

```scala
object SchemaError {
  def apply(details: String): SchemaError
}
```

```scala

val err = SchemaError("Value must be positive")
println(err.message)  // Value must be positive
```

### `SchemaError.message`

Creates a `Message` error with a free-form description and an optional `DynamicOptic` path. When no path is supplied it defaults to the root.

```scala
object SchemaError {
  def message(details: String, path: DynamicOptic = DynamicOptic.root): SchemaError
}
```

Here we create a message error at the root, and another with an explicit path:

```scala

// Root-level message (same as SchemaError.apply)
val atRoot = SchemaError.message("Unexpected null")
println(atRoot.message)  // Unexpected null

// Message with an explicit path
val path   = DynamicOptic.root.field("address")
val atPath = SchemaError.message("Unexpected null", path)
println(atPath.message)  // Unexpected null at: .address
```

### `SchemaError.validationFailed`

Convenience factory for validation failures. Equivalent to `SchemaError.conversionFailed(Nil, message)`, designed for smart constructors that return string-based error messages.

```scala
object SchemaError {
  def validationFailed(message: String): SchemaError
}
```

Here is an example:

```scala

val err = SchemaError.validationFailed("Age must be between 0 and 150")
println(err.message)  // Age must be between 0 and 150
```

### `SchemaError.conversionFailed`

Creates a `ConversionFailed` error for a failed type or value conversion. Two overloads exist.

```scala
object SchemaError {
  def conversionFailed(trace: List[DynamicOptic.Node], details: String): SchemaError
  def conversionFailed(contextMessage: String, cause: SchemaError): SchemaError
}
```

The first overload is used by codecs — `trace` is the list of path nodes accumulated during decoding. Pass `Nil` when constructing an error manually and use the `at*` methods to set the path. The second overload wraps a nested `SchemaError` with additional context; the nested failures are rendered under a "Caused by:" section.

```scala

// Root-level conversion failure
val err = SchemaError.conversionFailed(Nil, "Expected a positive integer")
println(err.message)  // Expected a positive integer

// Wrapping a nested failure with context
val inner = SchemaError("name must not be empty") ++
            SchemaError("age must be positive")
val outer = SchemaError.conversionFailed("Person construction failed", inner)
println(outer.message)
// Person construction failed
//   Caused by:
//   - name must not be empty
//   - age must be positive
```

### `SchemaError.missingField`

Creates a `MissingField` error indicating that a required field was absent from the decoded representation.

```scala
object SchemaError {
  def missingField(trace: List[DynamicOptic.Node], fieldName: String): SchemaError
}
```

```scala

val err = SchemaError.missingField(Nil, "email").atField("user")
println(err.message)  // Missing field 'email' at: .user
```

### `SchemaError.duplicatedField`

Creates a `DuplicatedField` error indicating that the same field key appeared more than once in the encoded form.

```scala
object SchemaError {
  def duplicatedField(trace: List[DynamicOptic.Node], fieldName: String): SchemaError
}
```

```scala

val err = SchemaError.duplicatedField(Nil, "id").atField("record")
println(err.message)  // Duplicated field 'id' at: .record
```

### `SchemaError.expectationMismatch`

Creates an `ExpectationMismatch` error indicating that the encountered `DynamicValue` variant does not match what the schema expected.

```scala
object SchemaError {
  def expectationMismatch(trace: List[DynamicOptic.Node], expectation: String): SchemaError
}
```

```scala

val err = SchemaError
  .expectationMismatch(Nil, "Expected Record, got Sequence")
  .atField("data")
println(err.message)  // Expected Record, got Sequence at: .data
```

### `SchemaError.unknownCase`

Creates an `UnknownCase` error indicating that the decoded discriminator value does not correspond to any known variant of a sealed trait.

```scala
object SchemaError {
  def unknownCase(trace: List[DynamicOptic.Node], caseName: String): SchemaError
}
```

```scala

val err = SchemaError.unknownCase(Nil, "Triangle").atField("shape")
println(err.message)  // Unknown case 'Triangle' at: .shape
```

## Core Operations

### Error Messages

#### `message`

Returns all individual error messages joined with newlines.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def message: String
}
```

```scala

val err = SchemaError("first failure") ++ SchemaError("second failure")
println(err.message)
// first failure
// second failure
```

#### `getMessage`

Delegates to `message`. Because `SchemaError` extends `Exception`, `getMessage` is called by the JVM when the exception is printed or logged by frameworks.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def getMessage: String
}
```

```scala

val err = SchemaError("something went wrong")
assert(err.getMessage == err.message)
```

### Error Aggregation

#### `++`

Combines two `SchemaError` values into one, preserving all individual `Single` failures from both sides. We use `++` to accumulate errors from independent parts of a schema — for example, multiple record fields that are each decoded independently.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def ++(other: SchemaError): SchemaError
}
```

```scala

val nameError = SchemaError.missingField(Nil, "name")
val ageError  = SchemaError.conversionFailed(Nil, "Age must be positive")
val combined  = nameError ++ ageError

println(combined.errors.length)   // 2
println(combined.message)
// Missing field 'name' at: .
// Age must be positive
```

`++` is associative: `(a ++ b) ++ c` and `a ++ (b ++ c)` produce the same set of errors.

### Path Annotation

Path annotation methods prepend a path segment to the `source` of every `SchemaError.Single` inside the error. Codecs call these methods as they unwind the call stack — the innermost call adds the innermost path segment, and the outermost call adds the outermost one.

#### `atField`

Prepends a record field access to the path of all errors.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atField(name: String): SchemaError
}
```

```scala

// Codec decoding 'city' inside 'address' inside 'user'
val err = SchemaError.missingField(Nil, "city")
  .atField("address")  // called by the address codec
  .atField("user")     // called by the user codec
println(err.message)   // Missing field 'city' at: .user.address
```

#### `atIndex`

Prepends a sequence index access to the path of all errors.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atIndex(index: Int): SchemaError
}
```

```scala

val err = SchemaError("invalid phone number").atIndex(2).atField("phones")
println(err.message)  // invalid phone number at: .phones[2]
```

#### `atCase`

Prepends a sealed-trait case access to the path of all errors. In the compact path notation used by `Message`, the case name appears as `<CaseName>`.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atCase(name: String): SchemaError
}
```

```scala

val err = SchemaError("conversion failed").atField("value").atCase("Right")
println(err.message)  // conversion failed at: <Right>.value
```

#### `atKey`

Prepends a map key access to the path of all errors. The key is a [`DynamicValue`](./dynamic-value.md), rendered with `{key}` in the compact path notation.

```scala
final case class SchemaError(errors: ::[SchemaError.Single]) {
  def atKey(key: DynamicValue): SchemaError
}
```

```scala

val key = DynamicValue.string("config")
val err = SchemaError("missing required entry").atKey(key)
println(err.message)  // missing required entry at: {"config"}
```

### Path Chaining

All path methods can be chained. Each call prepends to the existing path, so the outermost call appears as the leftmost segment in the rendered message.

```scala

val err = SchemaError("value out of range")
  .atField("amount")       // innermost — added first
  .atIndex(0)
  .atCase("Credit")
  .atField("transactions") // outermost — added last
println(err.message)
// value out of range at: .transactions<Credit>[0].amount
```

Path annotation applies to **every** `Single` inside the error, so combined errors accumulate paths correctly:

```scala

val error1 = SchemaError.missingField(Nil, "name")
val error2 = SchemaError.conversionFailed(Nil, "age must be positive")
val combined = (error1 ++ error2).atField("person")

// Both errors now include the "person" field prefix
println(combined.errors.head.source.nodes.nonEmpty)  // true (name)
println(combined.errors.tail.head.source.nodes.nonEmpty)  // true (age)
```

## Subtypes / Variants

`SchemaError.Single` is the sealed base trait for every individual failure. Each variant carries a `source: DynamicOptic` and a `message: String`.

```
SchemaError.Single (sealed trait)
 ├── SchemaError.IntoError (sealed sub-trait — marks conversion errors)
 │   └── ConversionFailed(source, details, cause: Option[SchemaError])
 ├── MissingField(source, fieldName)
 ├── DuplicatedField(source, fieldName)
 ├── ExpectationMismatch(source, expectation)
 ├── UnknownCase(source, caseName)
 └── Message(source, details)
```

| Subtype               | Factory                                | Typical cause                                                                             |
|-----------------------|----------------------------------------|-------------------------------------------------------------------------------------------|
| `ConversionFailed`    | `conversionFailed`, `validationFailed` | Type conversion or smart-constructor failure; may carry a nested `SchemaError` as `cause` |
| `MissingField`        | `missingField`                         | Required field absent in the decoded representation                                       |
| `DuplicatedField`     | `duplicatedField`                      | Same field key appears more than once                                                     |
| `ExpectationMismatch` | `expectationMismatch`                  | Wrong `DynamicValue` variant encountered                                                  |
| `UnknownCase`         | `unknownCase`                          | Discriminator names an unrecognised sealed-trait variant                                  |
| `Message`             | `message`, `apply`                     | Free-form error with an optional path                                                     |

We can pattern match on `errors` to handle specific failure kinds:

```scala

val err = SchemaError.missingField(Nil, "email") ++
          SchemaError.conversionFailed(Nil, "age must be positive")

err.errors.foreach {
  case SchemaError.MissingField(source, fieldName) =>
    println(s"Missing '$fieldName' at ${source.toScalaString}")
  case SchemaError.ConversionFailed(source, details, _) =>
    println(s"Conversion failed: $details")
  case other =>
    println(other.message)
}
```

### `SchemaError.IntoError`

`IntoError` is a sealed sub-trait of `Single` that marks errors produced during `Into` (type conversion) operations. Its only current implementation is `ConversionFailed`. Codec code pattern-matches on `IntoError` to distinguish conversion errors from structural schema errors:

```scala
sealed trait IntoError extends SchemaError.Single {
  def source: DynamicOptic
}
```

### `SchemaError.ConversionFailed`

Represents a failed type or value conversion. When a `cause: Option[SchemaError]` is present, the rendered `message` includes a "Caused by:" section showing the nested failures.

```scala

// Single nested cause
val inner1 = SchemaError.conversionFailed(Nil, "name is blank")
val outer1 = SchemaError.conversionFailed("Person construction failed", inner1)
println(outer1.message)
// Person construction failed
//   Caused by: name is blank

// Multiple nested causes
val inner2 = SchemaError.conversionFailed(Nil, "name is blank") ++
             SchemaError.conversionFailed(Nil, "age is negative")
val outer2 = SchemaError.conversionFailed("Person construction failed", inner2)
println(outer2.message)
// Person construction failed
//   Caused by:
//   - name is blank
//   - age is negative
```

## Integration

### With Schema Decoding

`Schema#fromDynamicValue` returns `Either[SchemaError, A]`. Every codec accumulates path nodes during decoding and calls `atField`, `atIndex`, or `atCase` as it unwinds, producing a fully-annotated `SchemaError` on failure.

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val result: Either[SchemaError, Person] =
  Schema[Person].fromDynamicValue(Schema[Person].toDynamicValue(Person("Alice", 30)))

result match {
  case Right(person) => println(s"Decoded: $person")
  case Left(err)     => println(s"Error:\n${err.message}")
}
```

See [Schema](./schema.md) and [DynamicValue](./dynamic-value.md) for the full encoding and decoding API.

### With Schema#transform

`Schema#transform` accepts `to` and `from` functions that can throw `SchemaError` to signal validation failures during encoding or decoding. We use `SchemaError.validationFailed` (which wraps a `ConversionFailed`) to turn a smart-constructor rejection into a structured schema error:

```scala

case class PositiveInt private (value: Int)

object PositiveInt {
  def make(n: Int): PositiveInt =
    if (n > 0) PositiveInt(n)
    else throw SchemaError.validationFailed("must be positive")

  implicit val schema: Schema[PositiveInt] =
    Schema[Int].transform(make, _.value)
}
```

When `make` throws a `SchemaError`, the codec catches it, preserves the full error (including any path already annotated), and surfaces it as `Left(schemaError)` from `Schema#fromDynamicValue` or `Codec#decode`. See [Schema](./schema.md) for the full `transform` API.

A runnable version of this example, including composite types and error aggregation, is available in the `schema-examples` module:

```bash
sbt "schema-examples/runMain schemaerror.SchemaErrorExample"
```

### With Validation

The [Validation](./validation.md) system uses `SchemaError` to report constraint violations. When a `PrimitiveType` carries a `Validation` and the value fails the check, the codec surfaces a `SchemaError.ConversionFailed` at the appropriate path.

### With DynamicOptic and Optics

Path annotation (`atField`, `atIndex`, `atKey`, `atCase`) builds a [`DynamicOptic`](./dynamic-optic.md) inside each error. Operations such as `DynamicValue#setOrFail` and `DynamicValue#modifyAtPathOrFail` return `Either[SchemaError, DynamicValue]`, using the same factory methods.

```scala

val data   = DynamicValue.Sequence(DynamicValue.int(1), DynamicValue.int(2))
val optic  = DynamicOptic.root.at(10)

val result: Either[SchemaError, DynamicValue] = data.setOrFail(optic, DynamicValue.int(99))
result match {
  case Left(err) => println(err.message)  // index out of range or similar
  case Right(v)  => println(v)
}
```

### As an Exception

Because `SchemaError` extends `Exception`, it can be thrown and caught with standard try/catch (In functional code, prefer `Either[SchemaError, A]` or `Option[SchemaError]` instead):

```scala

try {
  throw SchemaError("Unexpected data shape")
} catch {
  case e: SchemaError => println(s"Caught schema error: ${e.getMessage}")
}
```

:::note
`SchemaError` extends `scala.util.control.NoStackTrace`. Stack traces are suppressed for performance — error location is conveyed through the `DynamicOptic` path inside each `Single`, not the JVM stack.
:::

---

## As

`As[A, B]` is a **bidirectional conversion type class** that extends `Into[A, B]` with a reverse direction. In addition to converting `A → B` via `As#into`, it also converts `B → A` via `As#from`, providing a round-trip guarantee.

`As`:
- extends `Into[A, B]`, so every `As` can be used wherever an `Into` is expected
- returns `Right(b)` or `Right(a)` on success and `Left(error)` on validation failure in both directions
- derives automatically for case classes, sealed traits, tuples, and Scala 3 enums via `As.derived`
- enforces stricter derivation constraints than `Into` to guarantee that `A → B → A` always restores the original value

```scala
trait As[A, B] extends Into[A, B] {
  def from(input: B): Either[SchemaError, A]
  def reverse: As[B, A]
}
```

The bidirectional data flow looks like this:

```
  ┌──────────────────────────────────────────────────┐
  │               As[A, B]                           │
  │                                                  │
  │     into(a: A) ──────────────────────► B         │
  │                                                  │
  │     from(b: B) ◄────────────────────── B         │
  │                                                  │
  │     reverse: As[B, A]  (flips directions)        │
  └──────────────────────────────────────────────────┘
```

`As` is the right choice when the conversion must be safe to run in both directions — for example when synchronising data between a local model and a remote representation, or when migrating a database schema that must remain rollback-capable.

## Installation

`As` is part of `zio-blocks-schema`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

For Scala.js and Scala Native, use `%%%`:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

There are three ways to obtain an `As[A, B]`: construct it from a pair of `Into` instances, derive it automatically with the macro, or summon an implicit already in scope.

### `As.apply` — Manual Construction

`As.apply(intoAB, intoBA)` composes two `Into` instances into one `As`:

```scala
object As {
  def apply[A, B](intoAB: Into[A, B], intoBA: Into[B, A]): As[A, B]
}
```

We build an `As[Int, Long]` by supplying both directions explicitly:

```scala

val intoAB: Into[Int, Long] = a => Right(a.toLong)
val intoBA: Into[Long, Int] = b =>
  if (b >= Int.MinValue && b <= Int.MaxValue) Right(b.toInt)
  else Left(SchemaError.validationFailed("overflow"))

val manualAs: As[Int, Long] = As(intoAB, intoBA)
```

With `manualAs` in scope we can convert in both directions and verify overflow detection:

```scala
manualAs.into(42)
// res0: Either[SchemaError, Long] = Right(42L)
manualAs.from(100L)
// res1: Either[SchemaError, Int] = Right(100)
manualAs.from(Long.MaxValue)
// res2: Either[SchemaError, Int] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "overflow",
//         cause = None
//       )
//     )
//   )
// )
```

### `As.derived` — Macro Derivation

`As.derived[A, B]` generates an `As[A, B]` at compile time by deriving both `Into[A, B]` and `Into[B, A]` and running bidirectional compatibility checks:

```scala
object As {
  def derived[A, B]: As[A, B]
}
```

The macro works with case classes, sealed traits, Scala 3 enums, tuples, ZIO Prelude newtypes, Scala 3 opaque types, and structural types (JVM only). We derive an `As` for two case classes with matching fields:

```scala

case class PersonA(name: String, age: Int)
case class PersonB(name: String, age: Long)

val personAs: As[PersonA, PersonB] = As.derived[PersonA, PersonB]
```

Both `As#into` and `As#from` are now available, and we can verify that `A → B → A` restores the original value:

```scala
personAs.into(PersonA("Alice", 30))
// res3: Either[SchemaError, PersonB] = Right(
//   PersonB(name = "Alice", age = 30L)
// )
personAs.from(PersonB("Bob", 25L))
// res4: Either[SchemaError, PersonA] = Right(PersonA(name = "Bob", age = 25))
personAs.into(PersonA("Alice", 30)).flatMap(personAs.from)
// res5: Either[SchemaError, PersonA] = Right(
//   PersonA(name = "Alice", age = 30)
// )
```

### `As.apply[A, B]` — Summoning

`As.apply[A, B]` (with no arguments) summons an implicit `As[A, B]` already in scope — the same pattern used by `Into.apply`:

```scala
object As {
  def apply[A, B](implicit ev: As[A, B]): As[A, B]
}
```

This is useful when you want to retrieve a type-class instance by type rather than by variable name:

```scala

case class Foo(x: Int)
case class Bar(x: Int)

implicit val fooBarAs: As[Foo, Bar] = As.derived[Foo, Bar]

// Summon the implicit instance
val summoned = As[Foo, Bar]
summoned.into(Foo(1))
summoned.from(Bar(2))
```

## Core Operations

`As` exposes three operations: `As#into`, `As#from`, and `As#reverse`.

### `As#into` — Forward Conversion

`As#into` is inherited from `Into[A, B]` and converts an `A` into `Either[SchemaError, B]`:

```scala
trait As[A, B] extends Into[A, B] {
  def into(a: A): Either[SchemaError, B]
}
```

### `As#from` — Reverse Conversion

`As#from` is the operation that distinguishes `As` from `Into`. It converts a `B` back to `Either[SchemaError, A]`:

```scala
trait As[A, B] {
  def from(b: B): Either[SchemaError, A]
}
```

We define two simple wrapper types and derive an `As` between them to show both directions:

```scala

case class IntBox(value: Int)
case class LongBox(value: Long)

val boxAs: As[IntBox, LongBox] = As.derived[IntBox, LongBox]
```

`As#into` widens the value while `As#from` narrows it, validating that the result fits in the target type:

```scala
boxAs.into(IntBox(42))
// res8: Either[SchemaError, LongBox] = Right(LongBox(42L))
boxAs.from(LongBox(99L))
// res9: Either[SchemaError, IntBox] = Right(IntBox(99))
boxAs.from(LongBox(Long.MaxValue))
// res10: Either[SchemaError, IntBox] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field LongBox.value to IntBox.value failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(IndexedSeq()),
//                 details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

### `As#reverse` — Flipping Directions

`As#reverse` returns an `As[B, A]` whose `As#into` and `As#from` are swapped:

```scala
trait As[A, B] {
  def reverse: As[B, A]
}
```

`As#reverse` creates a new `As` without touching the original:

```scala
val revAs: As[LongBox, IntBox] = boxAs.reverse
// revAs: As[LongBox, IntBox] = zio.blocks.schema.As$$anon$1@6145661b

revAs.into(LongBox(5L))
// res11: Either[SchemaError, IntBox] = Right(IntBox(5))
revAs.from(IntBox(10))
// res12: Either[SchemaError, LongBox] = Right(LongBox(10L))
```

## Using `As` as `Into`

Because `As[A, B]` extends `Into[A, B]`, any `As` instance can be passed wherever an `Into` is expected — with no casts or wrapping needed.

We write a generic migration helper that requires only an `Into`, then pass an `As` directly:

```scala

case class P2D(x: Int, y: Int)
case class Coord(x: Int, y: Int)

def migrate[A, B](data: A)(implicit into: Into[A, B]): Either[SchemaError, B] =
  into.into(data)

implicit val pointAs: As[P2D, Coord] = As.derived[P2D, Coord]
```

Passing `pointAs` where the function expects `Into[P2D, Coord]` works because `As` is a subtype of `Into`:

```scala
migrate(P2D(1, 2))
// res13: Either[SchemaError, Coord] = Right(Coord(x = 1, y = 2))
```

## `As.reverseInto` Implicit

`AsLowPriorityImplicits` provides `As.reverseInto`, an implicit that materialises an `Into[B, A]` from any `As[A, B]` in scope. This lets libraries that only require `Into` automatically benefit from `As` instances without any extra wiring:

```scala
trait AsLowPriorityImplicits {
  implicit def reverseInto[A, B](implicit as: As[A, B]): Into[B, A]
}
```

With an `As[String, Int]` in scope, `As.reverseInto` synthesises `Into[Int, String]` automatically:

```scala

implicit val stringIntAs: As[String, Int] = new As[String, Int] {
  def into(s: String): Either[SchemaError, Int] =
    try Right(s.toInt)
    catch { case _: NumberFormatException => Left(SchemaError.validationFailed("not an int")) }
  def from(n: Int): Either[SchemaError, String] = Right(n.toString)
}
```

We import `As.reverseInto` and use it to obtain the reverse `Into[Int, String]`:

```scala

val intToStr: Into[Int, String] = reverseInto[String, Int]
// intToStr: Into[Int, String] = zio.blocks.schema.AsLowPriorityImplicits$$Lambda$17549/0x00007faac29bf110@e1a6316
intToStr.into(42)
// res14: Either[SchemaError, String] = Right("42")
```

## Derivation Rules

`As.derived` applies the same rules as `Into.derived` in both directions and adds bidirectional compatibility checks on top. The derivation supports the same type categories as `Into`.

### Products (Case Classes and Tuples)

For two case classes `A` and `B`, the macro checks:
- fields with matching names must be convertible in **both** directions
- fields present in one type but absent from the other must be `Option` (defaults are not allowed — see [Restrictions](#restrictions))

We derive `As` for two structurally compatible case classes:

```scala

case class UserV1(name: String, age: Int)
case class UserV2(name: String, age: Long)

val userAs: As[UserV1, UserV2] = As.derived[UserV1, UserV2]
```

Tuples are matched positionally, so field name checks are skipped:

```scala

val tupleAs: As[(Int, String), (Long, String)] = As.derived[(Int, String), (Long, String)]
```

### Coproducts (Sealed Traits and Enums)

`As.derived` handles sealed traits and Scala 3 enums the same way `Into.derived` does — each subtype is matched by name and derived recursively:

```scala

sealed trait ShapeV1
object ShapeV1 {
  case class Circle(radius: Int)  extends ShapeV1
  case class Rect(w: Int, h: Int) extends ShapeV1
}

sealed trait ShapeV2
object ShapeV2 {
  case class Circle(radius: Long) extends ShapeV2
  case class Rect(w: Long, h: Long) extends ShapeV2
}

val shapeAs: As[ShapeV1, ShapeV2] = As.derived[ShapeV1, ShapeV2]
```

### Numeric Coercions

All numeric primitive types (`Byte`, `Short`, `Int`, `Long`, `Float`, `Double`) are bidirectionally coercible. Widening always succeeds; narrowing validates at runtime and returns a `Left` on overflow:

```scala

case class IntModel(value: Int)
case class LongModel(value: Long)

val numericAs: As[IntModel, LongModel] = As.derived[IntModel, LongModel]
```

A value within `Int` range round-trips without loss; one outside it fails on the way back:

```scala
numericAs.into(IntModel(1000)).flatMap(numericAs.from)
// res15: Either[SchemaError, IntModel] = Right(IntModel(1000))
numericAs.from(LongModel(Long.MaxValue))
// res16: Either[SchemaError, IntModel] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field LongModel.value to IntModel.value failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(IndexedSeq()),
//                 details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

## Restrictions

`As` enforces constraints that `Into` does not. Because `As.derived` must produce valid conversions in both directions, it rejects configurations that would silently lose data during a round-trip.

**Default values on asymmetric fields are rejected.** A field with a default that has no counterpart in the other type cannot be round-tripped: when converting back, the field is missing and there is no way to distinguish a real default from a missing value:

```scala

case class WithDefault(name: String, age: Int = 25)
case class NoDefault(name: String)

// Does NOT compile — age has a default but is absent from NoDefault:
// As.derived[WithDefault, NoDefault]
```

Default values are allowed when the field exists in **both** types, because the value is never discarded during the round-trip:

```scala

case class PersonA(name: String, age: Int = 25)
case class PersonB(name: String, age: Int)

As.derived[PersonA, PersonB]  // compiles — age is present in both types
```

**`Option` fields on one side are allowed.** An `Option` field absent from the other type round-trips cleanly: `Some(v)` becomes `None` after a round-trip, which is the only safe behaviour for a missing field:

```scala

case class TypeA(name: String, nickname: Option[String])
case class TypeB(name: String)

As.derived[TypeA, TypeB]  // compiles
```

**Numeric coercions must be invertible in both directions.** Widening `Int → Long` is automatically paired with narrowing `Long → Int`. The narrowing validates at runtime, so the round-trip is safe even though it can fail:

```scala

case class IntVersion(value: Int)
case class LongVersion(value: Long)

As.derived[IntVersion, LongVersion]  // compiles — widening + narrowing form a valid pair
```

**Fields present in one type but absent from the other must be `Option`.** A non-optional field that exists only on one side cannot be populated in the reverse direction:

```scala

case class Short_(name: String)
case class Long_(name: String, extra: String)

// Does NOT compile — extra is not Optional and does not exist in Short_:
// As.derived[Short_, Long_]

case class Long2_(name: String, extra: Option[String])

As.derived[Short_, Long2_]  // compiles — extra is Optional
```

## DynamicValue Conversions

Like `Into`, `As` supports bidirectional conversions with `DynamicValue`, allowing you to define a single schema and use it for both type-safe operations and polyglot data handling.

### Bidirectional DynamicValue Support with JSON Round-Trip

You can derive `As[A, DynamicValue]` for any type with a `Schema[A]` and achieve full polyglot round-trips:

```scala

case class Config(host: String, port: Int)

object Config {
  implicit val schema: Schema[Config] = Schema.derived[Config]
  val asDynamic: As[Config, DynamicValue] = As.derived[Config, DynamicValue]
}
```

```scala
// Forward: Config → DynamicValue → JSON
Config.asDynamic.into(Config("localhost", 8080)).map(_.toJsonString)
// res21: Either[SchemaError, String] = Right(
//   "{\"host\":\"localhost\",\"port\":8080}"
// )
```

Now in the reverse direction, deserialize JSON back to Config:

```scala

case class Config(host: String, port: Int)

object Config {
  implicit val schema: Schema[Config] = Schema.derived[Config]
  val asDynamic: As[Config, DynamicValue] = As.derived[Config, DynamicValue]

  // JSON string to parse
  val jsonString = """{"host":"example.com","port":9000}"""
}
```

```scala
// Reverse: JSON → DynamicValue → Config
for {
  dv <- Config.jsonString.fromJson[DynamicValue]
  config <- Config.asDynamic.from(dv)
} yield config
// res22: Either[SchemaError, Config] = Right(
//   Config(host = "example.com", port = 9000)
// )
```

The call to `jsonString.fromJson[DynamicValue]` parses the JSON string into a `DynamicValue`, and `asDynamic.from` converts it back to the strongly-typed `Config`. (Equivalently, you could use `Schema[DynamicValue].getInstance(JsonFormat).decode(jsonString)` for the same decoding step.) This demonstrates the full cycle: **JSON → DynamicValue → Type**, ensuring perfect round-trip fidelity.

### Use Cases

`As` is ideal when data must flow in both directions within the same system, with guarantees that neither direction silently loses or corrupts data.

#### Polyglot configuration systems

Configuration is often stored externally (Consul, etcd, a JSON file) and must be read, modified in-place, and written back. A naive approach requires two separate conversions — `Into[DynamicValue, DatabaseConfig]` to read and `Into[DatabaseConfig, DynamicValue]` to write — with no guarantee they align. `As` solves this by providing a single bidirectional instance that the macro verifies will round-trip faithfully.

Consider a service that:
1. Reads config from an external store (JSON → `DynamicValue` → typed `DatabaseConfig`)
2. Applies business logic to the typed config (validate, scale, migrate)
3. Writes the updated config back to the store (typed `DatabaseConfig` → `DynamicValue` → JSON)

Without `As`, step 3 might serialize data differently than step 1 read it, causing silent corruption or misalignment. With `As`, the macro guarantees that `config → DynamicValue → config'` preserves the structure.

```scala

case class DatabaseConfig(host: String, port: Int, timeout: Long)

object DatabaseConfig {
  implicit val schema: Schema[DatabaseConfig] = Schema.derived[DatabaseConfig]
  val asDynamic: As[DatabaseConfig, DynamicValue] = As.derived[DatabaseConfig, DynamicValue]
}
```

```scala
// Simulate JSON arriving from the config store (e.g. Consul, etcd, a JSON file)
val storedJson = """{"host":"db.prod.example.com","port":5432,"timeout":30000}"""
// storedJson: String = "{\"host\":\"db.prod.example.com\",\"port\":5432,\"timeout\":30000}"

val result = for {
  stored  <- storedJson.fromJson[DynamicValue]          // Step 1: Read from store
  config  <- DatabaseConfig.asDynamic.from(stored)      // Step 2a: Hydrate into typed config
  updated  = config.copy(timeout = 60000)               // Step 2b: Apply business logic
  written <- DatabaseConfig.asDynamic.into(updated)     // Step 3: Serialize back to store (guaranteed round-trip)
} yield written.toJsonString
// result: Either[SchemaError, String] = Right(
//   "{\"host\":\"db.prod.example.com\",\"port\":5432,\"timeout\":60000}"
// )

result
// res23: Either[SchemaError, String] = Right(
//   "{\"host\":\"db.prod.example.com\",\"port\":5432,\"timeout\":60000}"
// )
```

## Scala 2 vs Scala 3 Differences

| Feature | Scala 2 | Scala 3 |
|---------|---------|---------|
| Derivation syntax | `As.derived[A, B]` | `As.derived[A, B]` |
| Enum support | Sealed traits only | Scala 3 enums + sealed traits |
| Opaque types | N/A | ✅ Supported |
| Structural types | JVM only (reflection) | JVM only (reflection) |
| ZIO Prelude newtypes | ✅ `assert { between(...) }` | ✅ `override def assertion` |
| Error messages | Detailed macro errors | Detailed macro errors |
| DynamicValue ambiguity detection | ✅ Two-pass implicit resolution | ✅ Built-in ambiguity reporting |

## Integration

`As[A, B]` is defined in `zio.blocks.schema` alongside `Into[A, B]`. Because `As` is a subtype of `Into`, the two type classes compose naturally: you can derive an outer `As` from inner `As` instances, or mix `As` and custom `Into` instances when some fields need one-way or custom logic.

For a full reference on one-way conversions and the derivation rules that `As` builds on, see [Into](./into.md).

---

## Schema Evolution

Schema evolution is the process of changing data structures over time while keeping existing data readable and systems interoperable. ZIO Blocks provides two type classes for this: `Into` for one-way conversions and `As` for bidirectional round-trip conversions.

```
         Into[A, B]                    As[A, B]
  ─────────────────────        ──────────────────────────
  A ──── into(a) ──── B        A ──── into(a) ────► B
                               A ◄─── from(b) ──── B
  One-way, asymmetric           Bidirectional, round-trip
  Allows defaults, drops        Requires fields to match
  extra fields freely           or be Option; no defaults
                                on asymmetric fields
```

## `Into[A, B]` — One-Way Conversion

[`Into[A, B]`](./into.md) converts a value of type `A` to `Either[SchemaError, B]`. It is the right choice whenever the migration is asymmetric — for example, when adding a field with a default value, removing a field, or transforming data in a way that cannot be reversed.

Typical use cases:

- Migrating records from an old schema version to a new one
- Translating an external DTO into a validated domain model
- Converting API responses to internal representations

## `As[A, B]` — Bidirectional Round-Trip

[`As[A, B]`](./as.md) extends `Into[A, B]` with a `from(b: B): Either[SchemaError, A]` reverse direction. It guarantees that `A → B → A` restores the original value (within the constraints of numeric precision and optional fields). Use `As` when both sides of the conversion must remain in sync.

Typical use cases:

- Synchronising a local model with a remote representation
- Persisting to a data format that must be readable back into the same type
- Bridging two live systems that both produce and consume the same data

## Choosing Between `Into` and `As`

| | `Into[A, B]` | `As[A, B]` |
|---|---|---|
| Reverse conversion | ✗ | ✅ `from(b)` |
| Default values on extra fields | ✅ allowed | ✗ not allowed |
| Optional asymmetric fields | ✅ | ✅ (`Option` only) |
| Numeric coercion | ✅ (widening + narrowing) | ✅ (must be invertible) |
| Use for one-way migrations | ✅ | possible but overly strict |
| Use for bidirectional sync | manual | ✅ |

When in doubt, start with `Into`. Upgrade to `As` only when you need the reverse direction and can satisfy its stricter derivation requirements.

---

## Into

`Into[-A, +B]` is a **one-way conversion type class** that converts values of type `A` into values of type `B`, returning `Either[SchemaError, B]` to represent both successful conversions and validation failures. The fundamental operation is `Into#into`, which performs the conversion at runtime.

`Into`:
- is contravariant in `A` and covariant in `B`, following standard type class variance
- returns `Right(b)` on success and `Left(error)` on validation failure
- accumulates multiple field errors into a single `SchemaError`
- derives automatically for case classes, sealed traits, tuples, and Scala 3 enums via `Into.derived`

```scala
trait Into[-A, +B] {
  def into(a: A): Either[SchemaError, B]
}
```

The variance and data flow can be visualised as:

```
  A ──── into ────► Either[SchemaError, B]
  │                       │
  │                  Left(error)    ← validation failure
  │                  Right(b)       ← successful conversion
  │
  Contravariant in A, Covariant in B
```

## Motivation

`Into` solves a common challenge in Scala applications: **type-safe, validated conversion between structurally similar but different types**. This arises in:

- **Schema evolution**: migrating data from an old API version to a new one
- **Domain boundaries**: converting between external DTOs and internal domain models
- **Type refinement**: promoting raw primitives into validated wrapper types
- **Collection reshaping**: converting between `List`, `Vector`, `Set`, `Array`, etc.

Without `Into`, developers write boilerplate conversion code that silently mismatches fields, misses validation, or accumulates errors inconsistently. `Into.derived` generates all of this automatically at compile time.

```scala

case class PersonV1(name: String, age: Int)
case class PersonV2(name: String, age: Long, email: Option[String])

val migrate = Into.derived[PersonV1, PersonV2]
```

With `migrate` derived, converting a `PersonV1` widens `age` to `Long` and defaults `email` to `None`:

```scala
migrate.into(PersonV1("Alice", 30))
// res0: Either[SchemaError, PersonV2] = Right(
//   PersonV2(name = "Alice", age = 30L, email = None)
// )
```

Compare this to a manual implementation:

| Approach          | Field mismatch detection | Error accumulation      | Collection coercion    |
|-------------------|--------------------------|-------------------------|------------------------|
| Manual conversion | ❌ Compile-time miss      | ❌ Requires custom logic | ❌ Requires custom code |
| `Into.derived`    | ✅ Compile-time check     | ✅ Automatic             | ✅ Automatic            |

## Installation

`Into` is part of the `zio-blocks-schema` module:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

For Scala.js:

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

There are four ways to obtain an `Into[A, B]` instance: summon a pre-existing implicit, derive one at compile time via macro, use the built-in identity instance, or implement the trait directly for custom logic.

### `Into.apply` — Summoning

Summons an implicit `Into[A, B]` instance from the implicit scope. This is the standard way to access a pre-existing instance:

```scala
object Into {
  def apply[A, B](implicit ev: Into[A, B]): Into[A, B]
}
```

We summon the pre-existing `Into[Int, Long]` widening instance and call `Into#into` on it:

```scala

val intToLong: Into[Int, Long] = Into[Int, Long]
```

With `intToLong` in scope, `Into#into` converts the value and returns a `Right`:

```scala
intToLong.into(42)
// res1: Either[SchemaError, Long] = Right(42L)
```

### `Into.derived` — Macro Derivation

Generates an `Into[A, B]` instance at compile time using a macro. This is the primary way to convert between case classes, sealed traits, tuples, and enums:

```scala
object Into {
  def derived[A, B]: Into[A, B] // macro
}
```

We derive the conversion between two case classes and observe the `count` field being widened from `Int` to `Long`:

```scala

case class Source(name: String, count: Int)
case class Target(name: String, count: Long)

val conv = Into.derived[Source, Target]
```

The derived `conv` maps each field by name, coercing types where needed:

```scala
conv.into(Source("events", 100))
// res2: Either[SchemaError, Target] = Right(
//   Target(name = "events", count = 100L)
// )
```

### `Into.identity` — Identity Conversion

A pre-provided implicit `Into[A, A]` that always succeeds. It is always in scope and is resolved automatically when source and target types are the same:

```scala
object Into {
  implicit def identity[A]: Into[A, A]
}
```

Any `Into[A, A]` resolves to this built-in — there is nothing to configure:

```scala

val same: Into[String, String] = Into[String, String]
```

The identity conversion always returns `Right` wrapping the original value:

```scala
same.into("hello")
// res3: Either[SchemaError, String] = Right("hello")
```

### Custom Instances

We can implement `Into` manually for any types that need custom conversion logic:

```scala

case class Celsius(value: Double)
case class Fahrenheit(value: Double)

implicit val celsiusToFahrenheit: Into[Celsius, Fahrenheit] =
  (c: Celsius) => Right(Fahrenheit(c.value * 9.0 / 5.0 + 32.0))
```

With `celsiusToFahrenheit` in implicit scope, `Into[Celsius, Fahrenheit]` resolves to it automatically:

```scala
Into[Celsius, Fahrenheit].into(Celsius(100.0))
// res4: Either[SchemaError, Fahrenheit] = Right(Fahrenheit(212.0))
```

## Predefined Instances

ZIO Blocks ships built-in `Into` instances for all standard numeric types and common container types. These are resolved automatically from implicit scope — no import or explicit call is needed.

### Numeric Widening (Lossless)

These instances always succeed because the conversion cannot lose information:

| From \ To | `Short` | `Int` | `Long` | `Float` | `Double` |
|-----------|---------|-------|--------|---------|----------|
| `Byte`    | ✅       | ✅     | ✅      | ✅       | ✅        |
| `Short`   |         | ✅     | ✅      | ✅       | ✅        |
| `Int`     |         |       | ✅      | ✅       | ✅        |
| `Long`    |         |       |        | ✅       | ✅        |
| `Float`   |         |       |        |         | ✅        |

Each widening conversion always returns `Right` since no information is lost:

```scala
Into[Byte, Int].into(42.toByte)
// res5: Either[SchemaError, Int] = Right(42)
Into[Int, Long].into(100)
// res6: Either[SchemaError, Long] = Right(100L)
Into[Float, Double].into(3.14f)
// res7: Either[SchemaError, Double] = Right(3.140000104904175)
```

### Numeric Narrowing (With Validation)

These instances check at runtime whether the value fits in the target type. They return `Left(SchemaError)` when the value is out of range or cannot be precisely represented:

| From     | To      | Fails when                                         |
|----------|---------|----------------------------------------------------|
| `Short`  | `Byte`  | value outside `[-128, 127]`                        |
| `Int`    | `Byte`  | value outside `[-128, 127]`                        |
| `Int`    | `Short` | value outside `[-32768, 32767]`                    |
| `Long`   | `Byte`  | value outside `[-128, 127]`                        |
| `Long`   | `Short` | value outside `[-32768, 32767]`                    |
| `Long`   | `Int`   | value outside `[Int.MinValue, Int.MaxValue]`       |
| `Double` | `Float` | value outside Float range                          |
| `Float`  | `Int`   | value is not a whole number, or outside Int range  |
| `Float`  | `Long`  | value is not a whole number, or outside Long range |
| `Double` | `Int`   | value is not a whole number, or outside Int range  |
| `Double` | `Long`  | value is not a whole number, or outside Long range |

A value within range returns `Right`; an overflow or fractional value returns `Left`:

```scala
Into[Long, Int].into(42L)
// res8: Either[SchemaError, Int] = Right(42)
Into[Long, Int].into(Long.MaxValue)
// res9: Either[SchemaError, Int] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//         cause = None
//       )
//     )
//   )
// )
Into[Double, Int].into(3.14)
// res10: Either[SchemaError, Int] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "Value 3.14 cannot be precisely converted to Int",
//         cause = None
//       )
//     )
//   )
// )
```

### Container Instances

`Into` composes through standard container types automatically:

#### `Option`

`optionInto` lifts an `Into[A, B]` to work over `Option`, coercing the element when present and passing `None` through unchanged:

```scala
implicit def optionInto[A, B](implicit into: Into[A, B]): Into[Option[A], Option[B]]
```

Both `Some` and `None` are handled:

```scala
Into[Option[Int], Option[Long]].into(Some(42))
// res11: Either[SchemaError, Option[Long]] = Right(Some(42L))
Into[Option[Int], Option[Long]].into(None)
// res12: Either[SchemaError, Option[Long]] = Right(None)
```

#### `Either`

`eitherInto` coerces both branches independently, requiring separate `Into` instances for the left and right types:

```scala
implicit def eitherInto[L1, R1, L2, R2](
  implicit leftInto: Into[L1, L2],
  rightInto: Into[R1, R2]
): Into[Either[L1, R1], Either[L2, R2]]
```

Both `Left` and `Right` branches are coerced independently:

```scala
Into[Either[Int, Int], Either[Long, Long]].into(Right(1))
// res13: Either[SchemaError, Either[Long, Long]] = Right(Right(1L))
Into[Either[Int, Int], Either[Long, Long]].into(Left(2))
// res14: Either[SchemaError, Either[Long, Long]] = Right(Left(2L))
```

#### `Map`

`mapInto` coerces both keys and values, requiring separate `Into` instances for each:

```scala
implicit def mapInto[K1, V1, K2, V2](
  implicit keyInto: Into[K1, K2],
  valueInto: Into[V1, V2]
): Into[Map[K1, V1], Map[K2, V2]]
```

Both keys and values are coerced element-by-element:

```scala
Into[Map[String, Int], Map[String, Long]].into(Map("a" -> 1, "b" -> 2))
// res15: Either[SchemaError, Map[String, Long]] = Right(
//   Map("a" -> 1L, "b" -> 2L)
// )
```

#### Iterables and Arrays

Four overloads cover all combinations of `Iterable` subtypes and arrays as source or target:

```scala
implicit def iterableInto[A, B, F1[X] <: Iterable[X], F2[_]](
  implicit intoAB: Into[A, B],
  factory: Factory[B, F2[B]]
): Into[F1[A], F2[B]]

implicit def arrayToIterable[A, B, F[_]](
  implicit intoAB: Into[A, B],
  factory: Factory[B, F[B]]
): Into[Array[A], F[B]]

implicit def iterableToArray[A, B, F[X] <: Iterable[X]](
  implicit intoAB: Into[A, B],
  ct: ClassTag[B]
): Into[F[A], Array[B]]

implicit def arrayToArray[A, B](
  implicit intoAB: Into[A, B],
  ct: ClassTag[B]
): Into[Array[A], Array[B]]
```

The source and target collection kinds are independent — elements are coerced individually and the target collection is built using its factory:

```scala
Into[List[Int], Vector[Long]].into(List(1, 2, 3))
// res16: Either[SchemaError, Vector[Long]] = Right(Vector(1L, 2L, 3L))
Into[List[Int], Set[Long]].into(List(1, 2, 2, 3))
// res17: Either[SchemaError, Set[Long]] = Right(Set(1L, 2L, 3L))
```

:::note
Converting to `Set` removes duplicates. Converting from `Set` does not guarantee any particular element order.
:::

## Core Operation

`Into` exposes a single abstract method, `Into#into`. All predefined instances, derived instances, and custom implementations reduce to this one operation. It performs the conversion from `A` to `B`, returning a `Right` on success or a `Left` with a `SchemaError` on failure.

```scala
trait Into[-A, +B] {
  def into(a: A): Either[SchemaError, B]
}
```

We derive an `Into[Raw, Narrow]` to show both the success path and the overflow path:

```scala

case class Raw(value: Long)
case class Narrow(value: Int)

val conv = Into.derived[Raw, Narrow]
```

A value that fits in `Int` returns `Right`; a value that overflows returns `Left`:

```scala
conv.into(Raw(42L))
// res18: Either[SchemaError, Narrow] = Right(Narrow(42))
conv.into(Raw(Long.MaxValue))
// res19: Either[SchemaError, Narrow] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field Raw.value to Narrow.value failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(IndexedSeq()),
//                 details = "Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

## Macro Derivation Rules

`Into.derived[A, B]` generates a conversion by matching fields from `A` to `B` using the following priority:

1. **Exact match**: same field name and same type
2. **Name match with coercion**: same name, types connected by an implicit `Into` (e.g. `Int` → `Long`)
3. **Unique type match**: the type appears exactly once in both `A` and `B`
4. **Position + type match**: fields in the same position with matching types

### Products (Case Classes and Tuples)

Fields are matched by name first; when names differ but types are unique across both types, unique-type matching kicks in:

```scala

case class Source(firstName: String, count: Int)
case class Target(label: String, total: Long)
```

Because `String` and `Long` each appear uniquely, the macro resolves `firstName` → `label` and `count` → `total`:

```scala
Into.derived[Source, Target].into(Source("events", 5))
// res20: Either[SchemaError, Target] = Right(
//   Target(label = "events", total = 5L)
// )
```

Tuples and case classes are interchangeable when their arities and element types match:

```scala

case class Point(x: Int, y: Int)
```

The macro treats a two-element tuple and a two-field case class as structurally equivalent:

```scala
Into.derived[(Int, Int), Point].into((3, 4))
// res21: Either[SchemaError, Point] = Right(Point(x = 3, y = 4))
Into.derived[Point, (Int, Int)].into(Point(3, 4))
// res22: Either[SchemaError, Tuple2[Int, Int]] = Right((3, 4))
```

Target fields missing from the source default to `None` for `Option` types and to their declared default value otherwise:

```scala

case class Source(name: String)
case class Target(name: String, nickname: Option[String], score: Int = 0)
```

Missing fields are filled with `None` or their declared defaults — no extra code is needed:

```scala
Into.derived[Source, Target].into(Source("Alice"))
// res23: Either[SchemaError, Target] = Right(
//   Target(name = "Alice", nickname = None, score = 0)
// )
```

For nested case classes, the macro automatically picks up implicit `Into` instances for the nested types. Defining the inner conversion as an implicit is enough — the outer derivation uses it automatically:

```scala

case class AddressV1(street: String, zip: Int)
case class AddressV2(street: String, zip: Long)

case class PersonV1(name: String, address: AddressV1)
case class PersonV2(name: String, address: AddressV2)

implicit val addressConv: Into[AddressV1, AddressV2] =
  Into.derived[AddressV1, AddressV2]

val personConv = Into.derived[PersonV1, PersonV2]
```

The `personConv` conversion delegates the `address` field to `addressConv` without any extra wiring:

```scala
personConv.into(PersonV1("Alice", AddressV1("123 Main St", 10001)))
// res24: Either[SchemaError, PersonV2] = Right(
//   PersonV2(
//     name = "Alice",
//     address = AddressV2(street = "123 Main St", zip = 10001L)
//   )
// )
```

### Coproducts (Sealed Traits and Enums)

Cases are matched by name; for case classes, field types must be convertible. Target coproducts may introduce new cases that are unreachable from the source — the macro requires only that every source case has a corresponding target case by name.

For case class variants, fields are coerced just like in product derivation:

```scala

sealed trait ShapeV1
object ShapeV1 {
  case class Circle(radius: Int) extends ShapeV1
  case class Square(side: Int) extends ShapeV1
}

sealed trait ShapeV2
object ShapeV2 {
  case class Circle(radius: Long) extends ShapeV2
  case class Square(side: Long) extends ShapeV2
}

val conv = Into.derived[ShapeV1, ShapeV2]
```

Each case is matched by name and its fields are coerced from `Int` to `Long`:

```scala
conv.into(ShapeV1.Circle(5))
conv.into(ShapeV1.Square(3))
```

For `case object` variants (no fields), the macro matches by name alone. New cases may be added to the target without affecting derivation:

```scala

sealed trait StatusV1
object StatusV1 {
  case object Active   extends StatusV1
  case object Inactive extends StatusV1
}

sealed trait StatusV2
object StatusV2 {
  case object Active   extends StatusV2
  case object Inactive extends StatusV2
  case object Pending  extends StatusV2  // new in V2 — unreachable from V1
}

val conv = Into.derived[StatusV1, StatusV2]
```

Each source case object maps to the identically-named target case object:

```scala
conv.into(StatusV1.Active)
conv.into(StatusV1.Inactive)
```

### ZIO Prelude Newtypes

`Into.derived` automatically detects ZIO Prelude `Newtype` and `Subtype` definitions and validates values through their smart constructors. The syntax for defining the assertion differs between Scala versions.

**Scala 2:**

```scala
object Age extends Subtype[Int] {
  override def assertion = assert {
    between(0, 150)
  }
}
```

**Scala 3:**

```scala
object Age extends Subtype[Int] {
  override def assertion: Assertion[Int] =
    zio.prelude.Assertion.between(0, 150)
}
```

The Scala 3 form is used in the mdoc examples below:

```scala

object Age extends Subtype[Int] {
  override def assertion: zio.prelude.Assertion[Int] =
    zio.prelude.Assertion.between(0, 150)
}
type Age = Age.Type

case class PersonRaw(name: String, age: Int)
case class PersonValidated(name: String, age: Age)

val validate = Into.derived[PersonRaw, PersonValidated]
```

Values within the assertion range succeed; out-of-range values return a `Left` from the smart constructor:

```scala
validate.into(PersonRaw("Alice", 30))
// res26: Either[SchemaError, PersonValidated] = Right(
//   PersonValidated(name = "Alice", age = 30)
// )
validate.into(PersonRaw("Bob", 200))
// res27: Either[SchemaError, PersonValidated] = Left(
//   SchemaError(
//     List(
//       ConversionFailed(
//         source = DynamicOptic(IndexedSeq()),
//         details = "converting field PersonRaw.age to PersonValidated.age failed",
//         cause = Some(
//           SchemaError(
//             List(
//               ConversionFailed(
//                 source = DynamicOptic(IndexedSeq()),
//                 details = "Validation failed for field 'age': NonEmptyChunk(200 did not satisfy between(0, 150))",
//                 cause = None
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
```

### Scala 3 Opaque Types

In Scala 3, `Into.derived` detects opaque types with companion `apply` or `unsafe` methods:

```scala

opaque type Email = String
object Email {
  def apply(s: String): Either[String, Email] =
    if (s.contains("@")) Right(s) else Left(s"Invalid email: $s")
  def unsafe(s: String): Email = s
}

case class UserRaw(name: String, email: String)
case class UserValidated(name: String, email: Email)

val validate = Into.derived[UserRaw, UserValidated]
```

A valid email address succeeds; an invalid one returns the error produced by the `apply` smart constructor:

```scala
validate.into(UserRaw("Alice", "alice@example.com"))
// res29: Either[SchemaError, UserValidated] = Right(
//   UserValidated(name = "Alice", email = "alice@example.com")
// )
validate.into(UserRaw("Alice", "not-an-email"))
// res30: Either[SchemaError, UserValidated] = Right(
//   UserValidated(name = "Alice", email = "not-an-email")
// )
```

The macro looks for `apply(value: Underlying): Either[_, OpaqueType]` first, then falls back to `unsafe(value: Underlying): OpaqueType`.

### Structural Types (JVM Only)

On JVM, `Into.derived` supports structural types (types defined by their members rather than their name). This is not available on Scala.js or Scala Native because structural type access requires runtime reflection.

| Conversion              | JVM | JS/Native |
|-------------------------|-----|-----------|
| Structural → Product    | ✅  | ❌         |
| Product → Structural    | ✅  | ❌         |

On non-JVM platforms, `Into.derived` fails at compile time with a descriptive message:

```
Cannot derive Into[..., Person]: Structural type conversions are not supported on JS.
Structural types require reflection APIs which are only available on JVM.
Consider using a case class or tuple instead.
```

On JVM, we use `scala.language.reflectiveCalls` and create the structural instance at the call site:

```scala
// JVM ONLY — structural types require reflection

def makePerson(n: String, a: Int): { def name: String; def age: Int } = new {
  def name: String = n
  def age: Int = a
}

case class Person(name: String, age: Int)

val into = Into.derived[{ def name: String; def age: Int }, Person]
// into.into(makePerson("Alice", 30)) == Right(Person("Alice", 30))
```

:::warning
For cross-platform code, replace structural types with case classes or tuples.
:::

## Error Handling

All `Into` conversions return `Either[SchemaError, B]`. `SchemaError` carries:

- A human-readable message via `.message` / `.getMessage`
- The field path where the failure occurred
- Accumulated errors from multiple failing fields

```scala

case class Source(a: Long, b: Long, c: Long)
case class Target(a: Int,  b: Int,  c: Int)

val conv    = Into.derived[Source, Target]
val result  = conv.into(Source(Long.MaxValue, Long.MinValue, 42L))
```

We pattern-match on the result to print either the converted value or the accumulated error message:

```scala
result match {
  case Right(t)    => println(s"OK: $t")
  case Left(error) => println(s"Failed:\n${error.message}")
}
// Failed:
// converting field Source.b to Target.b failed
//   Caused by: Value -9223372036854775808 is out of range for Int [-2147483648, 2147483647]
// converting field Source.a to Target.a failed
//   Caused by: Value 9223372036854775807 is out of range for Int [-2147483648, 2147483647]
```

When multiple fields fail, all errors are collected and reported together. The field `c` above succeeds (42 fits in `Int`), so only errors for `a` and `b` appear.

```scala

case class UserRaw(id: Long, email: String, age: Long)

opaque type PositiveId = Long
object PositiveId {
  def apply(n: Long): Either[String, PositiveId] =
    if (n > 0) Right(n) else Left(s"id must be positive, got $n")
  def unsafe(n: Long): PositiveId = n
}

opaque type Email = String
object Email {
  def apply(s: String): Either[String, Email] =
    if (s.contains("@")) Right(s) else Left(s"Invalid email: $s")
  def unsafe(s: String): Email = s
}

case class UserValidated(id: PositiveId, email: Email, age: Int)

val conv = Into.derived[UserRaw, UserValidated]
val res = conv.into(UserRaw(-1L, "not-an-email", 200L))
```

All three field errors are accumulated into a single `SchemaError` with a combined message:

```scala
res match {
  case Left(error) => println(s"Validation failed: ${error.message}")
  case Right(value) => println(s"Validation succeeded: $value")
} 
// Validation succeeded: UserValidated(-1,not-an-email,200)
```

## DynamicValue Conversions

`Into` has special macro support for converting any type with a `Schema` to or from `DynamicValue`, a semi-structured data representation. This is the primary way to achieve polyglot data handling—converting between type-safe domain models and formats like JSON, Avro, or Protobuf.

### Converting to DynamicValue and JSON

The simplest way to convert to DynamicValue and view as JSON:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  val toDynamic: Into[Person, DynamicValue] = Into.derived[Person, DynamicValue]
}
```

```scala
Person.toDynamic.into(Person("Alice", 30)).map(_.toJsonString)
// res34: Either[SchemaError, String] = Right(
//   "{\"name\":\"Alice\",\"age\":30}"
// )
```

The `toJsonString` method on `DynamicValue` provides a human-readable JSON representation (Extended JSON format with type annotations). The conversion uses `Schema[A].toDynamicValue` internally, ensuring consistency with how the type is serialized to other formats.

**Why this matters:**

- **Format independence**: Convert typed data once to DynamicValue, then serialize to any format (JSON, Avro, MessagePack, etc.)
- **Dynamic pipelines**: Accept or produce semi-structured data in systems that don't have compile-time type information
- **Schema-driven workflows**: Use the same schema definition for both type-safe operations and dynamic transformations

### Converting from DynamicValue with Round-Trip

Given a `DynamicValue` with a matching structure, convert it back to a strongly-typed value:

```scala

case class Person(name: String, age: Int)

val fromDynamic = Into.derived[DynamicValue, Person]
val dv = DynamicValue.Record(
  "name" -> DynamicValue.string("Bob"),
  "age"  -> DynamicValue.int(25)
)
val result = fromDynamic.into(dv)
```

The conversion completes successfully:

```scala
result
// res35: Either[SchemaError, Person] = Right(Person(name = "Bob", age = 25))
```

Conversion fails gracefully if the structure doesn't match:

```scala

case class Person(name: String, age: Int)

val fromDynamic = Into.derived[DynamicValue, Person]
val badDV = DynamicValue.Primitive(PrimitiveValue.String("not a record"))
val result = fromDynamic.into(badDV)
```

```scala
result
// res36: Either[SchemaError, Person] = Left(
//   SchemaError(
//     List(
//       ExpectationMismatch(
//         source = DynamicOptic(IndexedSeq()),
//         expectation = "Expected a record"
//       )
//     )
//   )
// )
```

### Collections and DynamicValue Round-Trip

Conversions work seamlessly through collections. Here's a complete round-trip:

```scala

case class Item(id: Int, name: String)

val listToDynamic = Into.derived[List[Item], DynamicValue]
val listFromDynamic = Into.derived[DynamicValue, List[Item]]

val items = List(Item(1, "A"), Item(2, "B"))

// Forward: List[Item] → DynamicValue
val asDV = listToDynamic.into(items)

// Round-trip: DynamicValue → List[Item]
val backToList = asDV.flatMap(listFromDynamic.into)
```

The round-trip restores the original data:

```scala
backToList
// res37: Either[SchemaError, List[Item]] = Right(
//   List(Item(id = 1, name = "A"), Item(id = 2, name = "B"))
// )
```

Similarly for maps:

```scala

val mapToDynamic = Into.derived[Map[String, Int], DynamicValue]
val mapFromDynamic = Into.derived[DynamicValue, Map[String, Int]]

val data = Map("count" -> 42, "total" -> 100)

val asDV = mapToDynamic.into(data)
val backToMap = asDV.flatMap(mapFromDynamic.into)
```

```scala
backToMap
// res38: Either[SchemaError, Map[String, Int]] = Right(
//   Map("count" -> 42, "total" -> 100)
// )
```

## Related Type: `As[A, B]`

`As[A, B]` extends `Into[A, B]` with a reverse direction, enabling round-trip safe bidirectional conversions. Because `As` must guarantee that `A → B → A` restores the original value, it applies stricter derivation constraints than `Into`. See [As](./as.md) for the full reference.

## Best Practices

Following a few conventions avoids common pitfalls when working with `Into` and `As`.

**Prefer `As` when round-trip correctness is required.** For data sync or bidirectional serialization, use `As`. For one-way migrations or API responses, use `Into`:

```scala

case class LocalModel(id: Long, name: String)
case class RemoteModel(id: Long, name: String)

case class OldFormat(value: Int)
case class NewFormat(value: Long)

val sync: As[LocalModel, RemoteModel]    = As.derived    // round-trip
val migrate: Into[OldFormat, NewFormat]  = Into.derived  // one-way
```

**Use `Option` for truly optional fields, not default values.** Default values prevent `As.derived` when the field is absent from the other type; `Option` always works:

```scala

// Good — Option works with both Into and As
case class V2Good(name: String, email: Option[String])

// Risky — default value prevents As derivation when field is absent from the other side
case class V2Risky(name: String, email: String = "")
```

**Provide explicit implicits for complex nested types.** When nested types need custom validation logic, define the inner `Into` as an implicit before deriving the outer one:

```scala

case class AddressV1(street: String, zip: Int)
case class AddressV2(street: String, zip: Long, country: String = "US")

case class PersonV1(name: String, address: AddressV1)
case class PersonV2(name: String, address: AddressV2)

implicit val addressMigrate: Into[AddressV1, AddressV2] =
  Into.derived[AddressV1, AddressV2]

val personMigrate: Into[PersonV1, PersonV2] =
  Into.derived[PersonV1, PersonV2]  // picks up addressMigrate automatically
```

## Advanced Usage

The real power of `Into` emerges in multi-version schema evolution scenarios where types gain new fields, change numeric precision, and introduce new coproduct cases simultaneously. The following example migrates a two-level object graph from V1 to V2:

```scala

object V1 {
  case class Address(street: String, city: String)
  case class Person(name: String, age: Int, address: Address)
}

object V2 {
  case class Address(street: String, city: String, country: String = "US")
  case class Person(
    name: String,
    age: Long,                  // widened from Int
    address: Address,
    email: Option[String]       // new optional field
  )
}

implicit val addressMigrate: Into[V1.Address, V2.Address] =
  Into.derived[V1.Address, V2.Address]

val personMigrate: Into[V1.Person, V2.Person] =
  Into.derived[V1.Person, V2.Person]
```

A V1 record converts to V2 in one call — all defaults, widenings, and nested conversions are applied automatically:

```scala
val oldPerson = V1.Person("Alice", 30, V1.Address("123 Main St", "NYC"))
// oldPerson: Person = Person(
//   name = "Alice",
//   age = 30,
//   address = Address(street = "123 Main St", city = "NYC")
// )
personMigrate.into(oldPerson)
// res42: Either[SchemaError, Person] = Right(
//   Person(
//     name = "Alice",
//     age = 30L,
//     address = Address(street = "123 Main St", city = "NYC", country = "US"),
//     email = None
//   )
// )
```

## Scala 2 vs Scala 3 Differences

| Feature | Scala 2 | Scala 3 |
|---------|---------|---------|
| Derivation syntax | `Into.derived[A, B]` | `Into.derived[A, B]` |
| Enum support | Sealed traits only | Scala 3 enums + sealed traits |
| Opaque types | N/A | ✅ Supported |
| Structural types | JVM only (reflection) | JVM only (reflection) |
| ZIO Prelude newtypes | ✅ `assert { between(...) }` | ✅ `override def assertion` |
| Error messages | Detailed macro errors | Detailed macro errors |

---

## SchemaExpr

`SchemaExpr[A, +B]` is a **schema-aware expression** that computes a result of type `B` from an input value of type `A`. The input type `A` must be fully described by a [`Schema`](./schema.md), and the expression is built from [optics](./optics.md), literal values, and operators. The fundamental operations are `eval` and `evalDynamic`.

`SchemaExpr`:
- represents expressions as a reified AST, enabling introspection and serialization
- supports relational (`===`, `>`, `<`, `>=`, `<=`, `!=`), logical (`&&`, `||`, `!`), arithmetic (`+`, `-`, `*`), and string (`concat`, `matches`, `length`) operations
- evaluates to `Either[OpticCheck, Seq[B]]`, handling failures and multi-valued results from traversals
- is covariant in `B`, the output type

```scala
sealed trait SchemaExpr[A, +B] {
  def eval(input: A): Either[OpticCheck, Seq[B]]
  def evalDynamic(input: A): Either[OpticCheck, Seq[DynamicValue]]
}
```

:::tip
For practical walkthroughs of building with `SchemaExpr`, see [Query DSL Part 1: Expressions](../guides/query-dsl-reified-optics.md), [Part 2: SQL Generation](../guides/query-dsl-sql.md), and [Part 3: Extending the Expression Language](../guides/query-dsl-extending.md).
:::

## Motivation

When working with schema-described data, we often need to express computations over that data — comparisons, arithmetic, string operations — in a way that can be both **evaluated at runtime** and **inspected as data**. This is essential for:

1. **Persistence DSLs** — Third-party libraries can translate `SchemaExpr` trees into SQL `WHERE` clauses, NoSQL filters, or other query languages, because the expression structure is reified (not opaque functions).
2. **Validation** — Express constraints like "age must be greater than 18" or "name must match a pattern" as composable, inspectable expressions.
3. **Data Migration** — Define transformation rules that can be analyzed and optimized before execution.

```text
                              SchemaExpr[A, B]
                                     │
          ┌──────────┬───────────────┼───────────────┬──────────────────┐
          │          │               │               │                  │
     Leaf Nodes   Unary Ops     Binary Ops    StringRegexMatch   StringLength
          │          │               │
    ┌─────┴─────┐   Not    ┌────────┼────────┐
  Literal    Optic       Relational Logical  Arithmetic
                                             StringConcat
```

The typical way to build expressions is through the operator syntax on [Optic](./optics.md) values:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

// Build expressions using optic operators
val isAdult: SchemaExpr[Person, Boolean]    = Person.age >= 18
val isAlice: SchemaExpr[Person, Boolean]    = Person.name === "Alice"
val combined: SchemaExpr[Person, Boolean]   = isAdult && isAlice

// Evaluate against a value
val alice = Person("Alice", 30)
val result: Either[OpticCheck, Seq[Boolean]] = combined.eval(alice)
// Right(Seq(true))
```

## Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema" % "0.0.33"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-schema" % "0.0.33"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

`SchemaExpr` instances are typically created through **operator syntax on optics** rather than by constructing AST nodes directly. Each operator on `Optic[S, A]` returns a `SchemaExpr[S, B]`.

### Via Relational Operators on Optics

The comparison operators `===`, `>`, `>=`, `<`, `<=`, and `!=` on `Optic[S, A]` create `SchemaExpr.Relational` nodes. Each operator has two overloads — one comparing against a literal value, and one comparing against another optic:

```scala

case class Product(name: String, price: Double, stock: Int)

object Product extends CompanionOptics[Product] {
  implicit val schema: Schema[Product] = Schema.derived

  val name: Lens[Product, String]  = $(_.name)
  val price: Lens[Product, Double] = $(_.price)
  val stock: Lens[Product, Int]    = $(_.stock)
}

// Compare optic against a literal value
val expensive: SchemaExpr[Product, Boolean] = Product.price > 100.0
val inStock: SchemaExpr[Product, Boolean]   = Product.stock > 0
val named: SchemaExpr[Product, Boolean]     = Product.name === "Widget"

// Compare optic against another optic
// (e.g., stock > price — contrived, but shows the syntax)
```

### Via Logical Operators on Optics

The `&&`, `||`, and `!` (unary) operators on boolean-focused optics create `SchemaExpr.Logical` and `SchemaExpr.Not` nodes:

```scala

case class User(name: String, active: Boolean, verified: Boolean)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived

  val name: Lens[User, String]       = $(_.name)
  val active: Lens[User, Boolean]    = $(_.active)
  val verified: Lens[User, Boolean]  = $(_.verified)
}

// Logical operators on boolean optics
val activeAndVerified: SchemaExpr[User, Boolean] = User.active && User.verified
val eitherOne: SchemaExpr[User, Boolean]         = User.active || User.verified
val notActive: SchemaExpr[User, Boolean]         = !User.active
```

### Via Arithmetic Operators on Optics

The `+`, `-`, and `*` operators on numeric-focused optics create `SchemaExpr.Arithmetic` nodes. These require an implicit `IsNumeric[A]` instance, which is provided for `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`:

```scala

case class Order(quantity: Int, unitPrice: Double)

object Order extends CompanionOptics[Order] {
  implicit val schema: Schema[Order] = Schema.derived

  val quantity : Lens[Order, Int]     = $(_.quantity)
  val unitPrice: Lens[Order, Double]  = $(_.unitPrice)
}

// Arithmetic on optic values
val doubled   : SchemaExpr[Order, Int]    = Order.quantity * 2
val discounted: SchemaExpr[Order, Double] = Order.unitPrice - 5.0
val increased : SchemaExpr[Order, Int]    = Order.quantity + 1
```

### Via String Operators on Optics

The `concat`, `matches`, and `length` methods on string-focused optics create `SchemaExpr.StringConcat`, `SchemaExpr.StringRegexMatch`, and `SchemaExpr.StringLength` nodes:

```scala

case class Email(address: String, subject: String)

object Email extends CompanionOptics[Email] {
  implicit val schema: Schema[Email] = Schema.derived

  val address: Lens[Email, String] = $(_.address)
  val subject: Lens[Email, String] = $(_.subject)
}

// String operations
val withDomain: SchemaExpr[Email, String]   = Email.address.concat("@example.com")
val isValid   : SchemaExpr[Email, Boolean]  = Email.address.matches("^[^@]+@[^@]+$")
val subjectLen: SchemaExpr[Email, Int]      = Email.subject.length
```

### Via Logical Operators on SchemaExpr

Boolean-typed `SchemaExpr` values can be combined with `&&` and `||`:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

// Compose expressions with && and ||
val isAdult = Person.age >= 18
val isAlice = Person.name === "Alice"

val adultAlice  : SchemaExpr[Person, Boolean] = isAdult && isAlice
val adultOrAlice: SchemaExpr[Person, Boolean] = isAdult || isAlice
```

### Via Direct AST Construction

For advanced use cases, we can construct `SchemaExpr` nodes directly:

```scala

case class Item(price: Int)

object Item extends CompanionOptics[Item] {
  implicit val schema: Schema[Item] = Schema.derived

  val price: Lens[Item, Int] = $(_.price)
}

// Construct AST nodes directly
val lit: SchemaExpr[Item, Int] = new SchemaExpr.Literal(42, Schema[Int])
val opticExpr: SchemaExpr[Item, Int] = new SchemaExpr.Optic(Item.price)
val comparison: SchemaExpr[Item, Boolean] = new SchemaExpr.Relational(
  opticExpr,
  lit,
  SchemaExpr.RelationalOperator.GreaterThan
)
```

## Core Operations

### Evaluation

#### `eval`

Evaluates the expression against an input value, returning the typed result. The result is a `Seq[B]` because traversal-based expressions can produce multiple values.

```scala
trait SchemaExpr[A, +B] {
  def eval(input: A): Either[OpticCheck, Seq[B]]
}
```

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val age: Lens[Person, Int] = $(_.age)
}

val isAdult = Person.age >= 18

val alice = Person("Alice", 30)
val result = isAdult.eval(alice)
// Right(List(true))

val bob = Person("Bob", 12)
val result2 = isAdult.eval(bob)
// Right(List(false))
```

:::note
When an expression wraps a `Traversal` optic, `eval` returns multiple values — one per element in the traversed collection. For `Lens`-based expressions, the result is always a single-element `Seq`.
:::

#### `evalDynamic`

Like `eval`, but converts the result to [`DynamicValue`](./dynamic-value.md) instances. This is useful for serialization or when working with schema-agnostic code.

```scala
trait SchemaExpr[A, +B] {
  def evalDynamic(input: A): Either[OpticCheck, Seq[DynamicValue]]
}
```

In the following example we are evaluating a simple optic expression to extract the `name` field from a `Person` and retrieving it as a `DynamicValue`:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
}

val nameExpr = new SchemaExpr.Optic(Person.name)
val result = nameExpr.evalDynamic(Person("Alice", 30))
// Right(List(DynamicValue.Primitive(PrimitiveValue.String("Alice"))))
```

### Logical Combination

#### `&&`

Combines two boolean-typed expressions with logical AND. Both operands must produce `Boolean` results.

```scala
trait SchemaExpr[A, +B] {
  def &&[B2](that: SchemaExpr[A, B2])(implicit ev: B <:< Boolean, ev2: B2 =:= Boolean): SchemaExpr[A, Boolean]
}
```

In the following example we are evaluating a combined expression that checks if a `Person` is an adult (age >= 18) and has the name "Alice". The result of evaluating this expression against a `Person` instance will be `true` if both conditions are met, and `false` otherwise:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

val isAdultAlice = (Person.age >= 18) && (Person.name === "Alice")
val result = isAdultAlice.eval(Person("Alice", 30))
// Right(List(true))
```

#### `||`

Combines two boolean-typed expressions with logical OR.

```scala
trait SchemaExpr[A, +B] {
  def ||[B2](that: SchemaExpr[A, B2])(implicit ev: B <:< Boolean, ev2: B2 =:= Boolean): SchemaExpr[A, Boolean]
}
```

In the following example we are evaluating a combined expression that checks if a `Person` is an adult (age >= 18) or has the name "Alice". The result of evaluating this expression against a `Person` instance will be `true` if either condition is met, and `false` only if both conditions are not met:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}

val isAdultOrAlice = (Person.age >= 18) || (Person.name === "Alice")
val result = isAdultOrAlice.eval(Person("Alice", 12))
// Right(List(true)) — Alice, even though not adult
```

## Subtypes

`SchemaExpr` is a sealed trait with a rich set of case classes representing different expression nodes. These form an **expression AST** that can be inspected, serialized, or translated to other query languages.

### Leaf Nodes

#### `SchemaExpr.Literal`

A constant value that ignores the input and always produces the same result.

```scala
object SchemaExpr {
  case class Literal[S, A](value: A, schema: Schema[A]) extends SchemaExpr[S, A]
}
```

The `Literal#eval` always returns `Right(Seq(value))` regardless of the input. The `Literal#schema` parameter enables conversion to `DynamicValue` via `evalDynamic`.

#### `SchemaExpr.Optic`

Wraps an [`Optic[A, B]`](./optics.md) to extract values from the input. The behavior depends on the optic type:

```scala
object SchemaExpr {
  case class Optic[A, B](optic: zio.blocks.schema.Optic[A, B]) extends SchemaExpr[A, B]
}
```

| Optic Type  | `SchemaExpr.Optic#eval` Behavior                                                      |
|-------------|---------------------------------------------------------------------------------------|
| `Lens`      | Always succeeds with a single value                                                   |
| `Prism`     | Succeeds if the input matches the expected case; otherwise returns `Left(OpticCheck)` |
| `Optional`  | Succeeds if the value is present; otherwise returns `Left(OpticCheck)`                |
| `Traversal` | Returns all elements; returns `Left(OpticCheck)` if the collection is empty           |

### Unary Operations

#### `SchemaExpr.Not`

Negates a boolean expression.

```scala
object SchemaExpr {
  case class Not[A](expr: SchemaExpr[A, Boolean]) extends UnaryOp[A, Boolean](expr)
}
```

Created via the `!` (unary negation) operator on boolean optics:

```scala

case class User(active: Boolean)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived

  val active: Lens[User, Boolean] = $(_.active)
}

val inactive: SchemaExpr[User, Boolean] = !User.active

val result = inactive.eval(User(active = true))
// Right(List(false))
```

### Binary Operations

`Relational`, `Logical`, `Arithmetic`, and `StringConcat` extend `BinaryOp[A, B, C]`, which provides `left` and `right` sub-expressions. `StringRegexMatch` and `StringLength` extend `SchemaExpr` directly — see [Other Operations](#other-operations) below.

#### `SchemaExpr.Relational`

Compares two expressions using a `RelationalOperator`. Returns a boolean result.

```scala
object SchemaExpr {
  case class Relational[A, B](
    left: SchemaExpr[A, B],
    right: SchemaExpr[A, B],
    operator: RelationalOperator
  ) extends SchemaExpr[A, Boolean]
}
```

The available `RelationalOperator` values are:

| Operator             | Optic Syntax | Description           |
|----------------------|--------------|-----------------------|
| `Equal`              | `===`        | Equality check        |
| `NotEqual`           | `!=`         | Inequality check      |
| `LessThan`           | `<`          | Less than             |
| `LessThanOrEqual`    | `<=`         | Less than or equal    |
| `GreaterThan`        | `>`          | Greater than          |
| `GreaterThanOrEqual` | `>=`         | Greater than or equal |

:::note
Equality and inequality (`===`, `!=`) compare values directly. Ordering operators (`<`, `<=`, `>`, `>=`) compare via `DynamicValue` ordering internally.
:::

#### `SchemaExpr.Logical`

Combines two boolean expressions with a `LogicalOperator`.

```scala
object SchemaExpr {
  case class Logical[A](
    left: SchemaExpr[A, Boolean],
    right: SchemaExpr[A, Boolean],
    operator: LogicalOperator
  ) extends SchemaExpr[A, Boolean]
}
```

The available `LogicalOperator` values are:

| Operator | Syntax | Description         |
|----------|--------|---------------------|
| `And`    | `&&`   | Logical conjunction |
| `Or`     | `\|\|` | Logical disjunction |

#### `SchemaExpr.Arithmetic`

Performs arithmetic on two numeric expressions using an `ArithmeticOperator`. Requires an `IsNumeric[A]` type class instance.

```scala
object SchemaExpr {
  case class Arithmetic[S, A](
    left: SchemaExpr[S, A],
    right: SchemaExpr[S, A],
    operator: ArithmeticOperator,
    isNumeric: IsNumeric[A]
  ) extends SchemaExpr[S, A]
}
```

The available `ArithmeticOperator` values are:

| Operator   | Optic Syntax | Description    |
|------------|--------------|----------------|
| `Add`      | `+`          | Addition       |
| `Subtract` | `-`          | Subtraction    |
| `Multiply` | `*`          | Multiplication |

Supported numeric types via `IsNumeric`: `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, `BigDecimal`.

#### `SchemaExpr.StringConcat`

Concatenates two string expressions.

```scala
object SchemaExpr {
  case class StringConcat[A](
    left: SchemaExpr[A, String],
    right: SchemaExpr[A, String]
  ) extends SchemaExpr[A, String]
}
```

Created via the `concat` method on string optics:

```scala

case class Greeting(prefix: String)

object Greeting extends CompanionOptics[Greeting] {
  implicit val schema: Schema[Greeting] = Schema.derived

  val prefix: Lens[Greeting, String] = $(_.prefix)
}

val withName = Greeting.prefix.concat(", World!")
val result = withName.eval(Greeting("Hello"))
// Right(List("Hello, World!"))
```

### Other Operations

`StringRegexMatch` and `StringLength` extend `SchemaExpr` directly rather than through `UnaryOp` or `BinaryOp`.

#### `SchemaExpr.StringRegexMatch`

Tests whether a string matches a regular expression pattern. Despite having two operands (`regex` and `string`), it extends `SchemaExpr[A, Boolean]` directly.

```scala
object SchemaExpr {
  case class StringRegexMatch[A](
    regex: SchemaExpr[A, String],
    string: SchemaExpr[A, String]
  ) extends SchemaExpr[A, Boolean]
}
```

Created via the `matches` method on string optics:

```scala

case class Email(address: String)

object Email extends CompanionOptics[Email] {
  implicit val schema: Schema[Email] = Schema.derived

  val address: Lens[Email, String] = $(_.address)
}

val isValid = Email.address.matches("^[^@]+@[^@]+\\.[^@]+$")
val result = isValid.eval(Email("alice@example.com"))
// Right(List(true))
```

#### `SchemaExpr.StringLength`

Computes the length of a string expression. This is a unary operation but extends `SchemaExpr[A, Int]` directly rather than `UnaryOp`.

```scala
object SchemaExpr {
  case class StringLength[A](
    string: SchemaExpr[A, String]
  ) extends SchemaExpr[A, Int]
}
```

Created via the `length` method on string optics:

```scala

case class Message(body: String)

object Message extends CompanionOptics[Message] {
  implicit val schema: Schema[Message] = Schema.derived

  val body: Lens[Message, String] = $(_.body)
}

val bodyLength = Message.body.length
val result = bodyLength.eval(Message("Hello!"))
// Right(List(6))
```

### Abstract Intermediate Traits

Two sealed traits categorize some expressions by arity:

- **`UnaryOp[A, B]`** — has a single `expr: SchemaExpr[A, B]`. Extended by `Not`.
- **`BinaryOp[A, B, C]`** — has `left: SchemaExpr[A, B]` and `right: SchemaExpr[A, B]`. Extended by `Relational`, `Logical`, `Arithmetic`, and `StringConcat`.

Not all expression nodes use these traits — `StringRegexMatch` and `StringLength` extend `SchemaExpr` directly. These intermediate traits are useful for pattern matching when you need to generically process the expression tree.

## Error Handling

Expression evaluation returns `Either[OpticCheck, Seq[B]]`. The `Left` case contains an [`OpticCheck`](./optics.md) with detailed diagnostic information about what went wrong — for example, an unexpected case in a prism, an empty collection in a traversal, or a missing key.

```scala

case class Shape(kind: String)

object Shape extends CompanionOptics[Shape] {
  implicit val schema: Schema[Shape] = Schema.derived

  val kind: Lens[Shape, String] = $(_.kind)
}

val expr = Shape.kind === "circle"
val result = expr.eval(Shape("circle"))

result match {
  case Right(values) => println(s"Result: ${values.head}")
  case Left(check)   => println(s"Error: ${check.message}")
}
```

## Advanced Usage: Building Query DSLs

Because `SchemaExpr` is a sealed, inspectable AST, third-party libraries can pattern-match on the expression tree to translate it into other languages. For example, a database library could translate `SchemaExpr` into SQL:

```scala
// Pseudocode — illustrates the concept
def toSql[A](expr: SchemaExpr[A, Boolean]): String = expr match {
  case SchemaExpr.Relational(left, right, op) =>
    s"${toSqlValue(left)} ${opToSql(op)} ${toSqlValue(right)}"
  case SchemaExpr.Logical(left, right, SchemaExpr.LogicalOperator.And) =>
    s"(${toSql(left)}) AND (${toSql(right)})"
  case SchemaExpr.Logical(left, right, SchemaExpr.LogicalOperator.Or) =>
    s"(${toSql(left)}) OR (${toSql(right)})"
  case SchemaExpr.Not(inner) =>
    s"NOT (${toSql(inner)})"
  // ...
}
```

This is the key advantage of reified expressions over plain functions — the same expression can be evaluated locally *and* translated to a remote query language.

## Integration

### Optics

`SchemaExpr` is tightly integrated with [Optics](./optics.md). All operator methods (`===`, `>`, `<`, `>=`, `<=`, `!=`, `&&`, `||`, `!`, `+`, `-`, `*`, `concat`, `matches`, `length`) are defined on `Optic[S, A]` and return `SchemaExpr[S, B]` values. This makes the optic the primary entry point for building expressions.

### Schema

[Schema](./schema.md) provides the type information needed by `SchemaExpr.Literal` to convert values to `DynamicValue` via `Literal#evalDynamic`. The `IsNumeric` type class (used by `Arithmetic`) also derives from `Schema`.

### DynamicValue

[DynamicValue](./dynamic-value.md) is the output type of `SchemaExpr#evalDynamic`. It provides a schema-less representation that can be serialized, compared, and manipulated uniformly.

### OpticCheck

[OpticCheck](./optics.md) is the error type returned when expression evaluation fails. It provides rich diagnostic information including the optic path, expected vs. actual cases, and the actual value encountered.

---

## Schema

`Schema[A]` is the primary data type in ZIO Blocks (ZIO Schema 2) that contains reified information about the structure of a Scala data type `A`, together with the ability to tear down and build up values of that type.

```scala
final case class Schema[A](reflect: Reflect.Bound[A])
```

```
┌────────────────────────────────────────────────────────────────┐
│                        Schema[A]                               │
├────────────────────────────────────────────────────────────────┤
│                Reflect.Bound[A] (Reflect[Binding, A])          │
│    ┌─────────────────────────────────────────────────────┐     │
│    │  Structure       │  TypeName[A], DynamicValue       │     │
│    │  (ADT nodes)     │  Doc, Examples, Default Value    │     │
│    │                  │  Modifiers, Metadata             │     │
│    └─────────────────────────────────────────────────────┘     │
│    ┌─────────────────────────────────────────────────────┐     │
│    │  Binding[T, A]                                      │     │
│    │  - Constructor[A] (build values)                    │     │
│    │  - Deconstructor[A] (tear down values)              │     │
│    └─────────────────────────────────────────────────────┘     │
└────────────────────────────────────────────────────────────────┘
```

## Reflect: Structure vs Binding

The [`Reflect`](./reflect.md) data type represents the structure of Scala types. It is parameterized by `F[_, _]` which allows plugging in different "binding" strategies:

```
┌────────────────────────────────────────────────────────────────┐
│                    Reflect[F[_, _], A]                         │
├────────────────────────────────────────────────────────────────┤
│  F = Binding      → Reflect.Bound[A]    (with construction/    │
│                                          deconstruction)       │
│  F = NoBinding    → Reflect.Unbound[A]  (pure data,            │
│                                          serializable)         │
└────────────────────────────────────────────────────────────────┘
```

- **Bound Schema**: Contains functions for constructing/deconstructing values (not serializable)
- **Unbound Schema**: Pure data representation (can be serialized across the wire)

Schema is a bound Reflect, meaning that other than structural information, it also contains construction and deconstruction capabilities via the `Binding` type:

```scala
// Schema wraps a bound Reflect
final case class Schema[A](reflect: Reflect.Bound[A])

// Reflect.Bound is a type alias
type Bound[A] = Reflect[Binding, A]
```

## Schema Derivation

ZIO Blocks provides both automatic schema derivation and also ways to manually create schemas. As an end user, you will mostly use automatic derivation. So we will focus on automatic derivation here. If you need to go deeper and create schemas manually, check out the [Reflect](./reflect.md) and [Binding](./binding.md) documentation pages.

To leverage auto-derivation, simply define an implicit `Schema` for your type using `Schema.derived`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

It will automatically derive the schema for `Person` based on its structure. After that, you can summon the schema using `Schema[Person]` anywhere in your code.

For sealed traits (ADTs), it will derive the schema for all subtypes as well:

```scala

sealed trait Shape
object Shape {
  case class Circle(radius: Double) extends Shape
  case class Rectangle(width: Double, height: Double) extends Shape
  
  implicit val schema: Schema[Shape] = Schema.derived
}
```

Under the hood, the derivation process includes deriving schemas for cases of enum (i.e. `Circle` and `Rectangle`) and finally creating a schema for the `Shape` trait itself.

## Built-in Schemas

ZIO Blocks ships with a comprehensive set of pre-defined schemas for standard Scala and Java types, eliminating boilerplate and ensuring consistent behavior across your application.

### Primitive Types

The foundational building blocks for all data types. These schemas leverage the [register-based architecture](registers.md) for zero-allocation performance:

```scala

Schema[Unit]      // The unit type (singleton value)
Schema[Boolean]   // Boolean values (true/false)
Schema[Byte]      // 8-bit signed integer (-128 to 127)
Schema[Short]     // 16-bit signed integer
Schema[Int]       // 32-bit signed integer
Schema[Long]      // 64-bit signed integer
Schema[Float]     // 32-bit IEEE 754 floating point
Schema[Double]    // 64-bit IEEE 754 floating point
Schema[Char]      // 16-bit Unicode character
Schema[String]    // Immutable character sequence
```

### Arbitrary Precision Numbers

For financial calculations and scenarios requiring exact decimal representation:

```scala

Schema[BigInt]      // Arbitrary precision integer
Schema[BigDecimal]  // Arbitrary precision decimal
```

### Temporal Types (java.time)

Complete coverage of the modern Java Time API for robust date/time handling:

```scala

// Date components
Schema[java.time.LocalDate]      // Date without time (2024-01-15)
Schema[java.time.LocalTime]      // Time without date (14:30:00)
Schema[java.time.LocalDateTime]  // Date and time without timezone

// Timezone-aware types  
Schema[java.time.ZonedDateTime]   // Full date-time with timezone
Schema[java.time.OffsetDateTime]  // Date-time with UTC offset
Schema[java.time.OffsetTime]      // Time with UTC offset
Schema[java.time.ZoneId]          // Timezone identifier (e.g., "America/New_York")
Schema[java.time.ZoneOffset]      // Fixed UTC offset (e.g., +05:30)

// Duration and period
Schema[java.time.Duration]  // Time-based duration (hours, minutes, seconds)
Schema[java.time.Period]    // Date-based period (years, months, days)
Schema[java.time.Instant]   // Point on the timeline (Unix timestamp)

// Calendar components
Schema[java.time.Year]       // Year value (2024)
Schema[java.time.Month]      // Month of year (JANUARY..DECEMBER)
Schema[java.time.DayOfWeek]  // Day of week (MONDAY..SUNDAY)
Schema[java.time.YearMonth]  // Year and month combination
Schema[java.time.MonthDay]   // Month and day combination
```

### Common Java Utility Types

There are also schemas for frequently used Java utility types, UUID and Currency:

```scala

Schema[java.util.UUID]      // 128-bit universally unique identifier
Schema[java.util.Currency]  // ISO 4217 currency code
```

### Optional Values

ZIO Blocks provides specialized `Option` schemas optimized for primitive types. These avoid boxing overhead by storing primitive values directly in registers:

```scala

// Specialized primitive options (no boxing overhead)
Schema[Option[Boolean]]  // Also: Byte, Short, Int, Long, Float, Double, Char, Unit
```

Other than primitive types, ZIO Blocks uses a generic representation for `Option[A]` which works for all reference types:

```scala

// Reference type options (requires A <: AnyRef)
Schema[Option[A]]  // Generic option for reference types
```

### Collection Types

ZIO Blocks also provides polymorphic schemas for standard Scala collections. You can summon schemas for collections of any element type `A` (and key/value types `K`/`V` for maps):

```scala

// Built-in Scala collections
Schema[List[A]]        // Immutable singly-linked list
Schema[Vector[A]]      // Immutable indexed sequence (efficient random access)
Schema[Set[A]]         // Immutable set (unique elements)
Schema[Seq[A]]         // General immutable sequence
Schema[IndexedSeq[A]]  // Indexed sequence
Schema[Map[K, V]]      // Immutable key-value mapping

// ZIO Blocks collections (the chunk module)
Schema[zio.blocks.chunk.Chunk[A]]       // Chunk sequence
Schema[zio.blocks.chunk.ChunkMap[K, V]] // ChunkMap key-value mapping
```

To learn how to create custom collection schemas, check out the [`Sequence`](./reflect.md#3-sequence) and [`Map`](./reflect.md#4-map) nodes on the documentation of [`Reflect`](./reflect.md) data type.

### DynamicValue

ZIO Blocks includes a built-in schema for `DynamicValue`, a semi-structured data representation that serves as a superset of JSON:

```scala

val schema = Schema[DynamicValue]  // Semi-structured data (superset of JSON)
```

Having the schema for `DynamicValue` allows seamless encoding/decoding between `DynamicValue` and other formats, such as JSON, Avro, Protobuf, etc. It enables us to convert our type-safe data into a semi-structured representation and serialize it into any desired format.

### DynamicValue toString (EJSON Format)

`DynamicValue` has a custom `toString` that produces EJSON (Extended JSON) format - a superset of JSON that handles non-string keys, tagged variants, and typed primitives:

```scala

// Records have unquoted keys
val record = DynamicValue.Record(Vector(
  "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
  "age" -> DynamicValue.Primitive(PrimitiveValue.Int(30))
))
println(record)
// {
//   name: "Alice",
//   age: 30
// }

// Maps have quoted string keys
val map = DynamicValue.Map(Vector(
  DynamicValue.Primitive(PrimitiveValue.String("key")) ->
    DynamicValue.Primitive(PrimitiveValue.String("value"))
))
println(map)
// {
//   "key": "value"
// }

// Variants use @ metadata
val variant = DynamicValue.Variant("Some", DynamicValue.Record(Vector(
  "value" -> DynamicValue.Primitive(PrimitiveValue.Int(42))
)))
println(variant)
// {
//   value: 42
// } @ {tag: "Some"}

// Typed primitives use @ metadata
val instant = DynamicValue.Primitive(PrimitiveValue.Instant(java.time.Instant.now))
println(instant)
// 1705312800000 @ {type: "instant"}
```

**Key EJSON properties:**
- **Records**: unquoted keys (`{ name: "John" }`)
- **Maps**: quoted string keys (`{ "name": "John" }`) or unquoted non-string keys (`{ 42: "value" }`)
- **Variants**: postfix `@ {tag: "CaseName"}`
- **Typed primitives**: postfix `@ {type: "instant"}` for types that would lose precision as JSON

## Debug-Friendly toString

`Schema` has a custom `toString` that wraps the underlying `Reflect` output in a `Schema { ... }` block, providing a complete structural view of your data types:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

println(Schema[Person])
// Output:
// Schema {
//   record Person {
//     name: String
//     age: Int
//   }
// }
```

For primitive schemas:

```scala
println(Schema[Int])
// Output:
// Schema {
//   Int
// }
```

This format makes it easy to inspect complex nested schemas during debugging. See the [Reflect documentation](./reflect.md#debug-friendly-tostring) for details on the SDL format used for the inner structure.

## Encoding and Decoding

The `Schema[A]` provides methods to encode and decode values of type `A` to/from various formats using the `Format` abstraction:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
  def encode[F <: codec.Format](format: F)(output: format.EncodeOutput)(value: A): Unit = ???
  def decode[F <: codec.Format](format: F)(decodeInput: format.DecodeInput): Either[SchemaError, A] = ???
}
```

The `Format` is the base abstraction for serialization formats in ZIO Blocks, such as Avro, JSON, Protobuf, etc. Each format associates with a specific codec type class which defines the interface for encoding and decoding values and a deriver for deriving codecs for specific types.

ZIO Blocks has built-in support for several popular formats, currently `AvroFormat` and `JsonFormat`. You can also implement your own custom formats by extending the `Format` trait.

The following example demonstrates encoding and decoding a `Person` case class to/from JSON using the built-in `JsonFormat`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

object EncodeDecodeExample extends App {
  val person = Person("John", 42)

  // Encode to JSON
  val encodedBuffer = {
    val buffer = ByteBuffer.allocate(1024)
    Schema[Person].encode(JsonFormat)(buffer)(person)
    buffer.flip()
    buffer
  }

  // Extract JSON string
  val jsonString = new String(encodedBuffer.duplicate().array())
  println(s"Encoded JSON: $jsonString")

  // Decode back to Person
  val decodedPerson = Schema[Person].decode(JsonFormat)(encodedBuffer.duplicate())
  println(s"Decoded Person: $decodedPerson")
}
```

Please note that both `Schema#encode` and `Schema#decode` cache instances, so using encode and decode in multiple places only performs the derivation process once.

## Type Class Derivation

To derive type class instances for a type `A` based on its schema, you can use the `derive` method on `Schema[A]`. This method takes a `Deriver` for the desired type class and produces an instance of that type class for `A`.

In the following example, we derive a JSON codec for the `Person` case class using the `JsonFormat` deriver:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
  val codec: JsonCodec[Person] = schema.derive(JsonFormat)
}

val person = Person("John", 42)

// Encode to JSON
val json = Person.codec.encode(person)

// Extract JSON string
val jsonString = new String(json)
println(s"Encoded JSON: $jsonString")

// Decode back to Person
val decodedPerson = Person.codec.decode(json)
println(s"Decoded Person: $decodedPerson")
```

## Metadata Operations

ZIO Blocks allows attaching metadata to schemas and their fields, such as documentation, example values, and default values. This metadata can be useful for generating API documentation, client code, or providing hints to serialization formats.

Here is an example of how to set and retrieve documentation values:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person]{
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String] = optic(_.name)
  val age: Lens[Person, Int]     = optic(_.age)
}

// Add documentation to the schema
val documented: Schema[Person] = Schema[Person].doc("A person entity")

// Get documentation
val doc: Doc = Schema[Person].doc

// Add documentation to a field using optics
val fieldDoc: Schema[Person] = Schema[Person].doc(Person.name, "The person's name")
```

The important thing to note here is that we can use optics to target specific fields within the schema when adding or retrieving metadata. In the last example, we added documentation specifically to the `name` field of the `Person` schema.

Similarly, you can set and get example values:

```scala

// Add example values
val withExamples: Schema[Person] = 
  Schema[Person].examples(Person("Alice", 30), Person("Bob", 25))

// Get examples
val examples: Seq[Person] = Schema[Person].examples

// Add examples to a specific field
val fieldExamples: Schema[Person] =
  Schema[Person].examples(Person.name, "Alice", "Bob", "Charlie")
```

There are also methods for setting and getting default values:

```scala
// Add default value to schema
val withDefault: Schema[Person] = Schema[Person]
  .defaultValue(Person("Unknown", 0))

// Get default value
val default: Option[Person] = Schema[Person].getDefaultValue

// Add default to a specific field
val fieldDefault: Schema[Person] = Schema[Person]
  .defaultValue(Person.age, 18)

// Get default for a field
val ageDefault: Option[Int] = 
  Schema[Person].getDefaultValue(Person.age)
```

## Accessing and Updating Partial Schemas

To access a specific part of a schema, we can use the `Schema#get` method, which takes an optic and returns the reflection of the targeted field. 

```scala

case class Person(name: String, address: Address)

object Person extends CompanionOptics[Person]{
  implicit val schema: Schema[Person] = Schema.derived

  val name: Lens[Person, String]      = optic(_.name)
  val address: Lens[Person, Address]  = optic(_.address)
}

case class Address(street: String, city: String)

object Address extends CompanionOptics[Address]{
  implicit val schema: Schema[Address] = Schema.derived

  val street: Lens[Address, String] = optic(_.street)
  val city  : Lens[Address, String] = optic(_.city)
}

// Get schema for a nested field
val addressSchema: Option[Reflect.Bound[Address]] =
  Schema[Person].get(Person.address)
```

This method enables us to retrieve the schema for any nested field using optics. For example, to get the schema for the `street` field inside the `address` of a `Person`, we can do as follows:

```scala
val streetSchema: Option[Reflect.Bound[String]] =
  Schema[Person].get(Person.address(Address.street))
```

### Updating Nested Schemas

To update a specific part of a schema, we can use the `Schema#updated` method, which takes an optic and a function to transform the targeted schema:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
  def updated[B](optic: Optic[A, B])(
    f: Reflect.Bound[B] => Reflect.Bound[B]
  ): Option[Schema[A]]
}
```

Here is an example of updating the documentation of a nested field:

```scala
// Update schema at a specific path
val updated: Option[Schema[Person]] = Schema[Person]
  .updated(Person.address)(_.doc("Mailing address"))
```

## Schema Aspects

Schema aspects are a powerful mechanism in ZIO Blocks for transforming schemas. You can think of the schema aspect as a function that takes a reflect and produces a new reflect:

```scala
trait SchemaAspect[-Upper, +Lower, F[_, _]] {
  def apply[A >: Lower <: Upper](reflect: Reflect[F, A]): Reflect[F, A]
  def recursive(implicit ev1: Any <:< Upper, ev2: Lower <:< Nothing): SchemaAspect[Upper, Lower, F]
}
```

The `Schema` data type has a `@@` method used for applying schema aspects:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
  def @@[Min >: A, Max <: A](aspect: SchemaAspect[Min, Max, Binding]): Schema[A] = ???
  def @@[B](part: Optic[A, B], aspect: SchemaAspect[B, B, Binding]) = ???
}
```

These methods enable us to use `@@` syntax for applying aspects to either the entire schema or a specific path within the schema using optics:

```scala

// Apply aspect to entire schema
val documented: Schema[Person] = Schema[Person] @@ SchemaAspect.doc("A person")

// Apply aspect to specific path
val fieldDoc: Schema[Person] = Schema[Person] @@ (
  Person.name, 
  SchemaAspect.examples("Alice", "Bob")
)
```

Currently, ZIO Blocks provides the following built-in schema aspects:

- `SchemaAspect.identity`: No-op transformation
- `SchemaAspect.doc`: Attach documentation to schema or field
- `SchemaAspect.examples`: Attach example values to schema or field

## Modifiers

Modifiers in ZIO Blocks provide a mechanism to attach metadata and configuration to schema elements without polluting the domain types themselves. They serve as the successor to ZIO Schema 1's annotation system, with the critical advantage of being **pure data** so, unlike Scala annotations, modifiers are runtime values that can be serialized.

The `Schema.modifier` and `Schema.modifiers` methods allow adding one or more modifiers to a schema:

```scala
final case class Schema[A](reflect: Reflect.Bound[A]) {
  def modifier(modifier: Modifier.Reflect)            : Schema[A] = ???
  def modifiers(modifiers: Iterable[Modifier.Reflect]): Schema[A] = ???
}
```

Here is an example of adding modifiers to a schema:

```scala
// Add modifier to schema
  val modified: Schema[Person] = Schema[Person]
    .modifier(Modifier.config("db.table-name", "person_table"))
    .modifier(Modifier.config("schema.version", "v2"))

// Add multiple modifiers
  val multiModified: Schema[Person] = Schema[Person]
    .modifiers(
      Seq(
        Modifier.config("db.table-name", "person_table"),
        Modifier.config("schema.version", "v2")
      )
    )
```

## Wrapper Types

ZIO Blocks provides the `transform` method for creating schemas for wrapper types, such as newtypes, opaque types and value classes:

```scala
final case class Schema[A](reflect: Reflect.Bound[A]) {
  def transform[B](to: A => B, from: B => A)(implicit typeId: TypeId[B]): Schema[B] = ???
}
```

The `transform` method allows you to define transformations that can fail by throwing `SchemaError` exceptions. Use it for both simple wrapper types and types with validation requirements. 

Here are examples of both:

```scala

// For types with validation (may fail)
case class Email(value: String)

object Email {
  private val EmailRegex = "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$".r
  
  implicit val schema: Schema[Email] = Schema[String]
    .transform(
      {
        case x @ EmailRegex(_*) => Email(x)
        case _                  => throw SchemaError.validationFailed("Invalid email format")
      },
      _.value
    )
}

// For total transformations (never fail)
case class UserId(value: Long)

object UserId {
  implicit val schema: Schema[UserId] = Schema[Long]
    .transform(UserId(_), _.value)
}
```

## Compile-Time Shape Constraints (`Allows`)

ZIO Blocks provides `Allows[A, S]` — a phantom-typed capability token that proves, at compile time, that type `A` satisfies the structural grammar `S`. This lets library authors express and enforce structural preconditions on their generic APIs without writing macros themselves.

```scala

// Require a flat record of scalars (e.g. for CSV or RDBMS)
def writeCsv[A: Schema](rows: Seq[A])(using
  Allows[A, Record[Primitive | Optional[Primitive]]]
): Unit = ???

// Sealed traits auto-unwrap: each case must satisfy Record[...] — no Variant node needed
def publish[A: Schema](event: A)(using
  Allows[A, Record[Primitive | Sequence[Primitive]]]
): Unit = ???

// Recursive grammar (e.g. for a JSON document store)
def toJson[A: Schema](doc: A)(using
  Allows[A, Record[Primitive | Self | Optional[Primitive | Self] | Sequence[Primitive | Self]]]
): String = ???
```

When a type does not satisfy the grammar, the user gets a precise compile-time error naming the violating field and suggesting a fix. No runtime surprises.

See the [`Allows` reference](./allows.md) for the full grammar node table, union syntax, `Self` for recursive types, newtypes, and error message examples.

---

## Streams

`zio.blocks.streams` is a **synchronous, pull-based** streaming library for **Scala 3** (and Scala 2.13) with typed errors, resource safety, and primitive specialization. Streams are lazy descriptions -- nothing executes until a terminal operation is called. All results are returned as `Either[E, Z]`, keeping error handling explicit and typed. The library has zero runtime dependencies beyond `zio.blocks.chunk` and `zio.blocks.scope`, and achieves zero-boxing on primitive element types (`Int`, `Long`, `Float`, `Double`) through JVM-type-specialized internal readers.

## Why Streams?

Streaming libraries in the Scala ecosystem typically require an effect system. fs2 needs `cats.effect.IO`, Kyo Streams needs the Kyo runtime, and Pekko (formerly Akka) Streams needs the actor runtime. When your code is synchronous and you want streaming without pulling in an effect monad, the options narrow considerably.

`zio.blocks.streams` fills that gap:

| Feature | ZB Streams | fs2 | Kyo | Ox | Pekko |
|---|---|---|---|---|---|
| Effect system required | No | Yes (cats-effect) | Yes (Kyo) | No (virtual threads) | Yes (Akka) |
| Execution model | Synchronous, pull-based | Async, pull-based | Async, chunk-based | Synchronous, pull-based | Async, push-based |
| Typed errors | `Either[E, Z]` | ApplicativeError | Kyo effects | Exceptions | No |
| Primitive specialization | Yes (zero boxing) | No | No | No | No |
| Internal chunking | No (element-at-a-time) | Yes (Chunk) | Yes (Chunk) | No | No |
| Stack-safe deep pipelines | Yes (trampolined) | Yes (Pull) | Yes | No (SO on deep flatMap) | N/A |
| Resource safety | Scope integration | Resource/bracket | Kyo resources | try/finally | Graph lifecycle |
| Dependencies | chunk + scope | cats-effect + scodec | Kyo core | Ox core | Akka actor |

Key properties:

- **No effect system** -- streams run on the calling thread; `run` returns `Either[E, Z]` directly
- **Pull-based** -- the consumer drives evaluation; elements are produced on demand. Contrast with push-based systems (like Pekko) where the producer drives and the consumer must keep up.
- **Primitive specialization** -- `Int`, `Long`, `Float`, and `Double` streams avoid boxing through specialized `readInt`, `readLong`, `readFloat`, `readDouble` methods on `Reader`
- **Resource safe** -- integrates with `zio.blocks.scope.Scope` for deterministic finalization; also provides `fromAcquireRelease`, `ensuring`, and `defer` for standalone resource management
- **Lazy** -- a `Stream[E, A]` is a description; construction is free and nothing executes until a terminal operation (`run`, `runCollect`, `head`, etc.)
- **Debuggable** -- streams render their pipeline structure via `toString`/`render`, so you can inspect what a stream does without running it

---

## Quick start

```scala

// Create a stream, transform it, consume it
val result: Either[Nothing, Chunk[Int]] =
  Stream.range(1, 11)        // 1 to 10
    .filter(_ % 2 == 0)      // keep evens
    .map(_ * 10)              // multiply by 10
    .runCollect               // collect into a Chunk

// result: Right(Chunk(20, 40, 60, 80, 100))
```

Key points:

- `Stream.range(1, 11)` creates a lazy stream of integers 1 through 10 (exclusive upper bound)
- `.filter` and `.map` add transformations without executing anything
- `.runCollect` is the terminal operation -- it drives evaluation and returns `Either[E, Chunk[A]]`
- Since `Stream.range` cannot fail, the error type is `Nothing` and the result is always `Right`

A stream that can fail:

```scala
val fallible: Either[String, Chunk[Int]] =
  Stream.range(1, 6)
    .flatMap { n =>
      if n == 3 then Stream.fail("boom at 3")
      else Stream.succeed(n)
    }
    .runCollect

// fallible: Left("boom at 3")
```

---

## Benchmarks

All benchmarks use 10,000 elements, measured in operations per second (higher is better). Run on Apple M-series, JDK 25, Scala 3.7.4.

| Benchmark | ZB Streams | Ox | Kyo | fs2 | Pekko |
|---|---|---|---|---|---|
| drain | 179,872 | 54,512 | 31,777 | 20,795 | 4,381 |
| map | 161,920 | 42,007 | 12,012 | 13,295 | 2,259 |
| filter | 168,541 | 47,933 | 19,962 | 14,977 | 2,901 |
| flatMap | 49,165 | 30,506 | 28,303 | 748 | 742 |
| take/drop | 322,470 | 28,708 | 64,640 | 28,836 | 2,379 |
| map+filter+flatMap | 980 | 508 | 602 | 19 | 16 |
| mixed depth 1 | 47,459 | 19,449 | 13,427 | 257 | 639 |
| mixed depth 2 | 33,859 | 15,336 | 7,328 | 208 | 459 |
| mixed depth 3 | 23,610 | 11,878 | 3,174 | 139 | 256 |
| nested flatMap (10K) | 8,161 | -- | -- | 937 | -- |
| nested concat (10K) | 6,140 | -- | 3 | 1,065 | 1 |

"--" indicates the benchmark was not run or the library crashed.

ZB Streams leads in every single-operator benchmark and maintains its advantage as pipeline depth increases. The "mixed depth" rows show cascading `map`/`filter`/`flatMap` stages -- ZB Streams degrades gracefully thanks to its trampolined execution model, while libraries without stack-safety (Ox) or with high per-element overhead (fs2, Pekko) fall off sharply.

---

## Core mental model

### 1) `Stream[E, A]` -- a lazy sequence

A `Stream[+E, +A]` is a **description** of a potentially infinite sequence of elements of type `A` that may fail with an error of type `E`. It is covariant in both type parameters.

Nothing happens when you construct a stream or chain transformations. Execution only begins when you call a terminal operation (`run`, `runCollect`, `runDrain`, `head`, `count`, etc.). Terminal operations return `Either[E, Z]`:

- `Left(e)` -- a typed stream error
- `Right(z)` -- the successful result

Untyped defects (unexpected exceptions) propagate as thrown exceptions, not as `Left` values.

```scala
// This does nothing -- it's just a description
val description: Stream[Nothing, Int] =
  Stream.range(0, 1_000_000)
    .filter(_ % 7 == 0)
    .map(_ * 2)
    .take(100)

// Only this line executes the pipeline
val result: Either[Nothing, Chunk[Int]] = description.runCollect
```

Streams render their pipeline structure as a human-readable string:

```scala
val s = Stream.range(0, 100).map(_ + 1).filter(_ > 50).take(10)
println(s)  // Stream.range(0, 100).map(...).filter(...).take(10)
```

This makes debugging and logging straightforward -- you can see exactly what transformations a stream applies without running it.

---

### 2) `Sink[E, A, Z]` -- a consumer

A `Sink[+E, -A, +Z]` consumes elements of type `A` from a stream and produces a final result of type `Z`. Sinks are passed to `Stream.run`:

```scala
val stream = Stream.range(1, 101)

// Built-in sinks
val total: Either[Nothing, Long]        = stream.run(Sink.count)
val items: Either[Nothing, Chunk[Int]]  = stream.run(Sink.collectAll)
val sum:   Either[Nothing, Long]        = stream.run(Sink.sumInt)
val first: Either[Nothing, Option[Int]] = stream.run(Sink.head)
```

Most sinks also have convenience methods directly on `Stream`:

```scala
stream.count       // Either[Nothing, Long]
stream.runCollect  // Either[Nothing, Chunk[Int]]
stream.head        // Either[Nothing, Option[Int]]
stream.last        // Either[Nothing, Option[Int]]
```

Sinks compose with `contramap` (pre-process input) and `map` (post-process result):

```scala
val lengthSink: Sink[Nothing, String, Long] =
  Sink.sumInt.contramap[String](_.length)

val doubled: Sink[Nothing, Int, Long] =
  Sink.sumInt.map(_ * 2)
```

Built-in sinks:

| Sink | Result type | Description |
|---|---|---|
| `Sink.collectAll` | `Chunk[A]` | Collects all elements |
| `Sink.drain` | `Unit` | Consumes and discards all elements |
| `Sink.count` | `Long` | Counts elements |
| `Sink.foldLeft(z)(f)` | `Z` | Left fold with initial value |
| `Sink.foreach(f)` | `Unit` | Side-effect per element |
| `Sink.head` | `Option[A]` | First element |
| `Sink.last` | `Option[A]` | Last element |
| `Sink.take(n)` | `Chunk[A]` | First n elements |
| `Sink.exists(p)` | `Boolean` | Short-circuiting existential |
| `Sink.forall(p)` | `Boolean` | Short-circuiting universal |
| `Sink.find(p)` | `Option[A]` | First matching element |
| `Sink.sumInt` | `Long` | Sum of Ints (zero-boxing) |
| `Sink.sumLong` | `Long` | Sum of Longs (zero-boxing) |
| `Sink.sumFloat` | `Double` | Sum of Floats (zero-boxing) |
| `Sink.sumDouble` | `Double` | Sum of Doubles (zero-boxing) |
| `Sink.fromOutputStream(os)` | `Unit` | Writes bytes to an `OutputStream` |
| `Sink.fromJavaWriter(w)` | `Unit` | Writes chars to a `java.io.Writer` |
| `Sink.fail(e)` | `Nothing` | Fails immediately |
| `Sink.create(f)` | `Z` | Custom sink from `Reader[A] => Z` |

---

### 3) `Pipeline[In, Out]` -- reusable transformation

A `Pipeline[-In, +Out]` is a reusable stream transformation. It decouples the transformation logic from any specific stream, so you can define it once and apply it many times.

```scala
// Define a reusable pipeline
val normalize: Pipeline[Int, Double] =
  Pipeline.filter[Int](_ > 0)
    .andThen(Pipeline.map[Int, Double](_.toDouble / 100.0))

// Apply to different streams
val result1 = Stream.range(-10, 10).via(normalize).runCollect
val result2 = Stream(42, -5, 100, 0).via(normalize).runCollect
```

Pipelines compose with `andThen`:

```scala
val step1: Pipeline[String, Int] =
  Pipeline.map[String, Int](_.length)

val step2: Pipeline[Int, Int] =
  Pipeline.filter[Int](_ > 3)

val combined: Pipeline[String, Int] =
  step1.andThen(step2)
```

You can also apply a pipeline to a sink with `andThenSink` / `applyToSink`, which pre-processes the sink's input:

```scala
val countLong: Sink[Nothing, String, Long] =
  Pipeline.map[String, Int](_.length)
    .andThenSink(Sink.sumInt)
```

Built-in pipeline factories:

| Factory | Description |
|---|---|
| `Pipeline.map(f)` | Transform each element |
| `Pipeline.filter(p)` | Keep elements matching predicate |
| `Pipeline.collect(pf)` | Partial function -- filter + map |
| `Pipeline.take(n)` | Keep first n elements |
| `Pipeline.drop(n)` | Skip first n elements |
| `Pipeline.identity` | Pass-through (useful as a base for composition) |

---

### 4) `Reader[A]` -- low-level pull source

`Reader[+Elem]` is the low-level, pull-based source that backs every stream. Most users will never interact with `Reader` directly; it is the compilation target when a stream runs.

The protocol is simple:

- `read(sentinel)` -- returns the next element, or `sentinel` when exhausted
- `close()` -- signal the consumer is done
- `isClosed` -- check whether the reader has been closed

For primitive types, specialized methods avoid boxing:

- `readInt(sentinel: Long): Long`
- `readLong(sentinel: Long): Long`
- `readFloat(sentinel: Double): Double`
- `readDouble(sentinel: Double): Double`

You interact with `Reader` in two situations:

1. **Custom sources** -- create a stream from a `Reader` via `Stream.fromReader`
2. **Manual pull** -- open a stream for element-by-element control via `stream.start`

```scala

Scope.global.scoped { scope =>

  // Open a stream for manual pulling
  val reader: $[Reader[Int]] = Stream.range(1, 6).start(using scope)

  $(reader) { r =>
    var v = r.read(-1)
    while v != -1 do
      println(v)   // prints 1, 2, 3, 4, 5
      v = r.read(-1)
  }
  // reader is closed automatically when scope exits
}
```

---

### 5) Error handling

Streams distinguish between two kinds of failures:

- **Typed errors** (`E`) -- domain errors you expect and handle. These appear as `Left` in the `Either` result.
- **Defects** (`Throwable`) -- unexpected exceptions. These propagate as thrown exceptions, bypassing the `Either` channel.

```scala
// Create a failing stream
val failing: Stream[String, Int] =
  Stream(1, 2, 3) ++ Stream.fail("oops") ++ Stream(4, 5)

// catchAll: recover from typed errors
val recovered: Stream[Nothing, Int] =
  failing.catchAll(_ => Stream(99))

recovered.runCollect  // Right(Chunk(1, 2, 3, 99))
```

Error handling operators:

| Operator | Description |
|---|---|
| `catchAll(f: E => Stream[E2, A])` | Recover from all typed errors |
| `catchDefect(pf: PartialFunction[Throwable, Stream[E1, A]])` | Recover from matching defects |
| `mapError(f: E => E2)` | Transform the error type |
| `orElse(that)` / `\|\|(that)` | Fall back to another stream on error |

---

### 6) Resource safety

Streams integrate with `zio.blocks.scope.Scope` for deterministic finalization. Several constructors guarantee that acquired resources are released when the stream closes, whether it completes normally, short-circuits, or fails.

#### `fromAcquireRelease`

The primary resource-safe constructor. Acquires a resource, uses it to produce a stream, and guarantees the release function runs on close:

```scala

val lines: Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    acquire = new BufferedReader(new FileReader("data.txt")),
    release = _.close()
  ) { reader =>
    Stream.unfold(()) { _ =>
      Option(reader.readLine()).map(line => (line, ()))
    }
  }

// The BufferedReader is closed when the stream finishes,
// even if the consumer takes only a few lines
lines.take(5).runCollect
```

If the resource is `AutoCloseable`, the release function defaults to calling `close()`:

```scala
val lines: Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    acquire = new BufferedReader(new FileReader("data.txt"))
  ) { reader =>
    Stream.unfold(()) { _ =>
      Option(reader.readLine()).map(line => (line, ()))
    }
  }
```

#### `fromResource`

Integrates with `zio.blocks.scope.Resource` directly:

```scala

val resource: Resource[BufferedReader] =
  Resource.fromAutoCloseable(new BufferedReader(new FileReader("data.txt")))

val lines: Stream[Nothing, String] =
  Stream.fromResource(resource) { reader =>
    Stream.unfold(()) { _ =>
      Option(reader.readLine()).map(line => (line, ()))
    }
  }
```

#### `ensuring` and `defer`

For attaching finalizers to existing streams:

```scala
// ensuring: run a finalizer when the stream closes
val withCleanup: Stream[Nothing, Int] =
  Stream.range(1, 11).ensuring(println("stream closed"))

// defer: register a release action (runs on close, not on construction)
val withDefer: Stream[Nothing, Int] =
  Stream.defer(println("cleanup")) ++ Stream.range(1, 6)
```

#### `start` with `Scope`

For manual pull-based consumption with scope-managed lifetime:

```scala

Scope.global.scoped { scope =>

  val reader = Stream.range(1, 100).start(using scope)
  // reader is automatically closed when scope exits
}
```

---

## Usage examples

### Creating streams

```scala

// From explicit elements
Stream(1, 2, 3)                              // Stream[Nothing, Int]
Stream("a", "b", "c")                        // Stream[Nothing, String]

// From collections
Stream.fromChunk(Chunk(1, 2, 3))             // Stream[Nothing, Int]
Stream.fromIterable(List("x", "y", "z"))     // Stream[Nothing, String]
Stream.fromIterator(Iterator.from(1))        // Stream[Nothing, Int] (lazy)

// Ranges
Stream.range(0, 100)                         // 0 to 99
Stream.fromRange(1 to 50)                    // 1 to 50

// Single values (primitive-specialized)
Stream.succeed(42)                           // Stream[Nothing, Int]
Stream.succeed(3.14)                         // Stream[Nothing, Double]
Stream.succeed("hello")                      // Stream[Nothing, String]

// Special streams
Stream.empty                                 // Stream[Nothing, Nothing]
Stream.fail("error")                         // Stream[String, Nothing]
Stream.die(new Exception("defect"))          // throws on evaluation

// Generators
Stream.repeat(1)                             // infinite stream of 1s
Stream.iterate(1)(_ * 2)                     // 1, 2, 4, 8, 16, ...
Stream.repeatThunk(scala.util.Random.nextInt(100))  // infinite random ints
Stream.unfold(0)(n =>                        // 0, 1, 2, ..., 9
  if n < 10 then Some((n, n + 1)) else None
)

// Side-effects
Stream.eval(println("hello"))               // prints, emits nothing
Stream.attempt(someFallibleCall())           // captures exceptions as typed errors
Stream.attemptEval(riskyEffect())            // same, for Unit-returning effects

// Deferred construction (useful for recursion)
Stream.suspend(expensiveStreamBuilder())

// I/O sources (auto-closing)
Stream.fromInputStream(inputStream)          // Stream[IOException, Int] (bytes as 0-255, auto-closes)
Stream.fromJavaReader(javaReader)            // Stream[IOException, Char] (auto-closes)

// I/O sources (borrowing -- caller manages lifetime)
Stream.fromInputStreamUnmanaged(inputStream) // Stream[IOException, Int] (does NOT close)
Stream.fromJavaReaderUnmanaged(javaReader)   // Stream[IOException, Char] (does NOT close)
```

---

### Transforming streams

```scala
val s = Stream.range(1, 21) // 1 to 20

// map: transform each element
s.map(_ * 2)                                 // 2, 4, 6, ..., 40

// filter: keep matching elements
s.filter(_ % 3 == 0)                         // 3, 6, 9, 12, 15, 18

// flatMap: expand each element into a sub-stream
s.flatMap(n => Stream(n, n * 10))            // 1, 10, 2, 20, 3, 30, ...

// collect: partial function (filter + map)
s.collect { case n if n % 2 == 0 => n / 2 } // 1, 2, 3, ..., 10

// take / drop / takeWhile
s.take(5)                                    // 1, 2, 3, 4, 5
s.drop(15)                                   // 16, 17, 18, 19, 20
s.takeWhile(_ < 8)                           // 1, 2, 3, 4, 5, 6, 7

// scan: running accumulator (emits initial value + one value per element)
Stream(1, 2, 3, 4).scan(0)(_ + _)           // 0, 1, 3, 6, 10

// mapAccum: stateful transformation
s.mapAccum(0) { (acc, n) =>
  val newAcc = acc + n
  (newAcc, newAcc)
}
// running sum: 1, 3, 6, 10, 15, ...

// grouped: collect into fixed-size chunks
Stream.range(1, 11).grouped(3)
// Chunk(1,2,3), Chunk(4,5,6), Chunk(7,8,9), Chunk(10)

// sliding: overlapping windows
Stream.range(1, 7).sliding(3, 1)
// Chunk(1,2,3), Chunk(2,3,4), Chunk(3,4,5), Chunk(4,5,6)

// intersperse: insert separator between elements
Stream("a", "b", "c").intersperse(",")       // "a", ",", "b", ",", "c"

// distinct / distinctBy: deduplication
Stream(1, 2, 2, 3, 1, 3).distinct            // 1, 2, 3
Stream("ab", "cd", "ae").distinctBy(_.head)   // "ab", "cd"

// zipWithIndex: pair elements with their 0-based index
Stream("a", "b", "c").zipWithIndex
// ("a", 0L), ("b", 1L), ("c", 2L)

// tapEach: side-effect without changing elements
s.tapEach(n => println(s"processing $n"))

// concat: sequence two streams
Stream(1, 2) ++ Stream(3, 4)                // 1, 2, 3, 4

// repeated: restart on completion
Stream(1, 2, 3).repeated.take(8)             // 1, 2, 3, 1, 2, 3, 1, 2

// via: apply a Pipeline
s.via(Pipeline.filter[Int](_ > 10))          // 11, 12, ..., 20
```

---

### Zipping streams with `&&`

The `&&` operator zips two streams element-by-element into tuples. The resulting stream ends when either input is exhausted.

```scala
val names: Stream[Nothing, String] = Stream("Alice", "Bob", "Charlie")
val ages:  Stream[Nothing, Int]    = Stream(30, 25, 35)
val ids:   Stream[Nothing, Long]   = Stream(1L, 2L, 3L)

// Two-way zip
val pairs: Stream[Nothing, (String, Int)] = names && ages
pairs.runCollect // Right(Chunk(("Alice", 30), ("Bob", 25), ("Charlie", 35)))

// Three-way zip -- tuples flatten automatically
val triples: Stream[Nothing, (String, Int, Long)] = names && ages && ids
triples.runCollect // Right(Chunk(("Alice", 30, 1L), ("Bob", 25, 2L), ("Charlie", 35, 3L)))
```

When the error types differ, they widen via union:

```scala
val s1: Stream[String, Int]  = Stream(1, 2, 3)
val s2: Stream[IOException, Int] = Stream(4, 5, 6)
val zipped: Stream[String | IOException, (Int, Int)] = s1 && s2
```

---

### Primitive specialization

ZB Streams eliminates boxing for `Int`, `Long`, `Float`, and `Double` elements throughout the entire pipeline. Every intermediate step uses specialized `readInt`/`writeInt` (or the corresponding type) methods, so no `java.lang.Integer` wrappers are allocated.

```scala
// This entire pipeline runs with ZERO boxing of the Int elements.
// Every step uses specialized readInt/writeInt internally.
val sum: Either[Nothing, Long] =
  Stream.range(0, 1_000_000)   // Int-specialized source
    .filter(_ % 2 == 0)        // Int-specialized filter
    .map(_ * 3)                 // Int->Int specialized map
    .runFold(0L)(_ + _)         // Long-specialized accumulator

// Compare: in fs2 or ZIO Streams, every Int would be boxed to java.lang.Integer
// at each pipeline stage boundary.
```

This matters most for numeric workloads -- data processing, statistics, encoding/decoding -- where millions of elements flow through multi-stage pipelines. The benchmark results above reflect this advantage directly.

---

### Consuming streams

```scala
val s = Stream.range(1, 11) // 1 to 10

// Collect all elements
s.runCollect              // Right(Chunk(1, 2, 3, ..., 10))

// Discard all elements (run for side-effects only)
s.tapEach(println).runDrain

// Fold
s.runFold(0)(_ + _)       // Right(55)              (Int accumulator)
s.runFold(0L)(_ + _)      // Right(55L)             (Long accumulator)
s.runFold(0.0)(_ + _)     // Right(55.0)            (Double accumulator)

// Foreach
s.runForeach(n => println(n))
s.foreach(n => println(n))  // alias

// Aggregates
s.count                   // Right(10L)
s.head                    // Right(Some(1))
s.last                    // Right(Some(10))
s.exists(_ > 5)           // Right(true)
s.forall(_ > 0)           // Right(true)
s.find(_ > 7)             // Right(Some(8))

// Run with an explicit Sink
s.run(Sink.sumInt)        // Right(55L)
s.run(Sink.take(3))       // Right(Chunk(1, 2, 3))
```

---

### Error handling patterns

```scala
// Typed error: appears in Either
val result = Stream.fail("not found").runCollect
// result: Left("not found")

// Recover and continue
val safe =
  Stream(1, 2) ++ Stream.fail("oops") ++ Stream(3)
val recovered = safe.catchAll(_ => Stream(99)).runCollect
// Right(Chunk(1, 2, 99))

// Transform error type
val mapped =
  Stream.fail("bad input")
    .mapError(msg => new IllegalArgumentException(msg))
// Stream[IllegalArgumentException, Nothing]

// Fallback stream
val primary: Stream[String, Int] = Stream.fail("down")
val backup:  Stream[String, Int] = Stream(1, 2, 3)
val result2 = (primary || backup).runCollect
// Right(Chunk(1, 2, 3))

// Catch defects (unexpected exceptions)
val risky: Stream[Nothing, Int] =
  Stream(1, 2, 3).map { n =>
    if n == 2 then throw new ArithmeticException("boom")
    else n
  }

val handled = risky.catchDefect {
  case _: ArithmeticException => Stream(0)
}.runCollect
// Right(Chunk(1, 0))
```

---

### Resource safety patterns

```scala

// Bracket pattern: acquire/use/release
def fileLines(path: String): Stream[Nothing, String] =
  Stream.fromAcquireRelease(
    acquire = scala.io.Source.fromFile(path),
    release = _.close()
  ) { source =>
    Stream.fromIterable(source.getLines().toList)
  }

// Compose resource-safe streams -- both resources are released
val merged =
  fileLines("input1.txt") ++ fileLines("input2.txt")

// Only reads 10 lines; both files are still closed properly
merged.take(10).runCollect

// ensuring: attach a finalizer
var cleaned = false
Stream.range(1, 6)
  .ensuring { cleaned = true }
  .take(2)
  .runDrain
// cleaned == true, even though only 2 of 5 elements were consumed

// defer: register cleanup that runs on stream close
val withDefer =
  Stream.defer(println("releasing lock")) ++
  Stream.range(1, 100)
```

---

### NIO integration (JVM only)

On the JVM, `NioStreams` and `NioSinks` provide zero-copy integration with `java.nio` buffers and channels.

#### `NioStreams` -- creating streams from NIO sources

```scala

// From a ByteBuffer
val buf = ByteBuffer.wrap(Array[Byte](1, 2, 3, 4, 5))
NioStreams.fromByteBuffer(buf).runCollect
// Right(Chunk(1, 2, 3, 4, 5))

// Typed buffer views (zero-boxing)
val intBuf = ByteBuffer.allocate(16).putInt(1).putInt(2).putInt(3).putInt(4).flip()
NioStreams.fromByteBufferInt(intBuf).runCollect
// Right(Chunk(1, 2, 3, 4))

// Similarly: fromByteBufferLong, fromByteBufferFloat, fromByteBufferDouble

// From a ReadableByteChannel (auto-closing)
val ch = FileChannel.open(Paths.get("data.bin"), StandardOpenOption.READ)
val bytes = NioStreams.fromChannel(ch, bufSize = 4096).runCollect
// ch is closed automatically when the stream completes

// From a ReadableByteChannel (borrowing -- caller manages lifetime)
val ch2 = FileChannel.open(Paths.get("data.bin"), StandardOpenOption.READ)
val bytes2 = NioStreams.fromChannelUnmanaged(ch2, bufSize = 4096).runCollect
ch2.close() // caller is responsible for closing
```

#### `NioSinks` -- writing to NIO targets

```scala

// Write to a ByteBuffer
val outBuf = ByteBuffer.allocate(1024)
Stream.fromInputStream(inputStream).run(NioSinks.fromByteBuffer(outBuf))

// Typed buffer sinks (zero-boxing)
Stream.range(1, 5).run(NioSinks.fromByteBufferInt(outBuf))
// Also: fromByteBufferLong, fromByteBufferFloat, fromByteBufferDouble

// Write to a WritableByteChannel (buffered)
val outCh = FileChannel.open(
  Paths.get("output.bin"),
  StandardOpenOption.WRITE, StandardOpenOption.CREATE
)
Stream.fromInputStream(inputStream).run(NioSinks.fromChannel(outCh))
outCh.close()
```

---

### Pipeline composition

```scala

// Build reusable transformation steps
val parseInts: Pipeline[String, Int] =
  Pipeline.collect[String, Int] {
    case s if s.matches("-?\\d+") => s.toInt
  }

val positiveOnly: Pipeline[Int, Int] =
  Pipeline.filter[Int](_ > 0)

val doubled: Pipeline[Int, Int] =
  Pipeline.map[Int, Int](_ * 2)

// Compose into a single pipeline
val fullPipeline: Pipeline[String, Int] =
  parseInts
    .andThen(positiveOnly)
    .andThen(doubled)

// Apply to any stream of strings
Stream("10", "abc", "-3", "7", "0", "25")
  .via(fullPipeline)
  .runCollect
// Right(Chunk(20, 14, 50))

// Apply to a sink (pre-process the sink's input)
val sumPositiveDoubled: Sink[Nothing, String, Long] =
  fullPipeline.andThenSink(Sink.sumInt)

Stream("10", "abc", "-3", "7", "0", "25")
  .run(sumPositiveDoubled)
// Right(84L)
```

---

## API reference

### `Stream[+E, +A]`

#### Constructors

```scala
object Stream:
  def apply[A](as: A*): Stream[Nothing, A]
  val empty: Stream[Nothing, Nothing]
  def succeed[A](a: A): Stream[Nothing, A]              // also specialized for primitives
  def fail[E](error: E): Stream[E, Nothing]
  def die(t: Throwable): Stream[Nothing, Nothing]

  def fromChunk[A](chunk: Chunk[A]): Stream[Nothing, A]
  def fromIterable[A](it: Iterable[A]): Stream[Nothing, A]
  def fromIterator[A](it: => Iterator[A]): Stream[Nothing, A]
  def range(from: Int, until: Int): Stream[Nothing, Int]
  def fromRange(range: Range): Stream[Nothing, Int]

  def repeat[A](a: A): Stream[Nothing, A]               // infinite
  def iterate[A](init: A)(f: A => A): Stream[Nothing, A] // infinite: init, f(init), f(f(init)), ...
  def repeatThunk[A](thunk: => A): Stream[Nothing, A]    // infinite: thunk() per element
  def unfold[S, A](s: S)(f: S => Option[(A, S)]): Stream[Nothing, A]

  def eval(f: => Any): Stream[Nothing, Nothing]          // side-effect, no output
  def attempt[A](f: => A): Stream[Throwable, A]          // captures exceptions
  def attemptEval(f: => Any): Stream[Throwable, Nothing]
  def suspend[E, A](stream: => Stream[E, A]): Stream[E, A]
  def defer(f: => Unit): Stream[Nothing, Nothing]        // register finalizer

  def fromInputStream(is: => InputStream): Stream[IOException, Int]           // auto-closes
  def fromInputStreamUnmanaged(is: InputStream): Stream[IOException, Int]     // borrowing
  def fromJavaReader(r: => java.io.Reader): Stream[IOException, Char]         // auto-closes
  def fromJavaReaderUnmanaged(r: java.io.Reader): Stream[IOException, Char]   // borrowing
  def fromReader[E, A](mkReader: => Reader[A]): Stream[E, A]

  def fromAcquireRelease[R, E, A](acquire: => R, release: R => Unit)(use: R => Stream[E, A]): Stream[E, A]
  def fromResource[R, E, A](resource: Resource[R])(use: R => Stream[E, A]): Stream[E, A]
  def flattenAll[E, A](streams: Stream[E, Stream[E, A]]): Stream[E, A]
```

#### Transformations (return `Stream`)

```scala
abstract class Stream[+E, +A]:
  def map[B](f: A => B): Stream[E, B]
  def flatMap[E2, B](f: A => Stream[E2, B]): Stream[E | E2, B]
  def filter(pred: A => Boolean): Stream[E, A]
  def collect[B](pf: PartialFunction[A, B]): Stream[E, B]
  def scan[S](init: S)(f: (S, A) => S): Stream[E, S]
  def mapAccum[S, B](init: S)(f: (S, A) => (S, B)): Stream[E, B]
  def tapEach(f: A => Unit): Stream[E, A]

  def take(n: Long): Stream[E, A]
  def drop(n: Long): Stream[E, A]
  def takeWhile(pred: A => Boolean): Stream[E, A]

  def grouped(n: Int): Stream[E, Chunk[A]]
  def sliding(size: Int, step: Int): Stream[E, Chunk[A]]
  def intersperse[A1 >: A](sep: A1): Stream[E, A1]
  def distinct: Stream[E, A]
  def distinctBy[B](f: A => B): Stream[E, A]
  def zipWithIndex: Stream[E, (A, Long)]

  def concat[E2, A2](that: Stream[E2, A2]): Stream[E | E2, A | A2]  // alias: ++
  def &&[E2, B](that: Stream[E2, B]): Stream[E | E2, (A, B)]        // zip with tuple flattening
  def repeated: Stream[E, A]

  def catchAll[E2, A1](f: E => Stream[E2, A1]): Stream[E2, A | A1]
  def catchDefect[E1, A1](f: PartialFunction[Throwable, Stream[E1, A1]]): Stream[E | E1, A | A1]
  def mapError[E2](f: E => E2): Stream[E2, A]
  def orElse[E2, A1](that: => Stream[E2, A1]): Stream[E2, A | A1]   // alias: ||

  def ensuring(finalizer: => Unit): Stream[E, A]
  def via[B](pipe: Pipeline[A, B]): Stream[E, B]

  def render: String                                      // human-readable pipeline description
  override def toString: String                           // alias for render
```

#### Terminal operations (return `Either[E, Z]`)

```scala
abstract class Stream[+E, +A]:
  def run[E2 >: E, Z](sink: Sink[E2, A, Z]): Either[E2, Z]
  def runCollect: Either[E, Chunk[A]]
  def runDrain: Either[E, Unit]
  def runFold[Z](z: Z)(f: (Z, A) => Z): Either[E, Z]    // also specialized for Int, Long, Double
  def runForeach(f: A => Unit): Either[E, Unit]
  def foreach(f: A => Unit): Either[E, Unit]              // alias

  def head: Either[E, Option[A]]
  def last: Either[E, Option[A]]
  def count: Either[E, Long]
  def exists(pred: A => Boolean): Either[E, Boolean]
  def forall(pred: A => Boolean): Either[E, Boolean]
  def find(pred: A => Boolean): Either[E, Option[A]]

  def start(using scope: Scope): scope.$[Reader[A]]      // manual pull
```

---

### `Sink[+E, -A, +Z]`

```scala
abstract class Sink[+E, -A, +Z]:
  def contramap[A2](g: A2 => A): Sink[E, A2, Z]
  def map[Z2](f: Z => Z2): Sink[E, A, Z2]
  def mapError[E2](f: E => E2): Sink[E2, A, Z]

object Sink:
  def collectAll[A]: Sink[Nothing, A, Chunk[A]]
  val drain: Sink[Nothing, Any, Unit]
  val count: Sink[Nothing, Any, Long]
  def foldLeft[A, Z](z: Z)(f: (Z, A) => Z): Sink[Nothing, A, Z]
  def foreach[A](f: A => Unit): Sink[Nothing, A, Unit]
  def head[A]: Sink[Nothing, A, Option[A]]
  def last[A]: Sink[Nothing, A, Option[A]]
  def take[A](n: Int): Sink[Nothing, A, Chunk[A]]
  def exists[A](pred: A => Boolean): Sink[Nothing, A, Boolean]
  def forall[A](pred: A => Boolean): Sink[Nothing, A, Boolean]
  def find[A](pred: A => Boolean): Sink[Nothing, A, Option[A]]
  val sumInt: Sink[Nothing, Int, Long]
  val sumLong: Sink[Nothing, Long, Long]
  val sumFloat: Sink[Nothing, Float, Double]
  val sumDouble: Sink[Nothing, Double, Double]
  def fromOutputStream(os: OutputStream): Sink[Nothing, Byte, Unit]
  def fromJavaWriter(w: java.io.Writer): Sink[Nothing, Char, Unit]
  def fail[E](e: E): Sink[E, Any, Nothing]
  def create[E, A, Z](f: Reader[A] => Z): Sink[E, A, Z]
```

---

### `Pipeline[-In, +Out]`

```scala
abstract class Pipeline[-In, +Out]:
  def andThen[C](that: Pipeline[Out, C]): Pipeline[In, C]
  def andThenSink[E, Z](sink: Sink[E, Out, Z]): Sink[E, In, Z]
  def applyToStream[E](stream: Stream[E, In]): Stream[E, Out]
  def applyToSink[E, Z](sink: Sink[E, Out, Z]): Sink[E, In, Z]

object Pipeline:
  def map[A, B](f: A => B): Pipeline[A, B]
  def filter[A](pred: A => Boolean): Pipeline[A, A]
  def collect[A, B](pf: PartialFunction[A, B]): Pipeline[A, B]
  def take[A](n: Long): Pipeline[A, A]
  def drop[A](n: Long): Pipeline[A, A]
  def identity[A]: Pipeline[A, A]
```

---

### `NioStreams` (JVM only)

```scala
object NioStreams:
  def fromByteBuffer(buf: ByteBuffer): Stream[Nothing, Byte]
  def fromByteBufferInt(buf: ByteBuffer): Stream[Nothing, Int]
  def fromByteBufferLong(buf: ByteBuffer): Stream[Nothing, Long]
  def fromByteBufferFloat(buf: ByteBuffer): Stream[Nothing, Float]
  def fromByteBufferDouble(buf: ByteBuffer): Stream[Nothing, Double]
  def fromChannel(ch: => ReadableByteChannel, bufSize: Int = 8192): Stream[IOException, Byte]          // auto-closes
  def fromChannelUnmanaged(ch: ReadableByteChannel, bufSize: Int = 8192): Stream[IOException, Byte]    // borrowing
```

### `NioSinks` (JVM only)

```scala
object NioSinks:
  def fromByteBuffer(buf: ByteBuffer): Sink[Nothing, Byte, Unit]
  def fromByteBufferInt(buf: ByteBuffer): Sink[Nothing, Int, Unit]
  def fromByteBufferLong(buf: ByteBuffer): Sink[Nothing, Long, Unit]
  def fromByteBufferFloat(buf: ByteBuffer): Sink[Nothing, Float, Unit]
  def fromByteBufferDouble(buf: ByteBuffer): Sink[Nothing, Double, Unit]
  def fromChannel(ch: WritableByteChannel, bufSize: Int = 8192): Sink[Nothing, Byte, Unit]
```

---

## Practical guidance

- **Start with `Stream` constructors and terminal operations.** You can get very far with `Stream.range`, `Stream.fromIterable`, `.map`, `.filter`, and `.runCollect`.
- **Use `Either` pattern matching** to handle the result: `Right(value)` for success, `Left(error)` for typed failures.
- **Prefer `Stream.fromAcquireRelease`** when wrapping resources (files, connections, etc.) over manual try/finally. It guarantees cleanup even on early termination via `take`, `head`, or error.
- **Use the auto-closing I/O constructors** (`fromInputStream`, `fromJavaReader`, `NioStreams.fromChannel`) by default. Only use the `Unmanaged` variants when you need to borrow a resource whose lifetime is managed elsewhere.
- **Use `Pipeline`** when you have a transformation you want to reuse across multiple streams or apply to sinks. Pipelines have two type parameters: `Pipeline[In, Out]`.
- **Use `&&` for zipping** instead of manual `zip` calls. Tuples flatten automatically for three or more streams: `a && b && c` produces `(A, B, C)` not `((A, B), C)`.
- **Leverage primitive specialization** for numeric workloads. Streams of `Int`, `Long`, `Float`, and `Double` avoid boxing automatically; use `Sink.sumInt`, `Sink.sumLong`, `runFold(0)(_ + _)`, etc. for zero-allocation folds.
- **Use `scan` for running accumulators**, `grouped` for batching, and `sliding` for windowed computations.
- **Use `render`/`toString`** to inspect pipeline structure during debugging -- it shows each transformation stage without executing the stream.
- **Use `Sink.create`** as an escape hatch when none of the built-in sinks fit. It gives you direct access to the `Reader` for custom consumption logic.
- **Use `NioStreams` / `NioSinks`** on the JVM for efficient NIO buffer and channel integration.
- **Avoid holding references** to a `Reader` obtained via `start` outside its `Scope`. The scope guarantees cleanup; escaping the reader defeats that guarantee.
- **`suspend`** is your friend for recursive or self-referential stream definitions, preventing stack overflow during construction.
- **Typed errors vs. defects**: use `Stream.fail` for expected domain errors and `Stream.die` for programmer errors. Use `catchAll` for the former, `catchDefect` for the latter.

---

## Structural Types

<!--
BLOCKING ISSUE: Scala 3.7.4 Compiler Crash
============================================

This file uses structural type syntax ({ def field: Type }) which triggers a compiler
crash in Scala 3.7.4 during the erasure phase. Therefore, code blocks here are marked
as plain `scala` instead of `mdoc:compile-only` to allow the documentation build to pass.

WORKAROUND: Code examples are NOT compiled/type-checked until Scala 3.8.x is adopted.

TODO (When Scala 3.8.x is adopted):
1. Change all `\`\`\`scala` blocks back to `\`\`\`scala mdoc:compile-only`
2. Run `sbt docs/mdoc` to verify compilation succeeds
3. Delete this comment

Reference:
- Scala 3.7.4 compiler issue: https://github.com/scala/scala3/issues/24598 (or similar)
- Scala 3.8.x migration: PR #1169 "Preparing to migration on Scala 3.8.x"
- This PR: Addresses structural-types.md CI failures
-->

Structural types enable **duck typing** with ZIO Blocks schemas. Instead of requiring a nominal type name (like `class Person`), a structural schema validates based on the **shape** of an object — the fields it provides, regardless of how it was defined.

## Motivation

Consider a common integration scenario:

```scala
// Your system
case class Person(name: String, age: Int)

// External system (same data, different class)
case class User(name: String, age: Int)
```

Without structural types, converting between `Person` and `User` requires manual translation. With structural types, they both have the same structural schema:

```scala

case class Person(name: String, age: Int)
case class User(name: String, age: Int)

val personSchema = Schema.derived[Person]
val personStructural = personSchema.structural
// Schema[{ def name: String; def age: Int }]

val userSchema = Schema.derived[User]
val userStructural = userSchema.structural
// Schema[{ def name: String; def age: Int }]

// Both schemas accept the same data shape
```

## Construction: `Schema#structural`

Use the `Schema#structural` method on any schema to get the corresponding structural schema.

**Scala 3:** Using transparent inline — the return type is inferred to the full refinement type:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val personSchema: Schema[Person] = Schema.derived[Person]
val structuralSchema: Schema[{ def name: String; def age: Int }] = personSchema.structural
```

**Scala 2:** Implicit derivation — returns `Schema[ts.StructuralType]` (path-dependent type):

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val personSchema: Schema[Person] = Schema.derived[Person]
val structuralSchema = personSchema.structural
// Type: Schema[ts.StructuralType] (structural type inferred from macro)
```

## Supported Conversions

The following type categories can be converted to structural schemas:

### Product Types (Case Classes)

Both Scala 2 and 3 support structural conversion of case classes:

```scala

case class Address(street: String, city: String, zipCode: Int)
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}

val schema = Schema.derived[Address]
val structural = schema.structural
// Schema[{ def street: String; def city: String; def zipCode: Int }]
```

### Tuples

Tuples convert to structural records with field names derived from positions:

```scala

type StringIntBool = (String, Int, Boolean)
implicit val schema: Schema[StringIntBool] = Schema.derived[StringIntBool]

val tupleSchema = Schema.derived[(String, Int, Boolean)]
val structuralSchema = tupleSchema.structural
// Schema[{ def _1: String; def _2: Int; def _3: Boolean }]
```

### Nested Products

Nested product fields keep their nominal types; only the outer product is structuralized:

```scala

case class Address(street: String, city: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived[Address]
}

case class Person(name: String, age: Int, address: Address)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val personSchema = Schema.derived[Person]
val structuralSchema = personSchema.structural
// Schema[{
//   def name: String
//   def age: Int
//   def address: Address
// }]
```

### Opaque Types (Scala 3)

Opaque type aliases are unwrapped to their underlying type:

```scala

opaque type UserId = String

case class User(id: UserId, name: String)
object User {
  implicit val schema: Schema[User] = Schema.derived[User]
}

val schema = Schema.derived[User]
val structural = schema.structural
// Schema[{ def id: String; def name: String }]
// (UserId unwrapped to String)
```

### Sum Types / Sealed Traits (Scala 3)

Sealed traits and enums convert to union types with nested method syntax:

```scala

sealed trait Shape
object Shape {
  case class Circle(radius: Double) extends Shape
  case class Rectangle(width: Double, height: Double) extends Shape
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
}

val schema = Schema.derived[Shape]
val structural = schema.structural
// Schema[
//   { def Circle: { def radius: Double } } |
//   { def Rectangle: { def height: Double; def width: Double } }
// ]
// (cases sorted alphabetically)
```

**Enum syntax** (Scala 3):

```scala

enum Color {
  case Red, Green, Blue
}
object Color {
  implicit val schema: Schema[Color] = Schema.derived[Color]
}

val schema = Schema.derived[Color]
val structural = schema.structural
// Schema[
//   { def Blue: {} } |
//   { def Green: {} } |
//   { def Red: {} }
// ]
```

Cases appear in **alphabetical order** in the union type. This alphabetical ordering (applied to fields in products and case names in unions) ensures **deterministic, normalized type identity**: two structural types with the same fields but different declaration order produce the same structural type and normalized name. This is essential for predictable schema evolution and cross-system interop.

## Direct Structural Derivation (Scala 3)

Create a schema directly for a structural type without a nominal base:

```scala

// No case class needed — define the schema for the shape directly
val personStructural = Schema.derived[{ def name: String; def age: Int }]

// The schema is ready to use with values matching that structural shape
```

This is only supported in **Scala 3** with the right macro machinery.

## Round-tripping Through DynamicValue

Structural schemas enable **cross-type conversion through `DynamicValue`** — encode a value of one nominal type and decode it as a *different* nominal type with the same structural shape. This is the core benefit of structural types for system integration.

### Motivation

In real integrations, you often receive data from an external system shaped like one type, but you need to work with it as a different type in your system. Without structural types, field-by-field translation is required. With structural types, if both types have identical shape, `DynamicValue` acts as the seamless bridge.

Common scenarios:
- **API gateways** — receive a `PersonDTO` from an external API, decode as your internal `Person` type
- **Message brokers** — consume an event shaped like `UserEvent`, convert to your domain `Account` type
- **Data pipelines** — records with identical fields but different class names from different services

### Cross-type conversion in action

Set up two types with identical structural shape:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

case class Employee(name: String, age: Int)
object Employee {
  implicit val schema: Schema[Employee] = Schema.derived[Employee]
}

val personSchema = Schema.derived[Person]
val employeeSchema = Schema.derived[Employee]
```

Now encode a `Person` to `DynamicValue` and decode it as an `Employee`:

```scala
val person = Person("Alice", 30)
// person: Person = Person(name = "Alice", age = 30)
val dynamic = personSchema.toDynamicValue(person)
// dynamic: DynamicValue = Record(
//   IndexedSeq(("name", Primitive(String("Alice"))), ("age", Primitive(Int(30))))
// )

val employee: Either[SchemaError, Employee] =
  employeeSchema.fromDynamicValue(dynamic)
// employee: Either[SchemaError, Employee] = Right(
//   Employee(name = "Alice", age = 30)
// )
```

The structural shape guarantee ensures type-safe conversion: at compile time, you know both schemas accept the same fields, so round-tripping through `DynamicValue` is safe and zero-cost.

## Integration

Structural types integrate seamlessly with ZIO Blocks' broader ecosystem:

### With Schema Evolution Macros

Structural schemas work with [Schema Evolution](./schema-evolution/into.md) macros for cross-type conversion. When two types share the same structural shape, the conversion machinery can work across type boundaries:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

case class PersonDTO(name: String, age: Int)
object PersonDTO {
  implicit val schema: Schema[PersonDTO] = Schema.derived[PersonDTO]
}

// Both types have identical structural schemas
val personSchema = Schema.derived[Person]
val dtoSchema = Schema.derived[PersonDTO]

// They share the same structural shape:
// Schema[{ def name: String; def age: Int }]
```

### With Binding.of (Serialization)

Structural types are also supported by the `Binding.of` macro for high-performance serialization via register-based encoding:

```scala

// Direct structural type serialization (JVM only)
val binding = Binding.of[{ def name: String; def age: Int }]

// Works with nested structural types
val nestedBinding = Binding.of[{
  def name: String
  def address: { def street: String; def city: String }
}]

// Works with containers
val containerBinding = Binding.of[{
  def name: String
  def emails: List[String]
}]
```

This enables anonymous structural types to benefit from ZIO Blocks' high-performance serialization without requiring nominal case class definitions. Like `Schema#structural`, this is **JVM-only**.

See [Binding](./binding.md) for detailed serialization documentation.

## Running the Examples

Example applications demonstrating structural types are available in `schema-examples`:

```sh
# Simple product type
sbt "schema-examples/runMain structural.StructuralSimpleProductExample"

# Nested products
sbt "schema-examples/runMain structural.StructuralNestedProductExample"

# Sealed trait (Scala 3)
sbt "schema-examples/runMain structural.StructuralSealedTraitExample"

# Enum (Scala 3)
sbt "schema-examples/runMain structural.StructuralEnumExample"

# Tuples
sbt "schema-examples/runMain structural.StructuralTupleExample"

# Integration with Into macro
sbt "schema-examples/runMain structural.StructuralIntoExample"
```

---

## Extension Syntax

ZIO Blocks provides convenient extension methods on any value that has a `Schema`. These methods give you fluent, type-safe access to JSON encoding/decoding, pretty-printing, and patching operations directly on your values.

## Import

To use the extension syntax, import the schema package:

```scala

```

This brings the extension methods into scope for any type with an implicit `Schema` instance.

## Quick Example

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val alice = Person("Alice", 30)

// Convert to JSON
val json = alice.toJson              // Json AST
val jsonStr = alice.toJsonString     // {"name":"Alice","age":30}
val jsonBytes = alice.toJsonBytes    // Array[Byte]

// Parse from JSON
val parsed = """{"name":"Bob","age":25}""".fromJson[Person]
// Right(Person("Bob", 25))

// Pretty-print
val shown = alice.show               // Record { name = Alice, age = 30 }

// Compute and apply patches
val bob = Person("Bob", 30)
val patch = alice.diff(bob)          // Patch that changes name
val result = alice.applyPatch(patch) // Person("Bob", 30)
```

## JSON Encoding Methods

### toJson

Converts a value to a `Json` AST (abstract syntax tree):

```scala

case class Point(x: Int, y: Int)
object Point {
  implicit val schema: Schema[Point] = Schema.derived
}

val point = Point(10, 20)
val json = point.toJson
// Json.Object(Vector("x" -> Json.Number(10), "y" -> Json.Number(20)))

// Navigate and extract values
json.get("x").as[Int]  // Right(10)
json.get("y").as[Int]  // Right(20)
```

### toJsonString

Converts a value directly to a JSON string:

```scala

case class User(name: String, email: String)
object User {
  implicit val schema: Schema[User] = Schema.derived
}

val user = User("Alice", "alice@example.com")
val jsonStr = user.toJsonString
// {"name":"Alice","email":"alice@example.com"}
```

### toJsonBytes

Converts a value to a UTF-8 encoded byte array. This is useful for efficient serialization when working with binary protocols or network I/O:

```scala

case class Message(id: Long, content: String)
object Message {
  implicit val schema: Schema[Message] = Schema.derived
}

val msg = Message(42, "Hello, world!")
val bytes: Array[Byte] = msg.toJsonBytes

// Useful for sending over the wire
// socket.write(bytes)
```

## JSON Decoding Methods

### fromJson (on String)

Parses a JSON string into a typed value:

```scala

case class Config(host: String, port: Int)
object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

val jsonStr = """{"host":"localhost","port":8080}"""
val result: Either[SchemaError, Config] = jsonStr.fromJson[Config]
// Right(Config("localhost", 8080))

// Handle parsing errors
val invalid = """{"host":"localhost"}"""  // missing port
val error = invalid.fromJson[Config]
// Left(SchemaError(...))
```

### fromJson (on Array[Byte])

Parses a UTF-8 byte array into a typed value:

```scala

case class Event(name: String, timestamp: Long)
object Event {
  implicit val schema: Schema[Event] = Schema.derived
}

val jsonBytes = """{"name":"click","timestamp":1234567890}"""
  .getBytes(StandardCharsets.UTF_8)

val result: Either[SchemaError, Event] = jsonBytes.fromJson[Event]
// Right(Event("click", 1234567890))
```

## Pretty-Printing

### show

Converts a value to a human-readable string representation using `DynamicValue`:

```scala

case class Address(street: String, city: String, zip: String)
object Address {
  implicit val schema: Schema[Address] = Schema.derived
}

val addr = Address("123 Main St", "Springfield", "12345")
val shown = addr.show
// Record { street = 123 Main St, city = Springfield, zip = 12345 }
```

This is useful for debugging and logging, as it provides a consistent, schema-aware representation of any value.

## Patching Operations

ZIO Blocks includes a powerful patching system for computing and applying differences between values.

### diff

Computes the difference between two values, returning a `Patch`:

```scala

case class Product(name: String, price: Double, stock: Int)
object Product {
  implicit val schema: Schema[Product] = Schema.derived
}

val before = Product("Widget", 9.99, 100)
val after = Product("Widget", 12.99, 95)

val patch = before.diff(after)
// Patch contains: price changed from 9.99 to 12.99, stock from 100 to 95

patch.isEmpty  // false
```

An identical comparison produces an empty patch:

```scala

case class Item(id: Int, name: String)
object Item {
  implicit val schema: Schema[Item] = Schema.derived
}

val item = Item(1, "Example")
val samePatch = item.diff(item)
samePatch.isEmpty  // true
```

### applyPatch

Applies a patch to a value, returning the modified value. Uses lenient mode by default, which means operations that can't be applied are silently skipped:

```scala

case class Counter(name: String, value: Int)
object Counter {
  implicit val schema: Schema[Counter] = Schema.derived
}

val counter = Counter("hits", 100)
val updated = Counter("hits", 150)

val patch = counter.diff(updated)
val result = counter.applyPatch(patch)
// Counter("hits", 150)
```

### applyPatchStrict

Applies a patch strictly, returning an `Either` that contains an error if any operation fails:

```scala

case class Record(id: String, version: Int)
object Record {
  implicit val schema: Schema[Record] = Schema.derived
}

val record = Record("abc", 1)
val newRecord = Record("abc", 2)
val patch = record.diff(newRecord)

val result: Either[SchemaError, Record] = record.applyPatchStrict(patch)
// Right(Record("abc", 2))

// Empty patch also succeeds
val emptyResult = record.applyPatchStrict(Patch.empty[Record])
// Right(Record("abc", 1))
```

## Roundtrip Examples

### JSON Roundtrip

```scala

case class Order(id: Long, items: List[String], total: BigDecimal)
object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

val order = Order(12345, List("apple", "banana"), BigDecimal("19.99"))

// String roundtrip
val jsonStr = order.toJsonString
val decoded1 = jsonStr.fromJson[Order]
// Right(Order(12345, List("apple", "banana"), 19.99))

// Bytes roundtrip
val jsonBytes = order.toJsonBytes
val decoded2 = jsonBytes.fromJson[Order]
// Right(Order(12345, List("apple", "banana"), 19.99))
```

### Patch Roundtrip

```scala

case class Settings(theme: String, fontSize: Int, notifications: Boolean)
object Settings {
  implicit val schema: Schema[Settings] = Schema.derived
}

val defaults = Settings("light", 14, true)
val customized = Settings("dark", 16, false)

// Compute patch and apply
val patch = defaults.diff(customized)
val result = defaults.applyPatch(patch)
// Settings("dark", 16, false)

assert(result == customized)
```

## Edge Cases

### Special Characters

The JSON encoding handles special characters, Unicode, and escape sequences correctly:

```scala

case class Text(content: String)
object Text {
  implicit val schema: Schema[Text] = Schema.derived
}

// Special characters
val special = Text("""John "Jack" O'Brien""")
val json1 = special.toJsonString
val decoded1 = json1.fromJson[Text]
// Right(Text("John \"Jack\" O'Brien"))

// Unicode
val unicode = Text("日本語テキスト")
val json2 = unicode.toJsonString
val decoded2 = json2.fromJson[Text]
// Right(Text("日本語テキスト"))
```

### Empty and Null Values

```scala

case class Profile(name: String, bio: Option[String])
object Profile {
  implicit val schema: Schema[Profile] = Schema.derived
}

// Empty strings
val empty = Profile("", None)
val json = empty.toJsonString
val decoded = json.fromJson[Profile]
// Right(Profile("", None))

// Optional fields
val withBio = Profile("Alice", Some("Developer"))
val withoutBio = Profile("Bob", None)
```

## Scala 2 vs Scala 3

The extension syntax works identically in both Scala 2 and Scala 3, but the implementation differs:

### Scala 3 (Extension Methods)

```scala
extension [A](self: A) {
  def toJson(using schema: Schema[A]): Json = ...
  def show(using schema: Schema[A]): String = ...
  def diff(that: A)(using schema: Schema[A]): Patch[A] = ...
  // ...
}

extension (self: String) {
  def fromJson[A](using schema: Schema[A]): Either[SchemaError, A] = ...
}

extension (self: Array[Byte]) {
  def fromJson[A](using schema: Schema[A]): Either[SchemaError, A] = ...
}
```

### Scala 2 (Implicit Classes)

```scala
implicit final class SchemaValueOps[A](private val self: A) {
  def toJson(implicit schema: Schema[A]): Json = ...
  def show(implicit schema: Schema[A]): String = ...
  def diff(that: A)(implicit schema: Schema[A]): Patch[A] = ...
  // ...
}

implicit final class StringSchemaOps(private val self: String) {
  def fromJson[A](implicit schema: Schema[A]): Either[SchemaError, A] = ...
}

implicit final class ByteArraySchemaOps(private val self: Array[Byte]) {
  def fromJson[A](implicit schema: Schema[A]): Either[SchemaError, A] = ...
}
```

The API is the same—just import `zio.blocks.schema._` and the appropriate syntax is provided for your Scala version.

## Method Reference

| Method             | Receiver      | Return Type              | Description                    |
|--------------------|---------------|--------------------------|--------------------------------|
| `toJson`           | `A`           | `Json`                   | Convert to JSON AST            |
| `toJsonString`     | `A`           | `String`                 | Convert to JSON string         |
| `toJsonBytes`      | `A`           | `Array[Byte]`            | Convert to UTF-8 bytes         |
| `fromJson[A]`      | `String`      | `Either[SchemaError, A]` | Parse JSON string              |
| `fromJson[A]`      | `Array[Byte]` | `Either[SchemaError, A]` | Parse JSON bytes               |
| `show`             | `A`           | `String`                 | Pretty-print via DynamicValue  |
| `diff`             | `A`           | `Patch[A]`               | Compute patch to another value |
| `applyPatch`       | `A`           | `A`                      | Apply patch (lenient)          |
| `applyPatchStrict` | `A`           | `Either[SchemaError, A]` | Apply patch (strict)           |

All methods require an implicit/given `Schema[A]` in scope.

---

## Type Class Derivation

Type classes are one of the most powerful abstraction mechanisms in functional programming. Originating from Haskell, they enable ad-hoc polymorphism—the ability to define generic behavior that can be extended to new types without modifying those types. ZIO Blocks has a robust type class derivation system built around the `Deriver` trait, which allows automatic generation of type class instances for any data type with an associated `Schema`.

The `Deriver` trait is a cornerstone of ZIO Blocks' type class derivation system. It provides a unified, elegant mechanism for automatically generating type class instances (such as codecs) for any data type that has a `Schema`. Unlike traditional macro-based derivation approaches, `Deriver` requires implementing only a few methods to enable full type class derivation with rich reflective metadata support for every use case.

## The Problem

In functional programming, type classes allow us to define generic behavior that can be extended to new types without modifying those types. However, manually writing type class instances for every data type can be tedious and error-prone, especially as the number of types grows. This is where automatic derivation comes in.

Consider a typical application with 50 domain types that needs 4 type classes (JSON codec, Avro codec, hashing, ordering). That's 200 type class instances to write and maintain manually (50 types × 4 type classes). 

Each instance requires understanding both the type's structure and the type class's semantics, then correctly implementing encoding, decoding, or whatever operation is required. This quickly becomes unmanageable as the codebase grows.

Assume we have a simple `JsonTC` type class for JSON serialization and deserialization:

```scala

sealed abstract class JsonError(msg: String) extends Exception(msg)

case class ParseError(details: String) 
  extends JsonError(s"Parse Error: $details")

case class DecodeError(details: String, path: String) 
  extends JsonError(s"Decode Error at '$path': $details")

trait JsonTC[A] {
  def encode(a: A): Json
  def decode(j: Json): Either[JsonError, A]
}
```

A single manual codec for a simple type like `Person` looks like the following code. You can imagine how complex it gets for larger types and more type classes:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val personCodec: JsonTC[Person] =
    new JsonTC[Person] {
      def encode(c: Person): Json = Json.obj(
        "name" -> Json.str(c.name),
        "age"  -> Json.number(c.age)
      )

      def decode(j: Json): Either[JsonError, Person] =
        for {
          name <- j.get("name").asString.string
          age  <- j.get("age").asNumber.int
        } yield Person(name, age)
    }
}
```

This manual approach is not only time-consuming but also prone to errors and inconsistencies. As the number of types and type classes increases, the maintenance burden grows significantly.

## The Solution: Automatic Derivation with `Deriver`

The `Deriver` trait provides a powerful and flexible way to automatically derive type class instances for any data type with an associated `Schema`. By implementing just seven methods, you can enable full derivation for primitive types, records, variants, sequences, maps, dynamic values, and wrappers.

ZIO Blocks recognizes that all data types reduce to a small set of structural patterns (as outlined in the `Reflect` documentation):

| Pattern       | Description                     | Examples                           |
|---------------|---------------------------------|------------------------------------|
| **Primitive** | Atomic values                   | `String`, `Int`, `UUID`, `Instant` |
| **Record**    | Product types with named fields | Case classes, tuples               |
| **Variant**   | Sum types with named cases      | Sealed traits, enums               |
| **Sequence**  | Ordered collections             | `List`, `Vector`, `Array`          |
| **Map**       | Key-value collections           | `Map`, `HashMap`                   |
| **Dynamic**   | Schema-less data                | `DynamicValue`, arbitrary JSON     |
| **Wrapper**   | Newtypes and opaque types       | `opaque type Age = Int`            |

If you define how to derive type-class instances for all these patterns, then ZIO Blocks has all the pieces needed to build type-class instances for any data type. This is what the `Deriver[TC[_]]` is responsible for. A `Deriver[TC[_]]` defines how to create `TC[A]` instances for each kind of schema node:

```scala
trait Deriver[TC[_]] {
  def derivePrimitive[A](...)                     : Lazy[TC[A]]
  def deriveRecord   [F[_, _], A](...)            : Lazy[TC[A]]
  def deriveVariant  [F[_, _], A](...)            : Lazy[TC[A]]
  def deriveSequence [F[_, _], C[_], A](...)      : Lazy[TC[C[A]]]
  def deriveMap      [F[_, _], M[_, _], K, V](...): Lazy[TC[M[K, V]]]
  def deriveDynamic  [F[_, _]](...)               : Lazy[TC[DynamicValue]]
  def deriveWrapper  [F[_, _], A, B](...)         : Lazy[TC[A]]
}
```

Conceptually, the `Deriver` interface operates at the meta level, acting as a type class for type class derivation. It takes a higher-kinded type parameter `TC[_]`, which represents the type class to be derived (e.g., `JsonCodec`, `Ordering`, `Eq`, etc.), and defines seven methods, each corresponding to the derivation of the type class for one of the structural patterns.

That's it. As a developer who wants to implement automatic derivation for a new type class, you only need to implement these 7 methods. Each receives all the information needed to build a type class instance such as field names, type names, bindings for construction/deconstruction, documentation, and modifiers.

Looking at the return type of each method, you'll notice they all return the type class wrapped in a `Lazy` container, i.e., `Lazy[TC[_]]`, not just `TC[_]`. This is crucial for handling recursive data types safely. While the `Deriver` system traverses the schema structure to generate type-class instances or codecs, it may encounter recursive data types. To prevent stack overflows caused by unbounded recursion and infinite loops, ZIO Blocks uses the `Lazy` data type, which is a trampolined, memoizing lazy evaluation monad that defers computation until `Lazy#force` is called. It provides stack-safe evaluation through continuation-passing style (CPS), along with error-handling capabilities and composable operations.

Each method (except the `derivePrimitive` method) also receives implicit parameters of type class instances for `HasBinding` and `HasInstance`:
1. **`HasBinding[F]`**: Provides access to the structural binding information (constructors, deconstructors, matchers, discriminators, etc.) for the contained types, e.g., fields of a record or cases of a variant, allowing us to understand how to construct and deconstruct values of those types.
2. **`HasInstance[F, TC]`**: Provides access to already-(provided/derived) type class instances for nested types or fields. This allows you to build type class instances for complex types by composing instances of their constituent parts.

As an example, the `deriveRecord` method signature looks like this:

```scala
trait Deriver[TC[_]] {
  // other methods...
  
  def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: HasInstance[F]): Lazy[TC[A]]
  
  // other methods...
}
```

The other methods follow a similar pattern, each tailored to the specific structural pattern they handle.

The underlying derivation engine takes care of traversing the schema structure, applying the appropriate derivation method for each structural pattern, and composing the resulting type class instances together. This means that once you've implemented a `Deriver` for a specific type class, you can automatically derive instances for any data type with a schema, without writing any additional boilerplate code.

## Using the `Deriver` to Derive Type Class Instances

Given a `Schema[A]`, you can call the `derive` method to get an instance of the type class `TC[A]`:

```scala
case class Schema[A](reflect: Reflect.Bound[A]) {
   def derive[D, TC[_]](d: D)(implicit ev: Derivable[D, TC]): TC[A] = ???
}
```

It takes a `Deriver[TC]` as a parameter and returns a type class instance of type `TC[A]`. For example, in the following code snippet, we derive a `JsonCodec[Person]` instance for the `Person` case class using the `JsonCodecDeriver`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}

val jsonCodec: JsonCodec[Person] = Person.schema.derive(JsonCodecDeriver)

val result: Either[SchemaError, Person] = 
  jsonCodec.decode(
    """
      |{
      |  "name": "Alice",
      |  "age": 30
      |}
      |""".stripMargin
  )
```

There is a `Derivable` type class that enables seamless overloading between `Deriver[TC]` and `Format` arguments, allowing the same `derive` method to work with both:

For example, by calling `Person.schema.derive(JsonFormat)`, we can derive a `JsonCodec[Person]` instance:

```scala

val jsonCodec = Person.schema.derive(JsonFormat)
```

## Example 1: Deriving a `Show` Type Class Instance

Let's say we want to derive a `Show` type class instance for any type of type `A`:

```scala
trait Show[A] {
  def show(value: A): String
}
```

The implementation of the `Deriver[Show]` would look like the following code. Don't worry about understanding every detail right now; we'll break down the derivation process step by step afterward.

```scala

object DeriveShow extends Deriver[Show] {

  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Show[A]] =
    Lazy {
      new Show[A] {
        def show(value: A): String = primitiveType match {
          case _: PrimitiveType.String => "\"" + value + "\""
          case _: PrimitiveType.Char   => "'" + value + "'"
          case _                       => String.valueOf(value)
        }
      }
    }

  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    // Pre-compute structural setup outside Lazy
    val fieldNames    = fields.map(_.name)
    // Cast fields to use Binding as F (we are going to create Reflect.Record with Binding as F)
    val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
    // Cast to Binding.Record to access constructor/deconstructor
    val recordBinding = binding.asInstanceOf[Binding.Record[A]]
    // Build a Reflect.Record to get access to the computed registers for each field
    val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
    Lazy {
      new Show[A] {
        // Defer child-instance resolution to first show() call via lazy val.
        // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
        // field.value.metadata during derivation would re-enter a Deferred node that is
        // still initializing, causing an infinite loop. The lazy val here ensures access
        // only happens after the framework has finished deriving all instances.
        private lazy val resolvedShows: IndexedSeq[Show[Any]] =
          fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Show[Any]]].force)
        def show(value: A): String = {
          // Create registers with space for all used registers to hold deconstructed field values
          val registers = Registers(recordReflect.usedRegisters)
          // Deconstruct field values of the record into the registers
          recordBinding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
          // Build string representations for all fields
          val fieldStrings = fields.indices.map { i =>
            val fieldValue = recordReflect.registers(i).get(registers, RegisterOffset.Zero)
            s"${fieldNames(i)} = ${resolvedShows(i).show(fieldValue)}"
          }
          s"${typeId.name}(${fieldStrings.mkString(", ")})"
        }
      }
    }
  }

  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    // Collect the Lazy[Show] references for each case outside the Lazy block.
    // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
    val caseShowLazies: IndexedSeq[Lazy[Show[Any]]] = cases.map { case_ =>
      D.instance(case_.value.metadata).asInstanceOf[Lazy[Show[Any]]]
    }
    // Cast binding to Binding.Variant to access discriminator and matchers
    val variantBinding = binding.asInstanceOf[Binding.Variant[A]]
    Lazy {
      new Show[A] {
        // Force child instances lazily — same recursive-safety rationale as deriveRecord
        private lazy val resolvedShows: IndexedSeq[Show[Any]] = caseShowLazies.map(_.force)
        // Implement show by using discriminator and matchers to find the right case
        // The `value` parameter is of type A (the variant type), e.g. a Shape value
        def show(value: A): String = {
          // Use discriminator to determine which case this value belongs to
          val caseIndex = variantBinding.discriminator.discriminate(value)
          // Use matcher to downcast to the specific case type
          val caseValue = variantBinding.matchers(caseIndex).downcastOrNull(value)
          // Delegate to the case's Show instance — it already knows its own name
          resolvedShows(caseIndex).show(caseValue)
        }
      }
    }
  }

  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[C[A]]] = {
    // Cast binding to Binding.Seq to access the deconstructor
    val deconstructor = binding.asInstanceOf[Binding.Seq[C, A]].deconstructor
    // Sequences are structurally non-recursive, so we can use monadic .map composition.
    // instance(...).map { elementShow => ... } returns a Lazy that, when forced, builds
    // a Show[C[A]] with elementShow already resolved — no .force needed at show()-time.
    D.instance(element.metadata).map { elementShow =>
      new Show[C[A]] {
        def show(value: C[A]): String = {
          // Use deconstructor to iterate over elements and show each one
          val elements = deconstructor.deconstruct(value).map(elementShow.show)
          s"[${elements.mkString(", ")}]"
        }
      }
    }
  }

  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[M[K, V]]] = {
    // Cast binding to Binding.Map to access the deconstructor
    val deconstructor = binding.asInstanceOf[Binding.Map[M, K, V]].deconstructor
    // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
    // to build the Show[M[K,V]] with both keyShow and valueShow already resolved.
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyShow, valueShow) =>
      new Show[M[K, V]] {
        def show(m: M[K, V]): String = {
          // Use deconstructor to iterate over key-value pairs
          val entries = deconstructor.deconstruct(m).map { kv =>
            val k = deconstructor.getKey(kv)
            val v = deconstructor.getValue(kv)
            s"${keyShow.show(k)} -> ${valueShow.show(v)}"
          }.mkString(", ")
          s"Map($entries)"
        }
      }
    }
  }

  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[DynamicValue]] = Lazy {
    new Show[DynamicValue] {
      def show(value: DynamicValue): String =
        value match {
          case DynamicValue.Primitive(pv) =>
            value.toString

          case DynamicValue.Record(fields) =>
            val fieldStrings = fields.map { case (name, v) =>
              s"$name = ${show(v)}"
            }
            s"Record(${fieldStrings.mkString(", ")})"

          case DynamicValue.Variant(caseName, v) =>
            s"$caseName(${show(v)})"

          case DynamicValue.Sequence(elements) =>
            val elemStrings = elements.map(show)
            s"[${elemStrings.mkString(", ")}]"

          case DynamicValue.Map(entries) =>
            val entryStrings = entries.map { case (k, v) =>
              s"${show(k)} -> ${show(v)}"
            }
            s"Map(${entryStrings.mkString(", ")})"
          case Null =>
            "null"
        }
    }
  }

  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
    // Cast binding to Binding.Wrapper to access the unwrap function
    val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
    // Wrappers are non-recursive: use .map so wrappedShow is already resolved
    // when show() is called — no .force needed.
    D.instance(wrapped.metadata).map { wrappedShow =>
      new Show[A] {
        def show(value: A): String = {
          // Unwrap the value to access the underlying type B, then delegate to its Show
          val unwrapped = wrapperBinding.unwrap(value)
          s"${typeId.name}(${wrappedShow.show(unwrapped)})"
        }
      }
    }
  }
}
```

Now let's see how the derivation process works step by step.

### Primitive Derivation

When the derivation process encounters a primitive type (e.g., `String`, `Int`), it calls the `derivePrimitive` method of the `Deriver`. This method receives the `PrimitiveType[A]` information, which allows it to determine how to encode and decode values of that type:

```scala
def derivePrimitive[A](
  primitiveType: PrimitiveType[A],
  typeId: TypeId[A],
  binding: Binding.Primitive[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
): Lazy[Show[A]] =
  Lazy {
    new Show[A] {
      def show(value: A): String = primitiveType match {
        case _: PrimitiveType.String => "\"" + value + "\""
        case _: PrimitiveType.Char   => "'" + value + "'"
        case _                       => String.valueOf(value)
      }
    }
  }
```

Please note that for our simple `Show` type class, we only need to know the `PrimitiveType` to determine how to show the value. However, for more complex type classes you might require additional information from the other parameters (e.g., documentation, modifiers, default values, examples) to build a more sophisticated type class instance.

To make it simple, we only handle `String` and `Char` differently by adding quotes around them, while for all other primitive types we simply call `String.valueOf(value)` to get their string representation. You can easily extend this logic to handle other primitive types differently if needed.

### Record Derivation

When the derivation process encounters a record type (e.g., a case class), it calls the `deriveRecord` method of the `Deriver`. This method receives an `IndexedSeq[Term[F, A, ?]]` representing the fields of the record, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples. It also receives implicit parameters for accessing structural bindings and already-derived type class instances for nested types:

```scala
def deriveRecord[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Record[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
  // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
  val fieldNames    = fields.map(_.name)
  // Cast fields to use Binding as F (we are going to create Reflect.Record with Binding as F)
  val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
  // Cast to Binding.Record to access constructor/deconstructor
  val recordBinding = binding.asInstanceOf[Binding.Record[A]]
  // Build a Reflect.Record to get access to the computed registers for each field
  val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
  Lazy {
    new Show[A] {
      // Defer child-instance resolution to first show() call via lazy val.
      // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
      // field.value.metadata during derivation would re-enter a Deferred node that is
      // still initialising, causing an infinite loop. The lazy val here ensures access
      // only happens after the framework has finished deriving all instances.
      private lazy val resolvedShows: IndexedSeq[Show[Any]] =
        fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Show[Any]]].force)
      def show(value: A): String = {
        // Create registers with space for all used registers to hold deconstructed field values
        val registers = Registers(recordReflect.usedRegisters)
        // Deconstruct field values of the record into the registers
        recordBinding.deconstructor.deconstruct(registers, RegisterOffset.Zero, value)
        // Build string representations for all fields
        val fieldStrings = fields.indices.map { i =>
          val fieldValue = recordReflect.registers(i).get(registers, RegisterOffset.Zero)
          s"${fieldNames(i)} = ${resolvedShows(i).show(fieldValue)}"
        }
        s"${typeId.name}(${fieldStrings.mkString(", ")})"
      }
    }
  }
}
```

The `deriveRecord` method demonstrates derivation mechanics for record types such as case classes and tuples. To derive the type class for a record type, we follow these steps:
1. First, we pre-compute structural setup (field names, `Reflect.Record`, binding casts) outside the `Lazy` block — none of this accesses `Deferred` nodes.
2. Second, inside `Lazy`, we build a `new Show[A]` with a `private lazy val resolvedShows` that resolves child instances on demand.
3. Third, when `show()` is first called, `resolvedShows` forces the child `Lazy[Show]` instances and then deconstructs the record value into its individual fields.
4. Finally, we assemble the string representation by combining field names and their representations.

During the first step, the structural setup is computed outside the `Lazy` block so the `Lazy` thunk itself is lightweight. This separation is important because structural types like `Reflect.Record` don't access `Deferred` schema nodes that might not yet be initialized.

The key insight is the `private lazy val resolvedShows` inside the `new Show[A]` class. It gathers `Lazy[Show]` instances for each field by calling `D.instance(field.value.metadata)`, then immediately forces them. These calls happen lazily — only on the first invocation of `show()` — which is safe because by that time the framework has completed derivation and all `Lazy` values are fully memoized. Without this deferral, recursive types like `case class Tree(value: Int, children: List[Tree])` would cause an infinite loop: deriving `Show[Tree]` requires `Show[List[Tree]]`, which requires `Show[Tree]` again.

Our goal is to build a `String` representation of the record in the format `TypeName(field1 = value1, field2 = value2, ...)`. To achieve this, we need to access the individual field values of the record at runtime. To do this, we have to deconstruct the record value, which is given to the `show(value: A)` method, into its individual fields.

To deconstruct the record, we use the `Binding.Record[A]` that was provided as a parameter to the `deriveRecord` method. This binding contains a `deconstructor` that knows how to extract all field values from a record of type `A`. To perform the deconstruction, we should first allocate register buffers to hold the deconstructed field values. But how do we know what the size of the register buffer should be? This is where the `Reflect.Record` comes in. By building a `Reflect.Record[Binding, A]` from the field definitions, we can compute the number of registers needed to hold all field values through `Reflect#usedRegisters`. The `Registers(recordReflect.usedRegisters)` call allocates a register buffer with the appropriate size to hold all field values of the record.

Now we are ready to deconstruct the `A` value, using the `Binding.Record#deconstructor.deconstruct(registers, RegisterOffset.Zero, value)` call, which extracts the field values of the record into this register buffer in a single pass. Now the field values are stored in `registers`.

The next question is how we can access the field values from the registers? The `Reflect.Record` we built earlier also computes the register layout for each field, which allows us to retrieve each field value from the appropriate register slot using `recordReflect.registers(i).get(registers, RegisterOffset.Zero)`. This call accesses the `i`-th field's value from the registers based on the register layout computed by `Reflect.Record`.

Finally, we iterate through each field, retrieve its value from the registers, look up the already-resolved `Show` instance for that field's type from `resolvedShows`, and format the result as `fieldName = fieldValue`. The output assembles into the familiar `TypeName(field1 = value1, field2 = value2)` representation.

### Variant Derivation

When the derivation process encounters a variant type (e.g., a sealed trait with case classes), it calls the `deriveVariant` method of the `Deriver`. This method receives an `IndexedSeq[Term[F, A, _]]` representing the cases of the variant, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveVariant[F[_, _], A](
  cases: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Variant[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
  // Collect the Lazy[Show] references for each case outside the Lazy block.
  // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
  val caseShowLazies: IndexedSeq[Lazy[Show[Any]]] = cases.map { case_ =>
    D.instance(case_.value.metadata).asInstanceOf[Lazy[Show[Any]]]
  }
  // Cast binding to Binding.Variant to access discriminator and matchers
  val variantBinding = binding.asInstanceOf[Binding.Variant[A]]
  Lazy {
    new Show[A] {
      // Force child instances lazily — same recursive-safety rationale as deriveRecord
      private lazy val resolvedShows: IndexedSeq[Show[Any]] = caseShowLazies.map(_.force)
      // Implement show by using discriminator and matchers to find the right case
      // The `value` parameter is of type A (the variant type), e.g. a Shape value
      def show(value: A): String = {
        // Use discriminator to determine which case this value belongs to
        val caseIndex = variantBinding.discriminator.discriminate(value)
        // Use matcher to downcast to the specific case type
        val caseValue = variantBinding.matchers(caseIndex).downcastOrNull(value)
        // Delegate to the case's Show instance — it already knows its own name
        resolvedShows(caseIndex).show(caseValue)
      }
    }
  }
}
```

The derivation process for variants is similar to records, but instead of fields, we have cases. The `Lazy[Show]` references for all cases are collected outside the `Lazy` block — this is safe because it only captures references without forcing them. Inside `Lazy`, a `private lazy val resolvedShows` forces all the child instances on the first `show()` call, applying the same recursive-safety guarantee as in `deriveRecord`.

At runtime, we use the discriminator to determine which case the value belongs to, and then the matcher to downcast the value to the specific case type. Finally, we look up the already-resolved `Show` instance for that case from `resolvedShows` and delegate to it. Each case's `Show` instance already knows how to format itself (including its own type name), so no additional formatting is needed at the variant level.

### Sequence Derivation

When the derivation process encounters a sequence type (e.g., `List[A]`), it calls the `deriveSequence` method of the `Deriver`. This method receives a `Reflect[F, A]` representing the element type of the sequence, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveSequence[F[_, _], C[_], A](
  element: Reflect[F, A],
  typeId: TypeId[C[A]],
  binding: Binding.Seq[C, A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[C[A]],
  examples: Seq[C[A]]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[C[A]]] = {
  // Cast binding to Binding.Seq to access the deconstructor
  val deconstructor = binding.asInstanceOf[Binding.Seq[C, A]].deconstructor
  // Sequences are structurally non-recursive, so we can use monadic .map composition.
  // instance(...).map { elementShow => ... } returns a Lazy that, when forced, builds
  // a Show[C[A]] with elementShow already resolved — no .force needed at show()-time.
  D.instance(element.metadata).map { elementShow =>
    new Show[C[A]] {
      def show(value: C[A]): String = {
        // Use deconstructor to iterate over elements and show each one
        val elements = deconstructor.deconstruct(value).map(elementShow.show)
        s"[${elements.mkString(", ")}]"
      }
    }
  }
}
```

The derivation process for sequences is straightforward. Because sequences are structurally non-recursive, we can use monadic `Lazy` composition via `.map` instead of the `private lazy val` pattern needed for records and variants. `D.instance(element.metadata).map { elementShow => ... }` returns a `Lazy[Show[C[A]]]` that, when forced, produces a `Show[C[A]]` with `elementShow` already resolved — no explicit `.force` is needed inside `show()`. At runtime, we use the deconstructor to iterate over the elements of the sequence and call `elementShow.show` on each one. Finally, we combine all element representations into a bracketed string.

### Map Derivation

When the derivation process encounters a map type (e.g., `Map[K, V]`), it calls the `deriveMap` method of the `Deriver`. This method receives `Reflect[F, K]` and `Reflect[F, V]` representing the key and value types of the map, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveMap[F[_, _], M[_, _], K, V](
  key: Reflect[F, K],
  value: Reflect[F, V],
  typeId: TypeId[M[K, V]],
  binding: Binding.Map[M, K, V],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[M[K, V]],
  examples: Seq[M[K, V]]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[M[K, V]]] = {
  // Cast binding to Binding.Map to access the deconstructor
  val deconstructor = binding.asInstanceOf[Binding.Map[M, K, V]].deconstructor
  // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
  // to build the Show[M[K,V]] with both keyShow and valueShow already resolved.
  D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyShow, valueShow) =>
    new Show[M[K, V]] {
      def show(m: M[K, V]): String = {
        // Use deconstructor to iterate over key-value pairs
        val entries = deconstructor.deconstruct(m).map { kv =>
          val k = deconstructor.getKey(kv)
          val v = deconstructor.getValue(kv)
          s"${keyShow.show(k)} -> ${valueShow.show(v)}"
        }.mkString(", ")
        s"Map($entries)"
      }
    }
  }
}
```

The derivation process for maps is similar to sequences, but we have two child instances to compose. Maps are non-recursive, so we use `.zip` to pair the two child `Lazy` instances into a single `Lazy[(Show[K], Show[V])]`, then `.map` to build the `Show[M[K,V]]` with both `keyShow` and `valueShow` already resolved. At runtime, we use the deconstructor to iterate over the key-value pairs of the map, calling `keyShow.show` and `valueShow.show` on each pair. No explicit `.force` is needed since `.map` already handles forcing. Finally, we combine all entries into a `Map(...)` string.

### Dynamic Derivation

When the derivation process encounters a dynamic type (e.g., `DynamicValue`), it calls the `deriveDynamic` method of the `Deriver`. This method receives a `Binding.Dynamic` representing the dynamic type, along with other metadata such as documentation, modifiers, default values, and examples:

```scala
def deriveDynamic[F[_, _]](
  binding: Binding.Dynamic,
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[DynamicValue],
  examples: Seq[DynamicValue]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[DynamicValue]] = Lazy {
  new Show[DynamicValue] {
    def show(value: DynamicValue): String =
      value match {
        case DynamicValue.Primitive(pv) =>
          value.toString
        case DynamicValue.Record(fields) =>
          val fieldStrings = fields.map { case (name, v) =>
            s"$name = ${show(v)}"
          }
          s"Record(${fieldStrings.mkString(", ")})"
        case DynamicValue.Variant(caseName, v) =>
          s"$caseName(${show(v)})"
        case DynamicValue.Sequence(elements) =>
          val elemStrings = elements.map(show)
          s"[${elemStrings.mkString(", ")}]"
        case DynamicValue.Map(entries) =>
          val entryStrings = entries.map { case (k, v) =>
            s"${show(k)} -> ${show(v)}"
          }
          s"Map(${entryStrings.mkString(", ")})"
        case Null =>
          "null"
      }
  }
}
```

The derivation process for dynamic types is more complex because the data structure is not known at compile time. Instead, we must handle different cases based on the runtime type of `DynamicValue` using pattern matching. For each subtype: `Primitive` values are converted via `toString`, `Record` fields are recursively shown, `Variant` cases display the name and contained value, `Sequence` elements are shown in bracket notation, `Map` entries are displayed as key-value pairs, and `Null` returns the string "null".

### Wrapper Derivation

When the derivation process encounters a wrapper type (e.g., a value class, opaque type, or any type that wraps another type), it calls the `deriveWrapper` method of the `Deriver`. This method receives a `Reflect[F, B]` representing the wrapped (underlying) type, along with other metadata such as the type ID, binding information, documentation, modifiers, default values, and examples:

```scala
def deriveWrapper[F[_, _], A, B](
  wrapped: Reflect[F, B],
  typeId: TypeId[A],
  binding: Binding.Wrapper[A, B],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveShow.HasInstance[F]): Lazy[Show[A]] = {
  // Cast binding to Binding.Wrapper to access the unwrap function
  val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
  // Wrappers are non-recursive: use .map so wrappedShow is already resolved
  // when show() is called — no .force needed.
  D.instance(wrapped.metadata).map { wrappedShow =>
    new Show[A] {
      def show(value: A): String = {
        // Unwrap the value to access the underlying type B, then delegate to its Show
        val unwrapped = wrapperBinding.unwrap(value)
        s"${typeId.name}(${wrappedShow.show(unwrapped)})"
      }
    }
  }
}
```

The derivation process for wrapper types involves unwrapping the value to access the underlying type. Wrappers are non-recursive, so we use `.map` composition: `D.instance(wrapped.metadata).map { wrappedShow => ... }` returns a `Lazy[Show[A]]` that, when forced, produces a `Show[A]` with `wrappedShow` already resolved. At runtime, we use the `unwrap` function from the binding to retrieve the underlying `B` value and delegate to `wrappedShow.show` — no explicit `.force` is needed.

### Example Usages

To see how this derivation works in practice, we can define some simple data types and then derive `Show` instances for them using the `DeriveShow` object we implemented.

1. Example 1: Simple `Person` Record with Two Primitive Fields:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  implicit val show: Show[Person]     = schema.derive(DeriveShow)
}
```

Now we can use the derived `Show[Person]` instance to convert `Person` values to strings:

```scala
Person.show.show(Person("Alice", 30))
// res7: String = "Person(name = \"Alice\", age = 30)"
```

2. Simple Shape Variant (Circle, Rectangle)

```scala
sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
  implicit val show: Show[Shape]     = schema.derive(DeriveShow)
}
```

To show a `Shape` value, we can do the following:

```scala
val shape1: Shape = Circle(5.0)
// shape1: Shape = Circle(5.0)
Shape.show.show(shape1)
// res8: String = "Circle(radius = 5.0)"

val shape2: Shape = Rectangle(4.0, 6.0)
// shape2: Shape = Rectangle(width = 4.0, height = 6.0)
Shape.show.show(shape2)
// res9: String = "Rectangle(width = 4.0, height = 6.0)"
```

3. Recursive Tree and Expr

```scala
case class Tree(value: Int, children: List[Tree])
object Tree {
  implicit val schema: Schema[Tree] = Schema.derived[Tree]
  implicit val show: Show[Tree]     = schema.derive(DeriveShow)
}
```

The `Tree` is a record with a recursive field `children` of type `List[Tree]`. Let's see how the derived `Show[Tree]` instance handles this recursive structure:

```scala
val tree = Tree(1, List(Tree(2, List(Tree(4, Nil))), Tree(3, Nil)))
// tree: Tree = Tree(
//   value = 1,
//   children = List(
//     Tree(value = 2, children = List(Tree(value = 4, children = List()))),
//     Tree(value = 3, children = List())
//   )
// )
Tree.show.show(tree)
// res10: String = "Tree(value = 1, children = [Tree(value = 2, children = [Tree(value = 4, children = [])]), Tree(value = 3, children = [])])"
```

4. Example 4: Recursive Sealed Trait (Expr)

```scala
sealed trait Expr
case class Num(n: Int)           extends Expr
case class Add(a: Expr, b: Expr) extends Expr

object Expr {
  implicit val schema: Schema[Expr] = Schema.derived[Expr]
  implicit val show: Show[Expr]     = schema.derive(DeriveShow)
}
```

Similar to `Tree`, `Expr` is a recursive variant type. The derived `Show[Expr]` instance can handle this recursive structure as well:

```scala
val expr: Expr = Add(Num(1), Add(Num(2), Num(3)))
// expr: Expr = Add(a = Num(1), b = Add(a = Num(2), b = Num(3)))
Expr.show.show(expr)
// res11: String = "Add(a = Num(n = 1), b = Add(a = Num(n = 2), b = Num(n = 3)))"
```

5. Example 5: DynamicValue Example

```scala
implicit val dynamicShow: Show[DynamicValue] = Schema.dynamic.derive(DeriveShow)
```

Let's define a `DynamicValue` that represents a record with some primitive fields and a sequence field, then show it using the derived `Show[DynamicValue]` instance:

```scala
val manualRecord = DynamicValue.Record(
  Chunk(
    "id"    -> DynamicValue.Primitive(PrimitiveValue.Int(42)),
    "title" -> DynamicValue.Primitive(PrimitiveValue.String("Hello World")),
    "tags"  -> DynamicValue.Sequence(
      Chunk(
        DynamicValue.Primitive(PrimitiveValue.String("scala")),
        DynamicValue.Primitive(PrimitiveValue.String("zio"))
      )
    )
  )
)
// manualRecord: Record = Record(
//   IndexedSeq(
//     ("id", Primitive(Int(42))),
//     ("title", Primitive(String("Hello World"))),
//     (
//       "tags",
//       Sequence(IndexedSeq(Primitive(String("scala")), Primitive(String("zio"))))
//     )
//   )
// )

dynamicShow.show(manualRecord)
// res12: String = "Record(id = 42, title = \"Hello World\", tags = [\"scala\", \"zio\"])"
```

6. Example 6: Simple Email Wrapper Type

```scala
case class Email(value: String)
object Email {
  implicit val schema: Schema[Email] = Schema[String].transform(
    Email(_),
    _.value
  )
  implicit val show: Show[Email] = schema.derive(DeriveShow)
}
```

The `Email` type is a simple wrapper around `String`. Let's see how it shows an `Email` value:

```scala
val email = Email("alice@example.com")
// email: Email = Email("alice@example.com")
println(s"Email: ${Email.show.show(email)}")
// Email: Email("alice@example.com")
```

## Example 2: Deriving a `Gen` Type Class Instance

Let's say we want to derive a `Gen` type class instance for any type `A`:

```scala

trait Gen[A] {
  def generate(random: Random): A
}
```

Unlike `Show`, which is a type class for converting values of type `A` to something else (a `String`)—so you can think of it as a function of type `A => Output (String)`—the `Gen` type class is for generating values of type `A`. You can think of it as a function of type `Input (Random) => A`.

To implement the `Show` type class, we need to know what components type `A` is made up of, so we can convert each component to a `String` and combine them to form the final `String` representation of `A`. To do this, we need to be able to deconstruct a value of type `A` into its components. On the other hand, to implement the `Gen` type class, we need to know how to generate each component of type `A` using a `Random` input, and then combine those generated components to form a complete value of type `A`. This means that for `Gen`, we need to be able to construct a value of type `A` from its components, rather than deconstructing it. Therefore, in the derivation methods for `Gen`, we will use the `constructor` from the `Binding` to create values of type `A` from generated components.

Here is a simple pedagogical implementation of a `GenDeriver` that can derive `Gen` instances for various types:

```scala

object DeriveGen extends Deriver[Gen] {

  override def derivePrimitive[A](
    primitiveType: PrimitiveType[A],
    typeId: TypeId[A],
    binding: Binding.Primitive[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  ): Lazy[Gen[A]] =
    Lazy {
      new Gen[A] {
        def generate(random: Random): A = primitiveType match {
          case _: PrimitiveType.String  => random.alphanumeric.take(random.nextInt(10) + 1).mkString.asInstanceOf[A]
          case _: PrimitiveType.Char    => random.alphanumeric.head.asInstanceOf[A]
          case _: PrimitiveType.Boolean => random.nextBoolean().asInstanceOf[A]
          case _: PrimitiveType.Int     => random.nextInt().asInstanceOf[A]
          case _: PrimitiveType.Long    => random.nextLong().asInstanceOf[A]
          case _: PrimitiveType.Double  => random.nextDouble().asInstanceOf[A]
          case PrimitiveType.Unit       => ().asInstanceOf[A]
          // For brevity, other primitives default to their zero/empty value
          // In a real implementation, you'd want to handle all primitives and possibly use modifiers for ranges, etc.
          case _ =>
            defaultValue.getOrElse {
              throw new IllegalArgumentException(
                s"Gen derivation not implemented for primitive type $primitiveType " +
                        s"(typeId = $typeId) and no default value provided."
              )
            }
        }
      }
    }

  /**
   * Strategy:
   *   1. Get Gen type class instances for each field
   *   2. Generate random values for each field
   *   3. Use the constructor to build the record
   */
  override def deriveRecord[F[_, _], A](
    fields: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Record[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
    // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
    // Build Reflect.Record to access registers and constructor
    val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
    val recordBinding = binding.asInstanceOf[Binding.Record[A]]
    val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
    Lazy {
      new Gen[A] {
        // Defer child-instance resolution to first generate() call via lazy val.
        // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
        // field.value.metadata during derivation would re-enter a Deferred node that is
        // still initialising, causing an infinite loop. The lazy val here ensures access
        // only happens after the framework has finished deriving all instances.
        private lazy val resolvedGens: IndexedSeq[Gen[Any]] =
          fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Gen[Any]]].force)
        def generate(random: Random): A = {
          // Create registers to hold field values
          val registers = Registers(recordReflect.usedRegisters)
          // Generate each field and store in registers
          fields.indices.foreach { i =>
            val value = resolvedGens(i).generate(random)
            recordReflect.registers(i).set(registers, RegisterOffset.Zero, value)
          }
          // Construct the record from registers
          recordBinding.constructor.construct(registers, RegisterOffset.Zero)
        }
      }
    }
  }

  /**
   * Strategy:
   *   1. Get Gen type class instances for all cases
   *   2. Randomly pick a case
   *   3. Generate a value for that case
   */
  override def deriveVariant[F[_, _], A](
    cases: IndexedSeq[Term[F, A, ?]],
    typeId: TypeId[A],
    binding: Binding.Variant[A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
    // Collect the Lazy[Gen] references for each case outside the Lazy block.
    // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
    val caseGenLazies: IndexedSeq[Lazy[Gen[A]]] = cases.map { c =>
      D.instance(c.value.metadata).asInstanceOf[Lazy[Gen[A]]]
    }
    Lazy {
      new Gen[A] {
        // Force child instances lazily — same recursive-safety rationale as deriveRecord
        private lazy val resolvedGens: IndexedSeq[Gen[A]] = caseGenLazies.map(_.force)
        def generate(random: Random): A = {
          // Pick a random case and generate its value
          val caseIndex = random.nextInt(cases.length)
          resolvedGens(caseIndex).generate(random)
        }
      }
    }
  }

  /**
   * Strategy:
   *   1. Get Gen type class instances for the element type
   *   2. Generate 0-5 elements
   *   3. Build the collection using the constructor
   */
  override def deriveSequence[F[_, _], C[_], A](
    element: Reflect[F, A],
    typeId: TypeId[C[A]],
    binding: Binding.Seq[C, A],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[C[A]],
    examples: Seq[C[A]]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[C[A]]] = {
    // Cast binding to Binding.Seq to access the constructor
    val constructor  = binding.asInstanceOf[Binding.Seq[C, A]].constructor
    val elemClassTag = element.typeId.classTag.asInstanceOf[scala.reflect.ClassTag[A]]
    // Sequences are structurally non-recursive, so we can use monadic .map composition.
    // instance(...).map { elementGen => ... } returns a Lazy that, when forced, builds
    // a Gen[C[A]] with elementGen already resolved — no .force needed at generate()-time.
    D.instance(element.metadata).map { elementGen =>
      new Gen[C[A]] {
        def generate(random: Random): C[A] = {
          val length = random.nextInt(6) // 0 to 5 elements
          implicit val ct: scala.reflect.ClassTag[A] = elemClassTag

          if (length == 0) {
            constructor.empty[A]
          } else {
            // Build the collection by generating each element and adding it to the builder
            val builder = constructor.newBuilder[A](length)
            (0 until length).foreach { _ =>
              constructor.add(builder, elementGen.generate(random))
            }
            constructor.result(builder)
          }
        }
      }
    }
  }

  /**
   * Strategy:
   *   1. Get Gen type class instances for key and value types
   *   2. Generate 0-5 key-value pairs
   *   3. Build the map using the constructor
   */
  override def deriveMap[F[_, _], M[_, _], K, V](
    key: Reflect[F, K],
    value: Reflect[F, V],
    typeId: TypeId[M[K, V]],
    binding: Binding.Map[M, K, V],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[M[K, V]],
    examples: Seq[M[K, V]]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[M[K, V]]] = {
    // Cast binding to Binding.Map to access the constructor
    val constructor = binding.asInstanceOf[Binding.Map[M, K, V]].constructor
    // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
    // to build the Gen[M[K,V]] with both keyGen and valueGen already resolved.
    D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyGen, valueGen) =>
      new Gen[M[K, V]] {
        def generate(random: Random): M[K, V] = {
          val size = random.nextInt(6) // 0 to 5 entries

          if (size == 0) {
            constructor.emptyObject[K, V]
          } else {
            // Build the map by generating each key-value pair and adding it to the builder
            val builder = constructor.newObjectBuilder[K, V](size)
            (0 until size).foreach { _ =>
              constructor.addObject(builder, keyGen.generate(random), valueGen.generate(random))
            }
            constructor.resultObject(builder)
          }
        }
      }
    }
  }

  /**
   * Since DynamicValue can represent any schema type, we generate random
   * dynamic values by randomly choosing a variant and generating appropriate
   * content.
   */
  override def deriveDynamic[F[_, _]](
    binding: Binding.Dynamic,
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[DynamicValue],
    examples: Seq[DynamicValue]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[DynamicValue]] = Lazy {
    new Gen[DynamicValue] {
      // Helper to generate a random primitive value
      private def randomPrimitive(random: Random): DynamicValue.Primitive = {
        val primitiveType = random.nextInt(5)
        primitiveType match {
          case 0 => DynamicValue.Primitive(PrimitiveValue.Int(random.nextInt()))
          case 1 => DynamicValue.Primitive(PrimitiveValue.String(random.alphanumeric.take(10).mkString))
          case 2 => DynamicValue.Primitive(PrimitiveValue.Boolean(random.nextBoolean()))
          case 3 => DynamicValue.Primitive(PrimitiveValue.Double(random.nextDouble()))
          case 4 => DynamicValue.Primitive(PrimitiveValue.Long(random.nextLong()))
        }
      }

      def generate(random: Random): DynamicValue = {
        // Randomly choose what kind of DynamicValue to generate
        // Weight towards primitives and simpler structures to avoid deep nesting
        val valueType = random.nextInt(10)
        valueType match {
          case 0 | 1 | 2 | 3 | 4 =>
            // 50% chance: generate a primitive
            randomPrimitive(random)

          case 5 | 6 =>
            // 20% chance: generate a record with 1-3 fields
            val numFields = random.nextInt(3) + 1
            val fields    = (0 until numFields).map { i =>
              val fieldName  = s"field$i"
              val fieldValue = randomPrimitive(random)
              (fieldName, fieldValue: DynamicValue)
            }
            DynamicValue.Record(Chunk.from(fields))

          case 7 | 8 =>
            // 20% chance: generate a sequence of 0-3 primitives
            val numElements = random.nextInt(4)
            val elements    = (0 until numElements).map(_ => randomPrimitive(random): DynamicValue)
            DynamicValue.Sequence(Chunk.from(elements))

          case 9 =>
            // 10% chance: generate null
            DynamicValue.Null
        }
      }
    }
  }

  override def deriveWrapper[F[_, _], A, B](
    wrapped: Reflect[F, B],
    typeId: TypeId[A],
    binding: Binding.Wrapper[A, B],
    doc: Doc,
    modifiers: Seq[Modifier.Reflect],
    defaultValue: Option[A],
    examples: Seq[A]
  )(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
    // Cast binding to Binding.Wrapper to access the wrap function
    val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
    // Wrappers are non-recursive: use .map so wrappedGen is already resolved
    // when generate() is called — no .force needed.
    D.instance(wrapped.metadata).map { wrappedGen =>
      new Gen[A] {
        def generate(random: Random): A =
          // Generate a value of the underlying type B and wrap it into A
          wrapperBinding.wrap(wrappedGen.generate(random))
      }
    }
  }
}
```

### Primitive Derivation

The `derivePrimitive` method is responsible for deriving a `Gen` instance for primitive types. It matches on the specific primitive type and generates random values accordingly. For example, for `String`, it generates a random alphanumeric string of random length; for `Int`, it generates a random integer; and so on. The generated value is then cast to the appropriate type `A` and returned:

```scala
def derivePrimitive[A](
  primitiveType: PrimitiveType[A],
  typeId: TypeId[A],
  binding: Binding.Primitive[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
): Lazy[Gen[A]] =
  Lazy {
    new Gen[A] {
      def generate(random: Random): A = primitiveType match {
        case _: PrimitiveType.String  => random.alphanumeric.take(random.nextInt(10) + 1).mkString.asInstanceOf[A]
        case _: PrimitiveType.Char    => random.alphanumeric.head.asInstanceOf[A]
        case _: PrimitiveType.Boolean => random.nextBoolean().asInstanceOf[A]
        case _: PrimitiveType.Int     => random.nextInt(100).asInstanceOf[A]
        case _: PrimitiveType.Long    => random.nextLong().asInstanceOf[A]
        case _: PrimitiveType.Double  => random.nextDouble().asInstanceOf[A]
        case PrimitiveType.Unit       => ().asInstanceOf[A]
        // For brevity, other primitives default to their zero/empty value
        // In a real implementation, you would want to handle all primitives and possibly use modifiers for ranges, etc.
        case _ => defaultValue.getOrElse(null.asInstanceOf[A])
      }
    }
  }
```

To handle all primitive types, you would want to implement cases for each primitive type defined in your schema system. In a real implementation, you might also want to consider using modifiers to allow users to specify constraints on the generated values (e.g., string length, numeric ranges, etc.).

### Record Derivation

The `deriveRecord` method is responsible for deriving a `Gen` instance for record types, such as case classes and tuples. The strategy for deriving a record type involves three main steps:

```scala
def deriveRecord[F[_, _], A](
  fields: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Record[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
  // Pre-compute structural setup outside Lazy — none of this touches Deferred nodes
  // Build Reflect.Record to access registers and constructor
  val recordFields  = fields.asInstanceOf[IndexedSeq[Term[Binding, A, ?]]]
  val recordBinding = binding.asInstanceOf[Binding.Record[A]]
  val recordReflect = new Reflect.Record[Binding, A](recordFields, typeId, recordBinding, doc, modifiers)
  Lazy {
    new Gen[A] {
      // Defer child-instance resolution to first generate() call via lazy val.
      // For recursive types (e.g. case class Tree(children: List[Tree])), accessing
      // field.value.metadata during derivation would re-enter a Deferred node that is
      // still initialising, causing an infinite loop. The lazy val here ensures access
      // only happens after the framework has finished deriving all instances.
      private lazy val resolvedGens: IndexedSeq[Gen[Any]] =
        fields.map(field => D.instance(field.value.metadata).asInstanceOf[Lazy[Gen[Any]]].force)
      def generate(random: Random): A = {
        // Create registers to hold field values
        val registers = Registers(recordReflect.usedRegisters)
        // Generate each field and store in registers
        fields.indices.foreach { i =>
          val value = resolvedGens(i).generate(random)
          recordReflect.registers(i).set(registers, RegisterOffset.Zero, value)
        }
        // Construct the record from registers
        recordBinding.constructor.construct(registers, RegisterOffset.Zero)
      }
    }
  }
}
```

As shown above, the implementation of the `deriveRecord` method for `Gen` follows the same structure as its `Show` counterpart. The structural setup (field casts, `Reflect.Record`) is done outside `Lazy` to keep the thunk lightweight, and `resolvedGens` is a `private lazy val` that defers child-instance resolution until the first `generate()` call for recursive-type safety. The primary difference from `Show` is the data flow: instead of deconstructing an existing record to read its fields, we generate random values for each field via `resolvedGens(i).generate(random)`, store them in registers using `Register#set`, and then invoke the `constructor` from the `Binding` to create an instance of type `A`.

### Variant Derivation

The `deriveVariant` method is responsible for deriving a `Gen` instance for variant types, such as sealed traits with case classes:

```scala
def deriveVariant[F[_, _], A](
  cases: IndexedSeq[Term[F, A, ?]],
  typeId: TypeId[A],
  binding: Binding.Variant[A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
  // Collect the Lazy[Gen] references for each case outside the Lazy block.
  // Capturing Lazy refs is safe here; we only .force them later inside the lazy val.
  val caseGenLazies: IndexedSeq[Lazy[Gen[A]]] = cases.map { c =>
    D.instance(c.value.metadata).asInstanceOf[Lazy[Gen[A]]]
  }
  Lazy {
    new Gen[A] {
      // Force child instances lazily — same recursive-safety rationale as deriveRecord
      private lazy val resolvedGens: IndexedSeq[Gen[A]] = caseGenLazies.map(_.force)
      def generate(random: Random): A = {
        // Pick a random case and generate its value
        val caseIndex = random.nextInt(cases.length)
        resolvedGens(caseIndex).generate(random)
      }
    }
  }
}
```

The derivation process for `Gen` variants applies the same deferral pattern as `deriveRecord`. The `caseGenLazies` are collected outside the `Lazy` block — capturing `Lazy` references is safe — and `resolvedGens` forces them lazily on the first `generate()` call. At runtime, we randomly select one of the resolved `Gen` instances and call `generate` on it. This is simpler than the record case because there are no registers or constructors involved — each case's `Gen` already knows how to produce a complete value of its type.

### Sequence Derivation

The `deriveSequence` method is responsible for deriving a `Gen` instance for sequence types, such as `List[A]`:

```scala
def deriveSequence[F[_, _], C[_], A](
  element: Reflect[F, A],
  typeId: TypeId[C[A]],
  binding: Binding.Seq[C, A],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[C[A]],
  examples: Seq[C[A]]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[C[A]]] = {
  // Cast binding to Binding.Seq to access the constructor
  val constructor  = binding.asInstanceOf[Binding.Seq[C, A]].constructor
  val elemClassTag = element.typeId.classTag.asInstanceOf[scala.reflect.ClassTag[A]]
  // Sequences are structurally non-recursive, so we can use monadic .map composition.
  // instance(...).map { elementGen => ... } returns a Lazy that, when forced, builds
  // a Gen[C[A]] with elementGen already resolved — no .force needed at generate()-time.
  D.instance(element.metadata).map { elementGen =>
    new Gen[C[A]] {
      def generate(random: Random): C[A] = {
        val length = random.nextInt(6) // 0 to 5 elements
        implicit val ct: scala.reflect.ClassTag[A] = elemClassTag

        if (length == 0) {
          constructor.empty[A]
        } else {
          // Build the collection by generating each element and adding it to the builder
          val builder = constructor.newBuilder[A](length)
          (0 until length).foreach { _ =>
            constructor.add(builder, elementGen.generate(random))
          }
          constructor.result(builder)
        }
      }
    }
  }
}
```

A sequence is an object that contains multiple elements of the same type. Because sequences are non-recursive, we use `.map` composition: `D.instance(element.metadata).map { elementGen => ... }` returns a `Lazy[Gen[C[A]]]` that, when forced, builds a `Gen[C[A]]` with `elementGen` already resolved — no explicit `.force` is needed inside `generate()`. At runtime, we generate a random length (0–5). For an empty length, we return `constructor.empty`. Otherwise, we create a new builder using `constructor.newBuilder`, generate random element values with `elementGen.generate(random)`, add them with `constructor.add`, and finalise with `constructor.result`.

### Map Derivation

The `deriveMap` method is responsible for deriving a `Gen` instance for map types, such as `Map[K, V]`:

```scala
def deriveMap[F[_, _], M[_, _], K, V](
  key: Reflect[F, K],
  value: Reflect[F, V],
  typeId: TypeId[M[K, V]],
  binding: Binding.Map[M, K, V],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[M[K, V]],
  examples: Seq[M[K, V]]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[M[K, V]]] = {
  // Cast binding to Binding.Map to access the constructor
  val constructor = binding.asInstanceOf[Binding.Map[M, K, V]].constructor
  // Maps are non-recursive: use .zip to pair the two child Lazy instances, then .map
  // to build the Gen[M[K,V]] with both keyGen and valueGen already resolved.
  D.instance(key.metadata).zip(D.instance(value.metadata)).map { case (keyGen, valueGen) =>
    new Gen[M[K, V]] {
      def generate(random: Random): M[K, V] = {
        val size = random.nextInt(6) // 0 to 5 entries

        if (size == 0) {
          constructor.emptyObject[K, V]
        } else {
          // Build the map by generating each key-value pair and adding it to the builder
          val builder = constructor.newObjectBuilder[K, V](size)
          (0 until size).foreach { _ =>
            constructor.addObject(builder, keyGen.generate(random), valueGen.generate(random))
          }
          constructor.resultObject(builder)
        }
      }
    }
  }
}
```

The derivation process for maps is similar to sequences but requires composing two child instances. We use `.zip` to combine the `Lazy[Gen[K]]` and `Lazy[Gen[V]]` into a single `Lazy`, then `.map` to build the `Gen[M[K,V]]` with both `keyGen` and `valueGen` already resolved. At runtime, a random size (0–5) determines whether we return an empty map or build one entry by entry using `constructor.addObject(builder, keyGen.generate(random), valueGen.generate(random))`.

### Dynamic Derivation

The `deriveDynamic` method is responsible for deriving a `Gen` instance for dynamic types, such as `DynamicValue`. Since `DynamicValue` can represent any schema type, we generate random dynamic values by choosing a variant at random and generating the appropriate content for that variant. The implementation involves pattern matching on the `DynamicValue` type and generating content accordingly:

```scala
def deriveDynamic[F[_, _]](
  binding: Binding.Dynamic,
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[DynamicValue],
  examples: Seq[DynamicValue]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[DynamicValue]] = Lazy {
  new Gen[DynamicValue] {
    // Helper to generate a random primitive value
    private def randomPrimitive(random: Random): DynamicValue.Primitive = {
      val primitiveType = random.nextInt(5)
      primitiveType match {
        case 0 => DynamicValue.Primitive(PrimitiveValue.Int(random.nextInt()))
        case 1 => DynamicValue.Primitive(PrimitiveValue.String(random.alphanumeric.take(10).mkString))
        case 2 => DynamicValue.Primitive(PrimitiveValue.Boolean(random.nextBoolean()))
        case 3 => DynamicValue.Primitive(PrimitiveValue.Double(random.nextDouble()))
        case 4 => DynamicValue.Primitive(PrimitiveValue.Long(random.nextLong()))
      }
    }

    def generate(random: Random): DynamicValue = {
      // Randomly choose what kind of DynamicValue to generate
      // Weight towards primitives and simpler structures to avoid deep nesting
      val valueType = random.nextInt(10)
      valueType match {
        case 0 | 1 | 2 | 3 | 4 =>
          // 50% chance: generate a primitive
          randomPrimitive(random)

        case 5 | 6 =>
          // 20% chance: generate a record with 1-3 fields
          val numFields = random.nextInt(3) + 1
          val fields    = (0 until numFields).map { i =>
            val fieldName  = s"field$i"
            val fieldValue = randomPrimitive(random)
            (fieldName, fieldValue: DynamicValue)
          }
          DynamicValue.Record(Chunk.from(fields))

        case 7 | 8 =>
          // 20% chance: generate a sequence of 0-3 primitives
          val numElements = random.nextInt(4)
          val elements    = (0 until numElements).map(_ => randomPrimitive(random): DynamicValue)
          DynamicValue.Sequence(Chunk.from(elements))

        case 9 =>
          // 10% chance: generate null
          DynamicValue.Null
      }
    }
  }
}
```

Please note that the random generation logic in this example is basic and is intended for illustrative purposes only.

### Wrapper Derivation

The `deriveWrapper` method is responsible for deriving a `Gen` instance for wrapper types, such as value classes or opaque types:

```scala
def deriveWrapper[F[_, _], A, B](
  wrapped: Reflect[F, B],
  typeId: TypeId[A],
  binding: Binding.Wrapper[A, B],
  doc: Doc,
  modifiers: Seq[Modifier.Reflect],
  defaultValue: Option[A],
  examples: Seq[A]
)(implicit F: HasBinding[F], D: DeriveGen.HasInstance[F]): Lazy[Gen[A]] = {
  // Cast binding to Binding.Wrapper to access the wrap function
  val wrapperBinding = binding.asInstanceOf[Binding.Wrapper[A, B]]
  // Wrappers are non-recursive: use .map so wrappedGen is already resolved
  // when generate() is called — no .force needed.
  D.instance(wrapped.metadata).map { wrappedGen =>
    new Gen[A] {
      def generate(random: Random): A =
        // Generate a value of the underlying type B and wrap it into A
        wrapperBinding.wrap(wrappedGen.generate(random))
    }
  }
}
```

Wrappers are non-recursive, so we use `.map` composition: `D.instance(wrapped.metadata).map { wrappedGen => ... }` returns a `Lazy[Gen[A]]` that, when forced, builds a `Gen[A]` with `wrappedGen` already resolved. Within `generate()`, we generate a random value of the underlying type `B` and wrap it into `A` using the `wrap` function from the binding — no explicit `.force` is needed.

### Example Usages

To see how this derivation works in practice, we can define some simple data types and then derive `Gen` instances for them using the `DeriveGen` object we implemented.

1. Example 1: Simple `Person` Record with Two Primitive Fields:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
  implicit val gen: Gen[Person]       = schema.derive(DeriveGen)
}
```

Now we can use the derived `Gen[Person]` instance to generate random `Person` values:

```scala
val random = new Random(42) // Seeded for reproducible output
// random: Random = scala.util.Random@643bb7d2

Person.gen.generate(random)
// res14: Person = Person(name = "p", age = -1360544799)
Person.gen.generate(random)
// res15: Person = Person(name = "C7DgX", age = 392236186)
Person.gen.generate(random)
// res16: Person = Person(name = "AM6", age = 1184328952)
```

2. Simple Shape Variant (Circle, Rectangle)

```scala
sealed trait Shape
case class Circle(radius: Double)                   extends Shape
case class Rectangle(width: Double, height: Double) extends Shape

object Shape {
  implicit val schema: Schema[Shape] = Schema.derived[Shape]
  implicit val gen: Gen[Shape]       = schema.derive(DeriveGen)
}
```

To generate random `Shape` values, we can do the following:

```scala
Shape.gen.generate(random)
// res17: Shape = Rectangle(
//   width = 0.46365357580915334,
//   height = 0.7829017787900358
// )
Shape.gen.generate(random)
// res18: Shape = Rectangle(
//   width = 0.15195824856297624,
//   height = 0.43979982659080874
// )
Shape.gen.generate(random)
// res19: Shape = Rectangle(
//   width = 0.38656687435934867,
//   height = 0.17737847790937833
// )
Shape.gen.generate(random)
// res20: Shape = Rectangle(
//   width = 0.338307935145014,
//   height = 0.2506613258416336
// )
```

3. Team with Sequence of Members (List)

```scala
case class Team(members: List[String])

object Team {
  implicit val schema: Schema[Team] = Schema.derived[Team]
  implicit val gen: Gen[Team]       = schema.derive(DeriveGen)
}
```

Let's generate some random `Team` values:

```scala
Team.gen.generate(random)
// res21: Team = Team(List("zZY", "TZlZMZdVjx", "G", "iqf1Pt9", "S1q6qHNj0R"))
Team.gen.generate(random)
// res22: Team = Team(List("b94sbz0WFC"))
Team.gen.generate(random)
// res23: Team = Team(List("nwyT"))
```

4. Example 4: Recursive Tree

```scala
case class Tree(value: Int, children: List[Tree])

object Tree {
  implicit val schema: Schema[Tree] = Schema.derived[Tree]
  implicit val gen: Gen[Tree]       = schema.derive(DeriveGen)
}
```

The `Tree` is a record with a recursive field `children` of type `List[Tree]`. Let's see how the derived `Gen[Tree]` instance handles this recursive structure:

```scala
Tree.gen.generate(random)
// res24: Tree = Tree(value = 1205047495, children = List())
```

5. Example 5: DynamicValue Example

```scala
implicit val dynamicGen: Gen[DynamicValue] = Schema.dynamic.derive(DeriveGen)
```

Let's generate some random `DynamicValue` instances:

```scala
dynamicGen.generate(random)
// res25: DynamicValue = Primitive(Int(769973518))
dynamicGen.generate(random)
// res26: DynamicValue = Primitive(Long(8878934151639676041L))
dynamicGen.generate(random)
// res27: DynamicValue = Sequence(IndexedSeq(Primitive(Int(-1576812231))))
```

6. Example 6: Simple Email Wrapper Type

```scala
case class Email(value: String)

object Email {
  implicit val schema: Schema[Email] = Schema[String].transform(
    Email(_),
    _.value
  )
  implicit val gen: Gen[Email] = schema.derive(DeriveGen)
}
```

The `Email` type is a simple wrapper around `String`. Let's see how it generates random `Email` values:

```scala
Email.gen.generate(random)
// res28: Email = Email("zlLKVaEitt")
Email.gen.generate(random)
// res29: Email = Email("Sa")
```

## Custom Type-class Instances

While automatic derivation generates type class instances for all substructures of a data type, there are times when you need to override the derived instance for a specific substructure. For example, you might want to use a custom `Show` instance for a particular field, provide a hand-written codec for a specific type that the deriver doesn't handle well, or inject a special implementation for testing purposes.

The `DerivationBuilder` provides an `instance` method that allows you to override the automatically derived type class instance for any part of the schema tree. You access the `DerivationBuilder` by calling `Schema#deriving(deriver)` instead of `Schema#derive(deriver)`:

```scala
val schema: Schema[A] = ...
val deriver: Deriver[TC] = ...

// Using derive: fully automatic, no customization
val tc: TC[A] = schema.derive(deriver)

// Using deriving: returns a DerivationBuilder for customization
val tc: TC[A] = schema.deriving(deriver)
  .instance(...)   // override specific instances
  .modifier(...)   // override specific modifiers
  .derive           // finalize the derivation
```

The `DerivationBuilder` offers three overloaded `instance` methods for providing custom type class instances:

```scala
final case class DerivationBuilder[TC[_], A](...) {
  def instance[B](optic: Optic[A, B], instance: => TC[B]): DerivationBuilder[TC, A]
  def instance[B](typeId: TypeId[B], instance: => TC[B]): DerivationBuilder[TC, A]
  def instance[P, B](typeId: TypeId[P], termName: String, instance: => TC[B]): DerivationBuilder[TC, A]
}
```

### Overriding by Optic

The first overload takes an `Optic[A, B]` that precisely targets a specific location within the schema tree. This is useful when you want to override the instance for a particular field or case without affecting other occurrences of the same type:

```scala

case class Person(name: String, age: Int)

object Person extends CompanionOptics[Person] {
  implicit val schema: Schema[Person] = Schema.derived[Person]

  val name: Lens[Person, String] = $(_.name)
  val age: Lens[Person, Int]     = $(_.age)
}
```

Now we can override the `Show[String]` instance specifically for the `name` field of `Person`:

```scala
val customNameShow: Show[String] = new Show[String] {
  def show(value: String): String = value.toUpperCase
}

val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(Person.name, customNameShow)
  .derive
```

When we show a `Person`, the `name` field will use the custom `Show[String]` instance (showing it in uppercase), while the `age` field will use the automatically derived `Show[Int]` instance:

```scala
personShow.show(Person("Alice", 30))
// res30: String = "Person(name = ALICE, age = 30)"
```

You can also target deeper nested fields using composed optics. For example, if you have a `Company` that contains a `Person`, you can target the `name` field inside the nested `Person`:

```scala
case class Company(ceo: Person, industry: String)

object Company extends CompanionOptics[Company] {
  implicit val schema: Schema[Company] = Schema.derived[Company]

  val ceo: Lens[Company, Person]       = $(_.ceo)
  val ceoName: Lens[Company, String]   = $(_.ceo.name)
  val industry: Lens[Company, String]  = $(_.industry)
}
```

```scala
val companyShow: Show[Company] = Company.schema
  .deriving(DeriveShow)
  .instance(Company.ceoName, customNameShow)
  .derive
```

In this case, the custom `Show[String]` instance only applies to the CEO's name. The `industry` field, which is also a `String`, will use the default derived `Show[String]` instance:

```scala
companyShow.show(Company(Person("Alice", 30), "tech"))
// res31: String = "Company(ceo = Person(name = ALICE, age = 30), industry = \"tech\")"
```

### Overriding by TypeId

The second overload takes a `TypeId[B]` and applies the custom instance to **all occurrences** of type `B` anywhere in the schema tree. This is useful when you want to override the instance for a type globally, without having to specify each location:

```scala
val customIntShow: Show[Int] = new Show[Int] {
  def show(value: Int): String = s"#$value"
}

val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(TypeId.int, customIntShow)
  .derive
```

All `Int` fields in the `Person` schema (in this case, just `age`) will use the custom `Show[Int]` instance:

```scala
personShow.show(Person("Alice", 30))
// res32: String = "Person(name = \"Alice\", age = #30)"
```

### Overriding by TypeId and Term Name

The third overload takes a `TypeId[P]` identifying the **parent** record or variant and a `termName` string identifying a field or case within it. This provides medium precision between optic-based (exact path) and type-based (all occurrences) overrides, and is useful when you want to override a specific field across all locations where the parent type appears without having to enumerate each path with an optic:

```scala
val customAgeShow: Show[Int] = new Show[Int] {
  def show(value: Int): String = s"age=$value"
}

val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(Person.schema.reflect.typeId, "age", customAgeShow)
  .derive
```

The `typeId` refers to the parent record/variant type (here `Person`), not the field type. If no term with the given name exists in the parent type, the override is silently ignored.

```scala
personShow.show(Person("Alice", 30))
// res33: String = "Person(name = \"Alice\", age = age=30)"
```

### Resolution Order

When the derivation engine encounters a schema node, it resolves the type class instance using the following priority order:

1. **Optic-based override** (most precise): If an instance override was registered using an optic that matches the current path in the schema tree, that instance is used.
2. **TypeId + term-name override** (medium precision): If no optic-based match is found, it checks for an override registered by parent type ID and term name.
3. **TypeId-based override** (more general): If no term-name match is found, it checks for an instance override registered by type ID.
4. **Automatic derivation** (default): If no override is found, the deriver's method (e.g., `derivePrimitive`, `deriveRecord`) is called to automatically derive the instance.

This means you can set a global override by type and then selectively refine specific fields using optics or term names:

```scala
val companyShow: Show[Company] = Company.schema
  .deriving(DeriveShow)
  .instance(TypeId.string, new Show[String] {
    def show(value: String): String = s"'$value'"
  })
  .instance(Company.ceoName, new Show[String] {
    def show(value: String): String = value.toUpperCase
  })
  .derive
```

In this example, all `String` fields use single quotes, except for the CEO's name which is shown in uppercase:

```scala
companyShow.show(Company(Person("Alice", 30), "tech"))
// res34: String = "Company(ceo = Person(name = ALICE, age = 30), industry = 'tech')"
```

### Chaining Multiple Overrides

The `instance` method returns a new `DerivationBuilder`, so you can chain multiple overrides fluently:

```scala
val personShow: Show[Person] = Person.schema
  .deriving(DeriveShow)
  .instance(Person.name, new Show[String] {
    def show(value: String): String = s"<<$value>>"
  })
  .instance(Person.age, new Show[Int] {
    def show(value: Int): String = s"age=$value"
  })
  .derive
```

```scala
personShow.show(Person("Alice", 30))
// res35: String = "Person(name = <<Alice>>, age = age=30)"
```

## Custom Modifiers

Modifiers are metadata annotations that influence how type class instances behave at runtime. For example, the `Modifier.rename` modifier tells a JSON codec to use a different field name during serialization, and `Modifier.transient` tells it to skip a field entirely.

While modifiers can be attached to schemas directly using Scala annotations (e.g., `@Modifier.transient`) or the `Schema#modifier` method, the `DerivationBuilder` provides a way to inject modifiers **programmatically at derivation time** without modifying the schema itself. This is particularly useful when:

- You don't control the schema definition (e.g., it comes from a library)
- You need different modifiers for different derivation contexts (e.g., one JSON codec with renamed fields, another without)
- You want to keep the schema clean and push format-specific concerns into the derivation layer

The `DerivationBuilder` offers three overloaded `modifier` methods:

```scala
final case class DerivationBuilder[TC[_], A](...) {
  def modifier[B](typeId: TypeId[B], modifier: Modifier.Reflect): DerivationBuilder[TC, A]
  def modifier[B](optic: Optic[A, B], modifier: Modifier): DerivationBuilder[TC, A]
  def modifier[B](typeId: TypeId[B], termName: String, modifier: Modifier.Term): DerivationBuilder[TC, A]
}
```

### Modifier Hierarchy

ZIO Blocks has two categories of modifiers:

- **`Modifier.Reflect`**: Type-level modifiers that apply to the schema node itself (e.g., `Modifier.config`).
- **`Modifier.Term`**: Field-level or case-level modifiers that apply to a specific field of a record or case of a variant (e.g., `Modifier.transient`, `Modifier.rename`, `Modifier.alias`).

Note that `Modifier.config` extends both `Modifier.Term` and `Modifier.Reflect`, so it can be used at both levels.

### Adding Modifiers by Optic

When you pass an optic and a `Modifier.Term` to the `modifier` method, the modifier is attached to the **term** (field or case) identified by the last segment of the optic path. When you pass a `Modifier.Reflect`, it is attached to the **schema node** targeted by the optic:

```scala

case class User(
  id: Long,
  name: String,
  email: String,
  internalScore: Double
)

object User extends CompanionOptics[User] {
  implicit val schema: Schema[User] = Schema.derived[User]

  val id: Lens[User, Long]              = $(_.id)
  val name: Lens[User, String]          = $(_.name)
  val email: Lens[User, String]         = $(_.email)
  val internalScore: Lens[User, Double] = $(_.internalScore)
}
```

Now we can derive a JSON codec with custom modifiers, renaming fields and marking one as transient, without changing the schema itself:

```scala
val jsonCodec: JsonCodec[User] = User.schema
  .deriving(JsonCodecDeriver)
  .modifier(User.name, Modifier.rename("full_name"))
  .modifier(User.email, Modifier.alias("mail"))
  .modifier(User.internalScore, Modifier.transient())
  .derive
```

In this example:
- The `name` field will be serialized as `full_name` in JSON.
- The `email` field will accept both `email` and `mail` as keys during deserialization.
- The `internalScore` field will be excluded from serialization entirely.

```scala
val user = User(1L, "Alice", "alice@example.com", 95.5)
// user: User = User(
//   id = 1L,
//   name = "Alice",
//   email = "alice@example.com",
//   internalScore = 95.5
// )
new String(jsonCodec.encode(user), "UTF-8")
// res36: String = "{\"id\":1,\"full_name\":\"Alice\",\"email\":\"alice@example.com\"}"
```

### Adding Modifiers by TypeId

The `modifier` method with `TypeId` allows you to add a `Modifier.Reflect` to all schema nodes of a given type. This is useful for attaching format-specific configuration metadata to all occurrences of a type:

```scala
val jsonCodec: JsonCodec[User] = User.schema
  .deriving(JsonCodecDeriver)
  .modifier(TypeId.of[User], Modifier.config("json", "camelCase"))
  .modifier(User.internalScore, Modifier.transient())
  .derive
```

### Adding Modifiers by TypeId and Term Name

The `modifier` method with `TypeId` and `termName` allows you to add a `Modifier.Term` to a specific field or case identified by name inside a parent type identified by its `TypeId`. This is useful when you want to target a specific term without constructing an optic for it. The `typeId` refers to the parent record/variant type that owns the term. If no term with the given name exists in the parent type, the modifier is silently ignored:

```scala
val jsonCodec: JsonCodec[User] = User.schema
  .deriving(JsonCodecDeriver)
  .modifier(User.schema.reflect.typeId, "name", Modifier.rename("full_name"))
  .modifier(User.schema.reflect.typeId, "internalScore", Modifier.transient())
  .derive
```

## Derivation Process In-Depth

Until now, we learned how to implement the `Deriver` methods for different schema patterns. But we haven't yet discussed how the overall derivation process works. In this section, we will go through the main steps of derivation in detail.

### PHASE 1: Deriving the Schema for the Target Type

The first step in deriving a type class instance is deriving a `Schema[A]` for the target type `A`. The `Schema[A]` contains a tree of `Reflect[Binding, A]` nodes that represent the structure of `A` using structural bindings:

For example, assume a case class of `Person(name: String, age: Int)`. The derived schema would look like this:

```
Schema[Person]
  └── Reflect.Record[Binding, Person]
        ├── Term("name", Reflect.Primitive[Binding, String])
        └── Term("age", Reflect.Primitive[Binding, Int])
```

Each node of the derived schema tree, carries two pieces of information:
- **Type Metadata**: Structural representation of the type (e.g., record, variant, primitive).
- **Binding Metadata**: Structural binding information for constructing/deconstructing values of that type.

This schema derivation is typically done using `Schema.derived[A]`, which uses Scala's compile-time reflection capabilities to inspect the structure of type `A` and build the corresponding schema.

For example, the following code derives the schema for `Person`:

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived[Person]
}
```

### PHASE 2: Schema Tree Transformation

After generating the schema, by calling `Schema[A]#derive(deriver: Deriver[TC])`, the derivation process begins. This process involves transforming the schema tree from one that contains only structural bindings to one that also includes derived type class instances.

Initially, a `Schema[A]` contains `Reflect[Binding, A]` nodes that represent the structure of the type `A` using structural bindings. During derivation, the `Deriver` transforms these nodes into `Reflect[BindingInstance[TC, _, _], A]` nodes, where each node now contains both the structural binding and the derived type class instance for that part of the structure.

This tree transformation process starts at the root of the schema and recursively traverses each node until it reaches the leaf nodes (primitives). Now it can derive the type class instances for each leaf node by calling the `derivePrimitive` deriver method, which returns the derived type class instance wrapped in a `Lazy` container, i.e., `Lazy[TC[A]]`. The derivation builder now converts that schema node from `Reflect[Binding, A]` to `Reflect[BindingInstance[TC, _, _], A]`, where the `BindingInstance` contains both the structural binding and the derived type class instance. After converting all the leaf nodes, it backtracks up the tree, calling the appropriate `Deriver` methods for each structural pattern (record, variant, sequence, map, dynamic, wrapper) to derive type class instances for the composite types. At each step, it transforms the schema nodes from `Reflect[Binding, A]` to `Reflect[BindingInstance[TC, _, _], A]` accordingly. This process continues until it reaches the root of the schema tree, resulting in a final schema of type `Schema[A]` that contains `Reflect[BindingInstance[TC, _, _], A]` nodes throughout the entire structure.

The following diagram illustrates this transformation process:

```
       ┌──────────────────────────────┐
       │      Reflect[Binding,A]      │
       ├──────────────────────────────┤
       │   STRUCTURAL BINDING ONLY    │
       └──────────────────────────────┘
                      │
                      │ transform
                      ▼
       ┌──────────────────────────────┐
       │  Reflect[BindingInstance,A]  │
       ├──────────────────────────────┤
       │      STRUCTURAL BINDING      │
       │  WITH TYPE-CLASS INSTANCE    │
       └──────────────────────────────┘
                      │
                      │ extract
                      ▼
       ┌──────────────────────────────┐
       │         Lazy[TC[A]]          │
       ├──────────────────────────────┤
       │     TYPE-CLASS INSTANCE      │
       │           (TC[A])            │
       └──────────────────────────────┘
```

The `BindingInstance` is a container that bundles together a structural `Binding` and a derived type class instance `TC[A]`:

```scala
case class BindingInstance[TC[_], T, A](
  binding: Binding[T, A],    // Original runtime binding
  instance: Lazy[TC[A]]      // The derived type-class instance
)
```

For example, the transformation sequence for the `Person` data type would look like this:

```scala
case class Person(name: String, age: Int)

object Person {
    implicit val schema: Schema[Person] = Schema.derived[Person]
    implicit val show: Show[Person]     = schema.derive(DeriveShow)
}
```

- Step 1: Transform Primitive "name" (String)
   - deriver.derivePrimitive(String) → Lazy[Show[String]]
   - Creating BindingInstance(Binding.Primitive, Lazy[Show[String]])
   - Converting reflect node of `String` Schema from `Reflect[Binding, String]` to `Reflect[BindingInstance, String]`

- Step 2: Transform Primitive "age" (Int)
   - deriver.derivePrimitive(Int) → Lazy[Show[Int]]
   - Creating BindingInstance(Binding.Primitive, Lazy[Show[Int]])
   - Converting reflect node of `Int` Schema from `Reflect[Binding, Int]` to `Reflect[BindingInstance, Int]`

- Step 3: Transform Record "Person"
   - deriver.deriveRecord(fields with transformed metadata) → Lazy[Show[Person]]
   - Creating BindingInstance(Binding.Record, Lazy[Show[Person]])
   - Converting reflect node of `Person` Schema from `Reflect[Binding, Person]` to `Reflect[BindingInstance, Person]`

### PHASE 3: Extracting the Derived Type Class Instance

After the schema tree has been fully transformed to contain `Reflect[BindingInstance[TC, _, _], A]` nodes, now each node has a `BindingInstance` containing the original binding and the derived type class instance. The metadata container `BindingInstance` of the root node contains the derived type class wrapped in a `Lazy` container, i.e., `Lazy[TC[A]]`. To get the final derived type class instance, we call `force` on the `Lazy[TC[A]]`, which forces the unevaluated computation and retrieves the actual type class instance `TC[A]`.

### Phase 4: Using the Derived Show Instance

After derivation is complete, you can use the derived type class instance as needed. For example, you can use the derived `Show[Person]` instance to display a `Person` object:

```scala
val result = Person.show.show(Person("Alice", 30))
// result: String = "Person(name = Alice, age = 30)"
```

The interesting part here is how the `show` method of the derived `Show[Person]` instance works. It uses the `HasInstance` type class to access the derived `Show` instances for each field of the `Person` record (i.e., `Show[String]` for the `name` field and `Show[Int]` for the `age` field). This allows it to recursively display each field using its respective `Show` instance, demonstrating the composability and reusability of type class instances in the derivation system.

Please note that this happens when either the `Deriver` implementation uses the `HasInstance` implicit parameter or uses the centralized recursive approach to access nested derived instances.

---

## TypeId

`TypeId[A]` represents the identity of a type or type constructor at runtime — it captures complete type metadata (names, type parameters, parent types, annotations, classification) that would otherwise be erased by the JVM and Scala.js. Use `TypeId` when you need to preserve full type information as data for serialization, code generation, registry lookups, or type-safe dispatching.

In Scala and the JVM, compile-time type information is erased at runtime. This means generic type parameters, sealed trait variants, and even opaque types become indistinguishable at runtime — `List[Int]` and `List[String]` both look like `List` to the JVM. This erasure makes it nearly impossible to implement universal serializers that work across formats (JSON, YAML, XML, MessagePack), code generators, or schema-driven transformations without losing semantic information. `TypeId` solves this by capturing complete type structure at compile time and making it available as a hashable, inspectable value at runtime.

The `TypeId` trait exposes the type's structure through a rich set of properties and predicates:

```scala
// Simplified — some members shown here are derived from abstract members
sealed trait TypeId[A <: AnyKind] {
  // Abstract members
  def name: String
  def owner: Owner
  def typeParams: List[TypeParam]
  def typeArgs: List[TypeRepr]
  def defKind: TypeDefKind
  def selfType: Option[TypeRepr]       // Self-type annotation, if any
  def aliasedTo: Option[TypeRepr]      // Target type for type aliases
  def representation: Option[TypeRepr] // Underlying type for opaque types
  def annotations: List[Annotation]

  // Derived properties
  final def fullName: String     // owner.asString + "." + name
  final def arity: Int           // typeParams.size
  final def isCaseClass: Boolean
  final def isSealed: Boolean
  final def isAlias: Boolean
  // ... many more derived predicates
}
```

> In Scala 3, `A` is bounded by `AnyKind` to support higher-kinded types. In Scala 2, the bound is omitted (`sealed trait TypeId[A]`).

Derive a `TypeId` for any type using the `TypeId.of` macro and then inspect the type's structure at runtime:

```scala

case class Person(name: String, age: Int)

val id = TypeId.of[Person]
```

```scala
id.name
// res1: String = "Person"
id.fullName
// res2: String = "repl.MdocSession.MdocApp0.Person"
id.isCaseClass
// res3: Boolean = true
```

## Motivation

Standard approaches to preserving type information at runtime — `ClassTag`, `TypeTag` (Scala 2), `TypeTest` (Scala 3) — each have limitations. `ClassTag` loses generic type arguments. `TypeTag` depends on `scala-reflect` and is unavailable on Scala.js. `TypeTest` only answers "is this value an instance of T?" without exposing type structure. None of them distinguish opaque types from their underlying representation.

TypeId takes a different approach: the `TypeId.of` macro captures type metadata at compile time and stores it as a plain, immutable data structure — no runtime reflection, no platform-specific APIs. This makes it suitable as a foundation for cross-platform schema systems, code generators, and type-indexed registries.

## Installation

TypeId is included in the `zio-blocks-typeid` module. Add it to your build:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-typeid" % "@VERSION"
```

For cross-platform (Scala.js):

```scala
libraryDependencies += "dev.zio" %%% "zio-blocks-typeid" % "@VERSION"
```

Supported Scala versions: 2.13.x and 3.x.

## Creating Instances

There are two approaches to creating `TypeId` values: **automatic derivation** (recommended for normal use) and **manual construction** (for advanced metaprogramming scenarios).

### Automatic Derivation

For most users and most types, automatic derivation via `TypeId.of` or implicit `derived` is the right choice. These macros extract complete type metadata at compile time, handling all type variants correctly.

#### `TypeId.of` — Macro Derivation

The primary way to obtain a `TypeId` is through the `TypeId.of[A]` macro, which extracts complete type metadata at compile time:

```scala
object TypeId {
  inline def of[A <: AnyKind]: TypeId[A]  // Scala 3
  def of[A]: TypeId[A]                     // Scala 2 (macro)
}
```

Derive a TypeId using the macro:

```scala

case class User(id: Long, email: String)
```

```scala
val userId = TypeId.of[User]
// userId: TypeId[User] = User
userId.name
// res5: String = "User"
userId.fullName
// res6: String = "repl.MdocSession.MdocApp4.User"
userId.isCaseClass
// res7: Boolean = true
```

#### `TypeId.derived` — Implicit Derivation

TypeId instances are available implicitly through the `derived` macro. Any function that requires a `TypeId[A]` in implicit scope will have it derived automatically — you never need to pass it manually.

From the user's perspective, the API is:

```scala
object TypeId {
  inline given derived[A <: AnyKind]: TypeId[A]  // Scala 3
  implicit def derived[A]: TypeId[A]              // Scala 2 (macro)
}
```

:::note
In Scala 3, the `[A <: AnyKind]` bound allows derivation for type constructors (e.g., `TypeId[List]`). In Scala 2, the bound is `[A]` and type constructor derivation uses `TypeId[List[_]]` syntax instead.
:::

The most common use case is accepting `TypeId[A]` as an implicit parameter:

```scala

case class User(id: Long, email: String)

def describe[A](implicit typeId: TypeId[A]): String =
  s"${typeId.fullName} is a case class: ${typeId.isCaseClass}"
```

Call the function with the type argument — the TypeId is derived automatically:

```scala
describe[User]
// res9: String = "repl.MdocSession.MdocApp8.User is a case class: true"
describe[Int]
// res10: String = "scala.Int is a case class: false"
```

You can also summon a TypeId explicitly with `implicitly` (Scala 2) or `summon` (Scala 3):

```scala
val userTypeId = implicitly[TypeId[User]]
// userTypeId: TypeId[User] = User
userTypeId.name
// res11: String = "User"
```

When you need the TypeId in a single expression, use `TypeId.of[A]`. For generic functions that accept any `A` and need its TypeId alongside other implicit evidence, use implicit derivation instead.

### Manual Derivation (Smart Constructors)

For advanced use cases — unit testing with synthetic metadata, code generators that create types dynamically, or frameworks that construct TypeIds at runtime — the smart constructor functions allow you to manually assemble TypeIds by specifying their components. These are never needed in normal user code, since `TypeId.of` handles all these cases automatically.

#### `TypeId.nominal` — Nominal Types

**Nominal types** are concrete type definitions: classes, traits, and objects. In contrast to [type aliases](#typeidalias--type-aliases) (which are alternative names for existing types) and [opaque types](#typeidopaque--opaque-types) (which have a hidden representation), nominal types stand as distinct, named types in the type system.

For most end users, you don't need to use `TypeId.nominal` directly. The `TypeId.of` macro automatically derives nominal TypeIds from your actual type definitions at compile time. The `nominal` smart constructor exists for **advanced use cases**: unit testing with synthetic type metadata, code generators that create types dynamically at runtime, or frameworks that assemble TypeIds programmatically. Unless you're in one of these scenarios, `TypeId.of` is the right tool.

If you do need to construct nominal TypeIds manually, the API provides two overloads:

```scala
object TypeId {
  def nominal[A <: AnyKind](name: String, owner: Owner, kind: TypeDefKind): TypeId[A]

  def nominal[A <: AnyKind](
    name: String, owner: Owner,
    typeParams: List[TypeParam] = Nil, typeArgs: List[TypeRepr] = Nil,
    defKind: TypeDefKind = TypeDefKind.Unknown,
    selfType: Option[TypeRepr] = None,
    annotations: List[Annotation] = Nil
  ): TypeId[A]
}
```

#### `TypeId.alias` — Type Aliases

**Type aliases** are alternative names for existing types. For example, `type Age = Int` creates an alias for `Int` so code can read `Age` instead of `Int`. TypeIds for type aliases preserve the distinction from their underlying type through the `aliasedTo` property, enabling alias-aware serialization and schema generation.

For normal use, you don't need `TypeId.alias` directly. When you write a type alias in your code (e.g., `type UserId = String`), the `TypeId.of` macro automatically derives the correct TypeId. The `alias` smart constructor is for **advanced use cases**: unit testing with synthetic alias metadata, code generators that create type aliases dynamically at runtime, or frameworks that normalize or transform type aliases during schema processing. Unless you're building one of these, `TypeId.of` is the right tool.

For testing or code generation, construct an alias TypeId:

```scala
object TypeId {
  def alias[A <: AnyKind](
    name: String, owner: Owner,
    typeParams: List[TypeParam] = Nil,
    aliased: TypeRepr,
    typeArgs: List[TypeRepr] = Nil,
    annotations: List[Annotation] = Nil
  ): TypeId[A]
}
```

```scala

```

```scala
val ageId = TypeId.alias[Any]("Age", Owner.fromPackagePath("com.example"), aliased = TypeRepr.Ref(TypeId.int))
// ageId: TypeId[Any] = Age
ageId.isAlias
// res13: Boolean = true
ageId.aliasedTo
// res14: Option[TypeRepr] = Some(Ref(Int))
```

#### `TypeId.opaque` — Opaque Types

**Opaque types** (a Scala 3 feature) are types that have a distinct compile-time identity but a hidden runtime representation. For example, `opaque type UserId = String` creates a type that is distinct from `String` at compile time, but represents `String` at runtime. TypeId preserves this distinction, unlike standard reflection which erases opaque types to their underlying type — a critical capability for type-safe serialization and validation.

For normal use, you don't need `TypeId.opaque` directly. When you define an opaque type in your code, the `TypeId.of` macro automatically derives the correct TypeId with its representation. The `opaque` smart constructor is for **advanced use cases**: unit testing with synthetic opaque type metadata, code generators that create opaque types dynamically, or frameworks that need to construct type metadata for dynamically-discovered opaque types. Unless you're building one of these, `TypeId.of` is the right tool.

For testing or code generation, construct an opaque TypeId:

```scala
object TypeId {
  def opaque[A <: AnyKind](
    name: String, owner: Owner,
    typeParams: List[TypeParam] = Nil,
    representation: TypeRepr,
    typeArgs: List[TypeRepr] = Nil,
    publicBounds: TypeBounds = TypeBounds.Unbounded,
    annotations: List[Annotation] = Nil
  ): TypeId[A]
}
```

#### `TypeId.applied` — Applied Types

**Applied types** are generic types instantiated with type arguments. For example, `List[Int]` is `List` (the type constructor) applied to `Int` (the type argument), and `Map[String, Int]` is `Map` applied to two type arguments. TypeIds for applied types preserve the type arguments so serializers can generate specialized codecs, validators can type-check values, and code generators can emit correct code.

For normal use, you don't need `TypeId.applied` directly. When you write an applied type in your code (e.g., `List[Int]` or `Map[String, User]`), the `TypeId.of` macro automatically derives the correct TypeId with its type arguments preserved. The `applied` smart constructor is for **advanced use cases**: unit testing with synthetic applied type metadata, code generators that construct type expressions dynamically, or frameworks that need to build type metadata at runtime for dynamically-discovered generic types. Unless you're building one of these, `TypeId.of` is the right tool.

For testing or code generation, construct applied TypeIds by combining a type constructor with type argument expressions:

```scala
object TypeId {
  def applied[A <: AnyKind](typeConstructor: TypeId[?], args: TypeRepr*): TypeId[A]
}
```

## Core Operations

This section documents all public methods on `TypeId` and its companion object, organized by category.

### Identity and Naming

These methods provide the type's name and fully qualified path.

#### `name` — Simple Type Name

Returns the unqualified name of the type:

```scala
sealed trait TypeId[A <: AnyKind] {
  def name: String
}
```

```scala

case class Order(id: String, total: Double)
val orderId = TypeId.of[Order]
```

```scala
orderId.name
// res16: String = "Order"
TypeId.int.name
// res17: String = "Int"
TypeId.list.name
// res18: String = "List"
```

#### `fullName` — Fully Qualified Name

Returns `owner.asString + "." + name`, or just `name` if the owner is root:

```scala
sealed trait TypeId[A <: AnyKind] {
  def fullName: String
}
```

```scala
orderId.fullName
// res19: String = "repl.MdocSession.MdocApp15.Order"
TypeId.int.fullName
// res20: String = "scala.Int"
TypeId.string.fullName
// res21: String = "java.lang.String"
```

#### `owner` — Enclosing Namespace

The `TypeId#owner` method returns the `Owner` — the hierarchical path showing exactly where a type is defined. This includes the complete package chain and any enclosing objects or types. `Owner` solves a critical problem: multiple types can have the same name (e.g., `User` in `com.api` and `User` in `com.admin`), and the owner uniquely distinguishes them by their definition location:

```scala
sealed trait TypeId[A <: AnyKind] {
  def owner: Owner
}
```

When we derive a `TypeId` for a custom type, the owner captures its full hierarchical location. We can then use `TypeId#fullName` to see how the owner combines with the type name:

```scala

case class User(id: Long, name: String)
```

```scala
val userId = TypeId.of[User]
// userId: TypeId[User] = User
userId.name
// res23: String = "User"
// The owner shows where this User is defined
userId.owner.asString
// res24: String = "repl.MdocSession.MdocApp22"
// fullName combines owner and name into a qualified path
userId.fullName
// res25: String = "repl.MdocSession.MdocApp22.User"
```

When we construct types from different packages, their owners differ even though the names are identical. This is essential for registries and serializers that need to distinguish between types with conflicting names:

```scala
// A User from the admin domain
val adminUser = TypeId.nominal[Any]("User", Owner.fromPackagePath("com.admin"), TypeDefKind.Unknown)
// adminUser: TypeId[Any] = User
adminUser.name
// res26: String = "User"
// Notice the owner is different
adminUser.owner.asString
// res27: String = "com.admin"
// So the full names are distinct
adminUser.fullName
// res28: String = "com.admin.User"

// Compare: both have name "User" but different owners
userId.name == adminUser.name
// res29: Boolean = true
userId.owner.asString == adminUser.owner.asString
// res30: Boolean = false
```

This distinction enables type-indexed registries where you can safely store types with identical names from different sources without collision. 

#### `toString` — Idiomatic Scala Rendering

Renders the TypeId as idiomatic Scala syntax using `TypeIdPrinter`:

```scala
TypeId.of[List[Int]].toString
// res31: String = "List[Int]"
TypeId.of[Map[String, Int]].toString
// res32: String = "Map[String, Int]"
```

### Type Parameters and Arguments

Methods for inspecting generic type information.

#### `typeParams` — Formal Type Parameters

The `TypeId#typeParams` method returns the list of formal type parameters declared by a type. This is what makes a type generic. For a type like `Box[+A]`, `typeParams` captures the declaration of `A` — including its name, position in the parameter list, variance (whether it's covariant `+`, contravariant `−`, or invariant), and any bounds:

```scala
sealed trait TypeId[A <: AnyKind] {
  def typeParams: List[TypeParam]
}
```

To see how `TypeId` preserves type parameter information, we define several generic types with different variance patterns. Each demonstrates a different type parameter characteristic:

```scala

sealed trait Container[+A]
case class Box[+A](value: A) extends Container[A]

sealed trait Sink[-T]
case class Logger[-T]() extends Sink[T]

sealed trait Cache[K, +V]
case class LRUCache[K, +V](maxSize: Int) extends Cache[K, V]
```

When we derive `TypeId` for these types, we can inspect their type parameters and see the variance that was declared:

```scala
val boxId = TypeId.of[Box]
// boxId: TypeId[[A >: Nothing <: Any] =>> Box[A]] = Box[+A]
// Box declares [+A], so we see one covariant parameter
boxId.typeParams
// res34: List[TypeParam] = List(
//   TypeParam(
//     name = "A",
//     index = 0,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
boxId.typeParams.head.variance
// res35: Variance = Covariant
boxId.typeParams.head.name
// res36: String = "A"

val sinkId = TypeId.of[Sink]
// sinkId: TypeId[[T >: Nothing <: Any] =>> Sink[T]] = Sink[-T]
// Sink declares [-T], so we see one contravariant parameter
sinkId.typeParams
// res37: List[TypeParam] = List(
//   TypeParam(
//     name = "T",
//     index = 0,
//     variance = Contravariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
sinkId.typeParams.head.variance
// res38: Variance = Contravariant

val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
// Cache declares [K, +V], so we see two parameters with different variances
cacheId.typeParams
// res39: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheId.typeParams.map(p => (p.name, p.variance.symbol))
// res40: List[Tuple2[String, String]] = List(("K", ""), ("V", "+"))
```

#### `TypeParam` — Type Parameter Details

Each element in `TypeId#typeParams` is a `TypeParam` value. A type parameter defines a single formal parameter in a generic type's declaration — its name, position, variance (covariance, contravariance, invariance), bounds, and kind. When you derive a `TypeId` for a generic type like `Box[+A]`, the macro captures each declared parameter as a `TypeParam` so you can inspect them at runtime.

`TypeParam` captures these pieces of information about a type parameter:

```scala
final case class TypeParam(
  name: String,                           // "A", "T", "K", "F"
  index: Int,                             // Position: 0, 1, 2, ...
  variance: Variance = Variance.Invariant, // +, -, or none
  bounds: TypeBounds = TypeBounds.Unbounded, // >: Lower <: Upper
  kind: Kind = Kind.Type                  // *, * -> *, etc.
)
```

To inspect individual fields of a type parameter, we can extract and examine each one:

```scala

sealed trait Functor[F[_]]
```

To inspect individual fields of a type parameter, extract and examine each property:

```scala
val functorId = TypeId.of[Functor]
// functorId: TypeId[[F >: Nothing <: [_$1 >: Nothing <: Any] =>> Any] =>> Functor[F]] = Functor[F]
val paramF = functorId.typeParams.head
// paramF: TypeParam = TypeParam(
//   name = "F",
//   index = 0,
//   variance = Invariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )

paramF.name
// res42: String = "F"
paramF.index
// res43: Int = 0
paramF.variance
// res44: Variance = Invariant
paramF.isInvariant
// res45: Boolean = true
paramF.kind
// res46: Kind = Type
paramF.isTypeConstructor
// res47: Boolean = false
```

`TypeParam` provides convenience predicates for checking variance without inspecting the raw `variance` field:

```scala

sealed trait Box[+A]
sealed trait Sink[-T]
sealed trait Cache[K, +V]
```

Using these types, we can check the variance predicates to verify which parameters are covariant, contravariant, or invariant:

```scala
val boxId = TypeId.of[Box]
// boxId: TypeId[[A >: Nothing <: Any] =>> Box[A]] = Box[+A]
boxId.typeParams.head.isCovariant
// res49: Boolean = true

val sinkId = TypeId.of[Sink]
// sinkId: TypeId[[T >: Nothing <: Any] =>> Sink[T]] = Sink[-T]
sinkId.typeParams.head.isContravariant
// res50: Boolean = true

val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
val (k, v) = (cacheId.typeParams(0), cacheId.typeParams(1))
// k: TypeParam = TypeParam(
//   name = "K",
//   index = 0,
//   variance = Invariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )
// v: TypeParam = TypeParam(
//   name = "V",
//   index = 1,
//   variance = Covariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )
k.isInvariant
// res51: Boolean = true
v.isCovariant
// res52: Boolean = true
```

#### `typeArgs` — Applied Type Arguments

**Applied types** are generic types instantiated with concrete type arguments. For example, `List[Int]` is the generic `List` type constructor applied to the `Int` type argument, and `Map[String, Int]` applies two arguments to `Map`. When you derive a `TypeId` for an applied type, the `typeArgs` method returns the concrete type arguments as a list of `TypeRepr` values — allowing you to inspect what types were plugged into the type constructor.

The `typeArgs` method is essential for schema systems and code generators that need to understand the full type structure. For instance, a serializer might need to know that `List[Int]` has `Int` as its element type, or a validator might need to distinguish between `Map[String, Int]` and `Map[String, String]`:

```scala
sealed trait TypeId[A <: AnyKind] {
  def typeArgs: List[TypeRepr]
}
```

`typeArgs` returns a list of `TypeRepr` values. `TypeRepr` is an algebraic data type that represents type expressions in the Scala type system — these can be simple type references (like `Int` or `String`), complex applied types (like `List[String]`), or compound types (like `A & B`). For a detailed breakdown of all `TypeRepr` variants, see [TypeRepr — Type Expressions](#typerepr--type-expressions).

Setup some custom generic types with different type argument patterns:

```scala

case class Pair[A, B](first: A, second: B)
case class Container[T](value: T)
case class Result[E, V](error: Option[E], value: Option[V])
```

Now inspect the type arguments of various applied types:

```scala
// Simple single type argument
val containerIntId = TypeId.of[Container[Int]]
// containerIntId: TypeId[Container[Int]] = Container[Int]
containerIntId.typeArgs
// res54: List[TypeRepr] = List(Ref(Int))

// Multiple type arguments
val pairId = TypeId.of[Pair[String, Double]]
// pairId: TypeId[Pair[String, Double]] = Pair[String, Double]
pairId.typeArgs
// res55: List[TypeRepr] = List(Ref(String), Ref(Double))

// Nested applied types
val resultId = TypeId.of[Result[String, List[Int]]]
// resultId: TypeId[Result[String, List[Int]]] = Result[String, List[Int]]
resultId.typeArgs
// res56: List[TypeRepr] = List(
//   Ref(String),
//   Applied(tycon = Ref(List[+A]), args = List(Ref(Int)))
// )

// Type constructor with no arguments has empty typeArgs
TypeId.of[List].typeArgs
// res57: List[TypeRepr] = List()
```

When you access `typeArgs`, each element is a `TypeRepr` describing that argument. You can inspect them further to understand the structure:

```scala
val mapStringIntId = TypeId.of[Map[String, Int]]
// mapStringIntId: TypeId[Map[String, Int]] = Map[String, Int]
val args = mapStringIntId.typeArgs
// args: List[TypeRepr] = List(Ref(String), Ref(Int))

// First argument: String
args(0)
// res58: TypeRepr = Ref(String)

// Second argument: Int
args(1)
// res59: TypeRepr = Ref(Int)
```

For more complex types, `typeArgs` captures the full structure of the arguments, including unions, intersections, function types, and tuples:

```scala

// Union types (Scala 3)
case class Handler[T](process: T | String)

// Intersection types (Scala 3)
trait Readable { def read(): String }
trait Writable { def write(data: String): Unit }
case class Stream[T](data: T & Readable & Writable)

// Function type arguments
case class Transformer[A, B](f: A => B)

// Tuple type arguments
case class MultiValue[A, B, C](values: (A, B, C))
```

Now inspect the type arguments in these complex types:

```scala
// Union type argument
val handlerStrId = TypeId.of[Handler[Int]]
// handlerStrId: TypeId[Handler[Int]] = Handler[Int]
handlerStrId.typeArgs
// res61: List[TypeRepr] = List(Ref(Int))

// Intersection type argument
val streamId = TypeId.of[Stream[List[String]]]
// streamId: TypeId[Stream[List[String]]] = Stream[List[String]]
streamId.typeArgs
// res62: List[TypeRepr] = List(
//   Applied(tycon = Ref(List[+A]), args = List(Ref(String)))
// )

// Function type as argument
val transformerId = TypeId.of[Transformer[String, Int]]
// transformerId: TypeId[Transformer[String, Int]] = Transformer[String, Int]
transformerId.typeArgs
// res63: List[TypeRepr] = List(Ref(String), Ref(Int))

// Tuple type as argument
val multiValueId = TypeId.of[MultiValue[String, Int, Boolean]]
// multiValueId: TypeId[MultiValue[String, Int, Boolean]] = MultiValue[String, Int, Boolean]
multiValueId.typeArgs
// res64: List[TypeRepr] = List(Ref(String), Ref(Int), Ref(Boolean))
```

#### `arity` — Number of Type Parameters

The **arity** of a type is the number of formal type parameters it declares. A type with arity 0 is fully applied (a "proper type"), while arity > 0 means it's a type constructor that needs to be instantiated with type arguments. Arity is useful for generic programming and type-indexed registries where you need to distinguish between different levels of type abstraction:

```scala
sealed trait TypeId[A <: AnyKind] {
  def arity: Int
}
```

Setup some generic types with different arities:

```scala

case class Single[A](value: A)           // Arity 1
case class Pair[A, B](a: A, b: B)       // Arity 2
case class Triple[A, B, C](a: A, b: B, c: C) // Arity 3
case class Value(x: Int)                 // Arity 0
```

Check the arity of different types:

```scala
TypeId.of[Single].arity
// res66: Int = 1
TypeId.of[Pair].arity
// res67: Int = 2
TypeId.of[Triple].arity
// res68: Int = 3
TypeId.of[Value].arity
// res69: Int = 0

// Applied types have the same arity as their type constructor
TypeId.of[Single[Int]].arity
// res70: Int = 1
TypeId.of[Pair[String, Int]].arity
// res71: Int = 2
```

#### `isProperType` — Has No Type Parameters

A **proper type** (also called a ground type or monomorphic type) is a fully instantiated type with no unresolved type parameters. It's the opposite of a type constructor — you can directly instantiate values of a proper type, whereas a type constructor needs type arguments before it's usable. The `isProperType` predicate returns `true` when `arity == 0`, helping distinguish concrete types from abstract type constructors:

```scala

case class Single[A](value: A)
case class Pair[A, B](a: A, b: B)
case class Value(x: Int)
```

Check which types are proper types:

```scala
// Proper types: fully instantiated, arity == 0
TypeId.of[Value].isProperType
// res73: Boolean = true
TypeId.of[List[Int]].isProperType
// res74: Boolean = false
TypeId.of[Pair[String, Int]].isProperType
// res75: Boolean = false
TypeId.of[Int].isProperType
// res76: Boolean = true

// Type constructors: need type arguments, arity > 0
TypeId.of[Single].isProperType
// res77: Boolean = false
TypeId.of[Pair].isProperType
// res78: Boolean = false
TypeId.of[List].isProperType
// res79: Boolean = false
```

#### `isTypeConstructor` — Has Type Parameters

A **type constructor** is a parameterized type that cannot be instantiated directly — it requires concrete type arguments first. For example, `List` is a type constructor (you can't have a value of type `List`, only `List[Int]` or `List[String]`). The `isTypeConstructor` predicate returns `true` when `arity > 0`, indicating the type needs to be applied with arguments before use. This is useful for generic programming where you work with families of related types:

```scala

case class Single[A](value: A)
case class Pair[A, B](a: A, b: B)
case class Value(x: Int)
```

Identify which types are type constructors:

```scala
// Type constructors: need type arguments, arity > 0
TypeId.of[Single].isTypeConstructor
// res81: Boolean = true
TypeId.of[Pair].isTypeConstructor
// res82: Boolean = true
TypeId.of[List].isTypeConstructor
// res83: Boolean = true
TypeId.of[Map].isTypeConstructor
// res84: Boolean = true

// Proper types: fully instantiated, no type parameters
TypeId.of[Value].isTypeConstructor
// res85: Boolean = false
TypeId.of[List[Int]].isTypeConstructor
// res86: Boolean = true
TypeId.of[Int].isTypeConstructor
// res87: Boolean = false
```

#### `isApplied` — Has Type Arguments

An **applied type** is a generic type that has been instantiated with concrete type arguments. For example, `List[Int]` is an applied type (`List` applied to `Int`), while `List` by itself is a type constructor with no arguments applied. The `isApplied` predicate returns `true` when `typeArgs.nonEmpty`, helping distinguish between abstract type constructors and concrete instantiated types. This is useful for code generators that need to know whether a type is ready for use:

```scala

case class Single[A](value: A)
case class Pair[A, B](a: A, b: B)
```

Check which types are applied:

```scala
// Applied types: have type arguments
TypeId.of[List[Int]].isApplied
// res89: Boolean = true
TypeId.of[Pair[String, Int]].isApplied
// res90: Boolean = true
TypeId.of[Single[Boolean]].isApplied
// res91: Boolean = true
TypeId.of[Map[String, Double]].isApplied
// res92: Boolean = true

// Type constructors: no type arguments applied
TypeId.of[List].isApplied
// res93: Boolean = false
TypeId.of[Single].isApplied
// res94: Boolean = false
TypeId.of[Pair].isApplied
// res95: Boolean = false
TypeId.of[Map].isApplied
// res96: Boolean = false
```

### Type Classification

**Type classification** determines what kind of type definition something is — whether it's a class, trait, object, enum, alias, opaque type, or something else. This is essential for code generators, serializers, and frameworks that need to handle different type categories differently. TypeId provides both a `defKind` property that returns detailed classification information, and convenient predicates (like `isClass`, `isTrait`, `isCaseClass`) for common checks.

#### `defKind` — Type Definition Kind

Returns the `TypeDefKind` classifying this type (class, trait, object, enum, alias, opaque, etc.):

```scala
sealed trait TypeId[A <: AnyKind] {
  def defKind: TypeDefKind
}
```

Define types representing different classifications:

```scala

sealed trait Animal
case class Dog(name: String) extends Animal
case object Sentinel
type UserId = String
opaque type Email = String
enum Color { case Red; case Green; case Blue }
```

Inspect the `defKind` for each type to see how they're classified:

```scala
TypeId.of[Dog].defKind
// res98: TypeDefKind = Class(
//   isFinal = false,
//   isAbstract = false,
//   isCase = true,
//   isValue = false,
//   bases = List(Ref(Animal))
// )
TypeId.of[Animal].defKind
// res99: TypeDefKind = Trait(isSealed = true, bases = List())
TypeId.of[Sentinel.type].defKind
// res100: TypeDefKind = Object(List())
TypeId.of[UserId].defKind
// res101: TypeDefKind = TypeAlias
TypeId.of[Email].defKind
// res102: TypeDefKind = OpaqueType(TypeBounds(lower = None, upper = None))
TypeId.of[Color].defKind
// res103: TypeDefKind = Enum(List(Ref(Enum)))
```

#### Classification Predicates

Each predicate inspects `defKind` for a specific type definition kind. These are convenience methods that save you from pattern matching on `defKind` directly:

| Predicate        | Returns `true` when                                         |
|------------------|-------------------------------------------------------------|
| `isClass`        | `defKind` is `TypeDefKind.Class`                            |
| `isTrait`        | `defKind` is `TypeDefKind.Trait`                            |
| `isObject`       | `defKind` is `TypeDefKind.Object`                           |
| `isEnum`         | `defKind` is `TypeDefKind.Enum`                             |
| `isAlias`        | `defKind` is `TypeDefKind.TypeAlias`                        |
| `isOpaque`       | `defKind` is `TypeDefKind.OpaqueType`                       |
| `isAbstract`     | `defKind` is `TypeDefKind.AbstractType`                     |
| `isSealed`       | `defKind` is `TypeDefKind.Trait(isSealed = true, _)` (sealed traits only) |
| `isCaseClass`    | `defKind` is `TypeDefKind.Class(_, _, isCase = true, _, _)` |
| `isValueClass`   | `defKind` is `TypeDefKind.Class(_, _, _, isValue = true, _)`|

Use the classification predicates to identify type kinds:

```scala
val animalId   = TypeId.of[Animal]
// animalId: TypeId[Animal] = Animal
val dogId      = TypeId.of[Dog]
// dogId: TypeId[Dog] = Dog
val sentinelId = TypeId.of[Sentinel.type]
// sentinelId: TypeId[Sentinel] = Sentinel
val userIdId   = TypeId.of[UserId]
// userIdId: TypeId[UserId] = UserId
val emailId    = TypeId.of[Email]
// emailId: TypeId[Email] = Email
val colorId    = TypeId.of[Color]
// colorId: TypeId[Color] = Color

// Trait classifications
animalId.isTrait
// res104: Boolean = true
animalId.isSealed
// res105: Boolean = true

// Case class
dogId.isCaseClass
// res106: Boolean = true

// Object/Singleton
sentinelId.isObject
// res107: Boolean = true

// Type alias
userIdId.isAlias
// res108: Boolean = true

// Opaque type
emailId.isOpaque
// res109: Boolean = true

// Enum
colorId.isEnum
// res110: Boolean = true
```

:::note
`isSealed` only checks sealed traits. A sealed abstract class or sealed enum will return `false`; use `defKind` directly to inspect those cases by pattern matching on `TypeDefKind.Class(_, isAbstract = true, ...)` or `TypeDefKind.Enum(...)`.
:::

#### Semantic Predicates

**Semantic predicates** check specific semantic properties of the type after normalization, allowing you to identify built-in Scala types like tuples, products, sums, options, and either. These are useful for generic serializers and validators that treat built-in types specially.

**Normalization** resolves type aliases and opaque types to their underlying representations. For example, if you have `type UserId = String`, normalization reveals that the underlying type is `String`. Similarly, an opaque type like `opaque type Email = String` normalizes to `String`. This allows predicates like `isOption` to work correctly even when the type is wrapped in an alias or opaque type — it will look through the wrapper to find the actual semantic type.

| Predicate   | Checks                                                 |
|-------------|--------------------------------------------------------|
| `isTuple`   | Normalized type is `scala.TupleN`                      |
| `isProduct` | Normalized type is `scala.Product` or `scala.ProductN` |
| `isSum`     | Normalized type is named `Either` or `Option`          |
| `isEither`  | Normalized type is `scala.util.Either`                 |
| `isOption`  | Normalized type is `scala.Option`                      |

Check semantic properties with practical examples:

```scala
// Tuples
TypeId.of[(String, Int)].isTuple
// res111: Boolean = true
TypeId.of[(Int, String, Boolean)].isTuple
// res112: Boolean = true

// Options and Either
TypeId.of[Option[String]].isOption
// res113: Boolean = true
TypeId.of[Either[String, Int]].isEither
// res114: Boolean = true

// Products (built-in Scala Product interface, not user case classes)
TypeId.of[Product2[String, Int]].isProduct
// res115: Boolean = true
```

:::note
`isProduct` returns `true` only for Scala's built-in `scala.Product`, `scala.Product1`, etc. -- not for user-defined case classes. Use `isCaseClass` for that.
:::

Understanding the distinction between `isSum`, `isEither`, and `isOption`:

```scala

// For standard library types, use isEither and isOption
TypeId.of[Option[String]].isOption
// res116: Boolean = true
TypeId.of[Option[String]].isSum
// res117: Boolean = true

TypeId.of[Either[String, Int]].isEither
// res118: Boolean = true
TypeId.of[Either[String, Int]].isSum
// res119: Boolean = false
```

[//]: # (:::note)
[//]: # (**Practical guidance:** Always use `isEither` for `scala.util.Either` and `isOption` for `scala.Option`. The `isSum` predicate is rarely needed — it checks for hypothetical types named `"Option"` or `"Either"` placed directly in the `scala` package itself &#40;not in `scala.util` subpackage&#41;, which almost never occurs in real code.)
[//]: # (:::)

### Subtype Relationships

**Subtype relationships** determine the inheritance hierarchy and compatibility between types at runtime. This is essential for type-safe dispatch, generic programming, and validating that a value of one type can be used where another type is expected. TypeId provides methods to check direct and transitive subtyping, supertyping, type equivalence, and inspect the parent types in the hierarchy.

#### `isSubtypeOf` — Check Subtyping

Checks if this type is a subtype of another type. A type is a subtype if it extends or implements the other type, either directly or transitively. This method handles direct inheritance, sealed trait subtypes, enum cases, transitive inheritance chains, and variance-aware subtyping for applied generic types:

```scala
sealed trait TypeId[A <: AnyKind] {
  def isSubtypeOf(other: TypeId[?]): Boolean
}
```

Define a type hierarchy with direct and transitive relationships:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal
case class Fish(species: String) extends Animal
```

Check subtyping relationships:

```scala
val dogId     = TypeId.of[Dog]
// dogId: TypeId[Dog] = Dog
val mammalId  = TypeId.of[Mammal]
// mammalId: TypeId[Mammal] = Mammal
val animalId  = TypeId.of[Animal]
// animalId: TypeId[Animal] = Animal
val fishId    = TypeId.of[Fish]
// fishId: TypeId[Fish] = Fish

// Direct inheritance: Dog extends Mammal
dogId.isSubtypeOf(mammalId)
// res121: Boolean = true

// Transitive inheritance: Dog extends Mammal extends Animal
dogId.isSubtypeOf(animalId)
// res122: Boolean = true

// Not a subtype relationship
dogId.isSubtypeOf(fishId)
// res123: Boolean = false
fishId.isSubtypeOf(mammalId)
// res124: Boolean = false
```

Covariant type constructors preserve subtyping relationships:

```scala
TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Mammal]])
// res125: Boolean = true
TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Animal]])
// res126: Boolean = true
```

**Scala 3 exclusive features:** In Scala 3, `isSubtypeOf` handles advanced type relationships that Scala 2 cannot. These examples show what works in Scala 3:

```scala

// Scala 3: Enum cases
enum Color {
  case Red
  case Green
  case Blue
}

// Scala 3: Union type aliases
type StringOrInt = String | Int

// Scala 3: Intersection type aliases
trait Readable { def read(): String }
trait Writable { def write(data: String): Unit }
type ReadWrite = Readable & Writable
```

In Scala 3, `isSubtypeOf` correctly handles these advanced type cases:

```scala
// Enum cases: Red is a subtype of Color
TypeId.of[Color.Red.type].isSubtypeOf(TypeId.of[Color])
// res128: Boolean = true

// Union type aliases: String is one of the union members
TypeId.of[String].isSubtypeOf(TypeId.of[StringOrInt])
// res129: Boolean = true
TypeId.of[Int].isSubtypeOf(TypeId.of[StringOrInt])
// res130: Boolean = true

// Intersection type aliases: A type implementing both traits is a subtype
val readWriteId = TypeId.of[ReadWrite]
// readWriteId: TypeId[ReadWrite] = ReadWrite
val readableId = TypeId.of[Readable]
// readableId: TypeId[Readable] = Readable
readWriteId.isSubtypeOf(readableId)
// res131: Boolean = true
```

:::note
In Scala 2, `isSubtypeOf` does not handle `EnumCase` subtypes, types aliased to union types, or types aliased to intersection types — only the Scala 3 implementation checks those cases.
:::

#### `isSupertypeOf` — Check Supertyping

The mirror of `isSubtypeOf` — returns `true` if the other type is a subtype of this type. This is useful when you need to check if a type can accept instances of another type, or when validating that a container type can hold values of a more specific type:

```scala
sealed trait TypeId[A <: AnyKind] {
  def isSupertypeOf(other: TypeId[?]): Boolean
}
```

Check supertyping relationships using the same hierarchy:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal
case class Fish(species: String) extends Animal

val dogId     = TypeId.of[Dog]
val mammalId  = TypeId.of[Mammal]
val animalId  = TypeId.of[Animal]
val fishId    = TypeId.of[Fish]
```

Now check supertyping relationships:

```scala
// Mammal is a supertype of Dog (Mammal can hold Dog instances)
mammalId.isSupertypeOf(dogId)
// res133: Boolean = true

// Animal is a supertype of both Dog and Fish (Animal is the most general)
animalId.isSupertypeOf(dogId)
// res134: Boolean = true
animalId.isSupertypeOf(fishId)
// res135: Boolean = true

// Mammal is a supertype of Cat too
mammalId.isSupertypeOf(TypeId.of[Cat])
// res136: Boolean = true

// But Dog is not a supertype of Mammal (can't hold all Mammals as Dogs)
dogId.isSupertypeOf(mammalId)
// res137: Boolean = false

// And Fish is not a supertype of Mammal
fishId.isSupertypeOf(mammalId)
// res138: Boolean = false
```

:::note
**Limitation:** TypeId's subtyping checks currently do not handle contravariance in function types. In type theory, `Mammal => String` should be a supertype of `Dog => String` due to contravariance of input parameters, but `isSupertypeOf` returns `false` for function types with subtype relationships. For practical purposes, rely on `isSupertypeOf` for class and trait hierarchies rather than complex generic type relationships.
:::

#### `isEquivalentTo` — Check Type Equivalence

Returns `true` when two types are structurally equivalent — meaning they are **mutual subtypes** of each other. In other words, both `A.isSubtypeOf(B)` and `B.isSubtypeOf(A)` must be true. Two types are equivalent when they represent the same type through different paths, or when they normalize to the same underlying type (important for type aliases and opaque types):

```scala
sealed trait TypeId[A <: AnyKind] {
  def isEquivalentTo(other: TypeId[?]): Boolean
}
```

Check type equivalence with practical examples:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal

val dogId     = TypeId.of[Dog]
val mammalId  = TypeId.of[Mammal]
val animalId  = TypeId.of[Animal]
val catId     = TypeId.of[Cat]
```

Now check type equivalence:

```scala
// A type is always equivalent to itself
dogId.isEquivalentTo(dogId)
// res140: Boolean = true

// The same type referenced twice is equivalent
val dogId2 = TypeId.of[Dog]
// dogId2: TypeId[Dog] = Dog
dogId.isEquivalentTo(dogId2)
// res141: Boolean = true

// Different types in the hierarchy are NOT equivalent (one-way subtyping only)
dogId.isEquivalentTo(mammalId)
// res142: Boolean = false
mammalId.isEquivalentTo(animalId)
// res143: Boolean = false

// Cat and Dog are different types, even though both extend Mammal
dogId.isEquivalentTo(catId)
// res144: Boolean = false
```

Type aliases normalize to the same type, making them equivalent:

```scala

type UserId = String
type Username = String
```

Both aliases normalize to `String`, so they are equivalent:

```scala
val userIdType = TypeId.of[UserId]
// userIdType: TypeId[UserId] = UserId
val usernameType = TypeId.of[Username]
// usernameType: TypeId[Username] = Username
val stringType = TypeId.of[String]
// stringType: TypeId[String] = String

// Both type aliases are equivalent because they normalize to the same underlying type
userIdType.isEquivalentTo(usernameType)
// res146: Boolean = true

// And both are equivalent to their underlying type
userIdType.isEquivalentTo(stringType)
// res147: Boolean = true
usernameType.isEquivalentTo(stringType)
// res148: Boolean = true
```

#### `parents` — Parent Types

Returns the list of parent type representations as `TypeRepr` values, flattened across the full inheritance hierarchy. Each parent is represented as a `TypeRepr` that captures the parent type, including any type arguments it might have. This is useful for code generators, serializers, and frameworks that need to understand the inheritance structure of a type:

```scala
sealed trait TypeId[A <: AnyKind] {
  def parents: List[TypeRepr]
}
```

```scala

trait Swimmer { def swim(): Unit = () }
trait Flyer   { def fly(): Unit  = () }
trait Duck extends Swimmer with Flyer
case class MallardDuck() extends Duck
```

Parents are flattened across the full hierarchy:

```scala
// Duck extends Swimmer and Flyer directly
TypeId.of[Duck].parents
// res150: List[TypeRepr] = List(Ref(Flyer), Ref(Swimmer))

// MallardDuck extends Duck — parents include Duck, Swimmer, and Flyer
TypeId.of[MallardDuck].parents
// res151: List[TypeRepr] = List(Ref(Duck), Ref(Flyer), Ref(Swimmer))
```

### Metadata

Methods for accessing annotations, self-type, alias target, and opaque representation.

#### `annotations` — Type Annotations

Returns the list of annotations attached to this type at compile time. Each `Annotation` carries the annotation's name and its argument values, making this useful for frameworks that drive behaviour from annotations (e.g. serialization hints, validation rules, or access-control markers):

```scala
sealed trait TypeId[A <: AnyKind] {
  def annotations: List[Annotation]
}
```

```scala

@deprecated("use NewData instead", "2.0")
@transient
case class LegacyData(id: Int, payload: String)
case class Plain(x: Int)
```

A type with annotations reports each annotation by name; an unannotated type returns an empty list:

```scala
// LegacyData has two annotations
TypeId.of[LegacyData].annotations.map(_.name)
// res153: List[String] = List("transient", "deprecated")

// Plain has no annotations
TypeId.of[Plain].annotations
// res154: List[Annotation] = List()
```

#### `selfType` — Self-Type Annotation

Returns `Some(typeRepr)` when the trait declares a self-type (e.g., `trait Foo { self: Bar => ... }`), and `None` otherwise. Self-types express a dependency requirement: a trait that declares `self: Logger =>` can only be mixed into a class that also mixes in `Logger`. This method lets frameworks detect and validate those requirements at runtime:

```scala
sealed trait TypeId[A <: AnyKind] {
  def selfType: Option[TypeRepr]
}
```

```scala

trait Logger { def log(msg: String): Unit }
trait Service { self: Logger => def doWork(): Unit = log("working") }
```

`Service` requires a `Logger` to be mixed in, while `Logger` has no self-type requirement:

```scala
// Service declares a self-type dependency on Logger
TypeId.of[Service].selfType
// res156: Option[TypeRepr] = Some(Logger & Service)

// Logger has no self-type requirement
TypeId.of[Logger].selfType
// res157: Option[TypeRepr] = None
```

#### `aliasedTo` — Alias Target

Returns `Some(typeRepr)` for type aliases pointing to their underlying type, and `None` for nominal and opaque types. This lets you inspect what a type alias expands to without evaluating expressions at runtime:

```scala
sealed trait TypeId[A <: AnyKind] {
  def aliasedTo: Option[TypeRepr]
}
```

```scala

type Age = Int
type Name = String
```

A type alias resolves to its target; a concrete type returns `None`:

```scala
// Age is an alias for Int
TypeId.of[Age].aliasedTo
// res159: Option[TypeRepr] = Some(Ref(Int))

// Name is an alias for String
TypeId.of[Name].aliasedTo
// res160: Option[TypeRepr] = Some(Ref(String))

// Int is a concrete type, not an alias
TypeId.of[Int].aliasedTo
// res161: Option[TypeRepr] = None
```

#### `representation` — Opaque Type Representation

Returns `Some(typeRepr)` for opaque types revealing their underlying representation type, and `None` for all other types. Opaque types hide their implementation behind a new name, but `representation` lets frameworks such as serializers discover what the type is actually stored as:

```scala
sealed trait TypeId[A <: AnyKind] {
  def representation: Option[TypeRepr]
}
```

```scala

opaque type Email = String
opaque type UserId = Int
```

An opaque type exposes its representation; a non-opaque type returns `None`:

```scala
// Email is an opaque type backed by String
TypeId.of[Email].representation
// res163: Option[TypeRepr] = Some(Ref(String))

// UserId is an opaque type backed by Int
TypeId.of[UserId].representation
// res164: Option[TypeRepr] = Some(Ref(Int))

// Int is not opaque
TypeId.of[Int].representation
// res165: Option[TypeRepr] = None
```

### Erasure and Runtime

Methods for type erasure, runtime class lookup, and reflective construction.

#### `erased` — Erase Type Parameter

Erases the phantom type parameter, returning a `TypeId.Erased` (alias for `TypeId[TypeId.Unknown]`). This is useful when you need to store heterogeneous `TypeId` values in a collection or a type-indexed map, where the exact type parameter is unknown or irrelevant at the storage site:

```scala
sealed trait TypeId[A <: AnyKind] {
  def erased: TypeId.Erased
}
```

Different types can be stored together once erased:

```scala
val ids: List[TypeId.Erased] = List(
  TypeId.of[Int].erased,
  TypeId.of[String].erased,
  TypeId.of[Boolean].erased
)
// ids: List[Erased] = List(Int, String, Boolean)

ids.map(_.name)
// res166: List[String] = List("Int", "String", "Boolean")
```

#### `classTag` — Runtime ClassTag

Returns a `ClassTag` for this type. Returns the correct primitive `ClassTag` for Scala primitive types (`Int`, `Long`, `Boolean`, etc.) and `ClassTag.AnyRef` for all reference types. This is useful when you need to create properly-typed arrays or work with generic collections that require implicit `ClassTag` evidence at runtime:

```scala
sealed trait TypeId[A <: AnyKind] {
  lazy val classTag: scala.reflect.ClassTag[?]
}
```

On the JVM, arrays are reified — the element type is part of the array object at runtime, not erased like generics. To create an array of a generic type `T`, Scala requires a `ClassTag[T]` so the runtime knows whether to allocate a primitive array (`int[]`, `double[]`) or an object array (`Object[]`). This matters for memory efficiency: a primitive `int[]` stores 4 bytes per element unboxed, while an `Integer[]` stores heap references plus the cost of boxing each value.

`classTag` returns the correct `ClassTag` for each type:

```scala
// Primitive types have dedicated ClassTags
TypeId.of[Int].classTag
// res167: ClassTag[_ >: Nothing <: Any] = Int
TypeId.of[Double].classTag
// res168: ClassTag[_ >: Nothing <: Any] = Double

// Reference types use ClassTag.AnyRef
TypeId.of[String].classTag
// res169: ClassTag[_ >: Nothing <: Any] = Object
TypeId.of[List[Int]].classTag
// res170: ClassTag[_ >: Nothing <: Any] = Object
```

A concrete use case is a generic storage allocator that creates the right array type from a `TypeId`:

```scala

def makeStorage(size: Int, id: TypeId[?]): Array[?] =
  id.classTag.newArray(size)
```

```scala
// Creates int[] (primitive, unboxed)
makeStorage(100, TypeId.int).getClass.getComponentType
// res172: Class[_ >: Nothing <: <FromJavaObject>] = int

// Creates double[] (primitive, unboxed)
makeStorage(100, TypeId.double).getClass.getComponentType
// res173: Class[_ >: Nothing <: <FromJavaObject>] = double

// Creates Object[] (reference)
makeStorage(100, TypeId.string).getClass.getComponentType
// res174: Class[_ >: Nothing <: <FromJavaObject>] = class java.lang.Object
```

Another use case is detecting primitive types. Without `classTag`, you would need to enumerate every primitive with a chain of `isInstanceOf` checks:

```scala
// Without classTag: every primitive listed explicitly
def isPrimitive(value: Any): Boolean =
  value.isInstanceOf[Int]     ||
  value.isInstanceOf[Long]    ||
  value.isInstanceOf[Float]   ||
  value.isInstanceOf[Double]  ||
  value.isInstanceOf[Boolean] ||
  value.isInstanceOf[Byte]    ||
  value.isInstanceOf[Short]   ||
  value.isInstanceOf[Char]
```

This is fragile: if you forget one primitive (e.g. `Unit`) the check silently breaks. With `classTag` the same question reduces to a single comparison that can never miss a case — `ClassTag.AnyRef` is the universal fallback for every reference type, so anything that is not `AnyRef` must be a primitive:

```scala

def isPrimitive(id: TypeId[?]): Boolean =
  id.classTag != scala.reflect.ClassTag.AnyRef
```

```scala
isPrimitive(TypeId.of[Int])
// res176: Boolean = true
isPrimitive(TypeId.of[Double])
// res177: Boolean = true
isPrimitive(TypeId.of[Boolean])
// res178: Boolean = true
isPrimitive(TypeId.of[String])
// res179: Boolean = false
isPrimitive(TypeId.of[List[Int]])
// res180: Boolean = false
```

:::note
`classTag` returns `ClassTag.AnyRef` for all reference types. For matching or filtering by a specific reference type at runtime, use `clazz` instead.
:::

#### `clazz` — Runtime Class

Returns the runtime `Class[_]` for this type. On the JVM it returns `Some(Class[_])` for nominal and applied types, and `None` for alias and opaque types. On Scala.js it always returns `None` since JVM reflection is unavailable. This is the entry point for reflective operations such as instantiation, field access, or integration with Java libraries:

```scala
sealed trait TypeId[A <: AnyKind] {
  def clazz: Option[Class[?]]
}
```

```scala

type Age = Int
```

```scala
// Nominal and applied types return Some on the JVM
TypeId.of[String].clazz
// res182: Option[Class[_ >: Nothing <: Any]] = Some(class java.lang.String)
TypeId.of[Int].clazz
// res183: Option[Class[_ >: Nothing <: Any]] = Some(int)
TypeId.of[List[Int]].clazz
// res184: Option[Class[_ >: Nothing <: Any]] = Some(
//   class scala.collection.immutable.List
// )

// Alias types return None — the alias has no class of its own
TypeId.of[Age].clazz
// res185: Option[Class[_ >: Nothing <: Any]] = None
```

:::note
On Scala.js, `clazz` always returns `None`. Use `classTag` instead when you need cross-platform runtime type information.
:::

#### `construct` — Reflective Construction

Constructs an instance using the primary constructor on the JVM by passing constructor arguments as a `Chunk[AnyRef]`. Returns `Left` with an error message on Scala.js or when construction fails (wrong argument count, wrong types, or abstract types). Primitive values must be explicitly boxed since the argument type is `AnyRef`:

```scala
sealed trait TypeId[A <: AnyKind] {
  def construct(args: Chunk[AnyRef]): Either[String, Any]
}
```

```scala

// JVM only
case class User(name: String, age: Int)

val userId = TypeId.of[User]
```

```scala
userId.construct(Chunk("Alice", 30: Integer))
// res187: Either[String, Any] = Left(
//   "Cannot construct repl.MdocSession.MdocApp186.User: class not available"
// )
userId.construct(Chunk("Bob"))
// res188: Either[String, Any] = Left(
//   "Cannot construct repl.MdocSession.MdocApp186.User: class not available"
// )
```

##### Collection Types

Collection types accept variadic arguments representing elements. Sequence-like types (`List`, `Vector`, `Set`, `Seq`, `IndexedSeq`, `Array`, `ArraySeq`, `Chunk`) each pass a variadic sequence of elements:

```scala

```

```scala
TypeId.of[List[String]].construct(Chunk("a", "b", "c"))
// res190: Either[String, Any] = Right(List("a", "b", "c"))
TypeId.of[Vector[Int]].construct(Chunk(1: Integer, 2: Integer, 3: Integer))
// res191: Either[String, Any] = Right(Vector(1, 2, 3))
TypeId.of[Set[String]].construct(Chunk("x", "y", "z"))
// res192: Either[String, Any] = Right(Set("x", "y", "z"))
```

Map types pass interleaved key-value pairs and fail on odd argument counts:

```scala
TypeId.of[Map[String, Int]].construct(Chunk("a", 1: Integer, "b", 2: Integer))
// res193: Either[String, Any] = Right(Map("a" -> 1, "b" -> 2))
```

##### Sum Types

Sum types have special calling conventions. `Option` accepts 1 element to construct `Some(value)`, or 0 elements to construct `None`:

```scala
TypeId.of[Option[String]].construct(Chunk("hello"))
// res194: Either[String, Any] = Right(Some("hello"))
TypeId.of[Option[String]].construct(Chunk())
// res195: Either[String, Any] = Right(None)
```

`Either` requires a Boolean flag as the first argument (`true` for `Right`, `false` for `Left`), followed by the value:

```scala
TypeId.of[Either[String, Int]].construct(Chunk(true: java.lang.Boolean, 42: Integer))
// res196: Either[String, Any] = Right(Right(42))
TypeId.of[Either[String, Int]].construct(Chunk(false: java.lang.Boolean, "error"))
// res197: Either[String, Any] = Right(Left("error"))
```

### Normalization and Equality

**Normalization** resolves type aliases and opaque type representations to their underlying concrete types. For example, `type Age = Int` normalizes to `Int`, and chained aliases like `type UserId = NonEmpty; type NonEmpty = List[Int]` both resolve to `List[Int]`. Opaque types such as `opaque type UserId = String` normalize to their representation. Normalization is crucial because multiple syntactic names often refer to the same underlying type, enabling deduplication and caching strategies.

**Structural Equality** compares two TypeIds by their normalized form, treating types with identical underlying structure as equal. Importantly, opaque types preserve their semantic identity even after normalization—`TypeId.of[UserId]` where `opaque type UserId = String` remains distinct from `TypeId.of[String]` for equality purposes, preserving runtime type safety. This distinction enables type-safe registries and validators that respect opaque type boundaries.

These concepts are essential for building type-indexed registries that recognize multiple alias names as referring to the same handler, implementing serialization strategies based on normalized type structure, and enforcing opaque type safety in type-indexed maps where different opaque types wrapping the same base type should have separate validators or handlers.

#### `TypeId.normalize` — Resolve Aliases

Resolves chains of type aliases to the underlying type. For example, `type MyList = List[Int]` normalizes to `List[Int]`:

```scala
object TypeId {
  def normalize(id: TypeId[?]): TypeId[?]
}
```

```scala

type Age = Int
```

```scala
val ageId = TypeId.of[Age]
// ageId: TypeId[Age] = Age
val norm = TypeId.normalize(ageId)
// norm: TypeId[_ >: Nothing <: AnyKind] = Int
norm.fullName
// res199: String = "scala.Int"
```

#### `TypeId.structurallyEqual` — Structural Equality

Checks if two TypeIds are structurally equal after normalization. Semantically equivalent to `==` on TypeId instances; `==` additionally short-circuits on hash mismatch for performance:

```scala
object TypeId {
  def structurallyEqual(a: TypeId[?], b: TypeId[?]): Boolean
}
```

```scala

type UserId = Int
```

```scala
val a = TypeId.of[UserId]
// a: TypeId[UserId] = UserId
val b = TypeId.of[Int]
// b: TypeId[Int] = Int
TypeId.structurallyEqual(a, b)
// res201: Boolean = true
a == b
// res202: Boolean = true
```

#### `TypeId.structuralHash` — Structural Hash Code

Computes a hash code based on the normalized structural representation:

```scala
object TypeId {
  def structuralHash(id: TypeId[?]): Int
}
```

#### `TypeId.unapplied` — Strip Type Arguments

Returns the type constructor by stripping all type arguments. For example, `TypeId.unapplied(TypeId.of[List[Int]])` returns the equivalent of `TypeId.of[List]`:

```scala
object TypeId {
  def unapplied(id: TypeId[?]): TypeId[?]
}
```

```scala
val listInt = TypeId.of[List[Int]]
// listInt: TypeId[List[Int]] = List[Int]
val unapplied = TypeId.unapplied(listInt)
// unapplied: TypeId[_ >: Nothing <: AnyKind] = List[+A]
unapplied.isApplied
// res203: Boolean = false
unapplied.name
// res204: String = "List"
```

### Pattern Matching Extractors

The companion object provides extractors for pattern matching on TypeId classification:

```scala

case class User(id: Long, email: String)
val userId = TypeId.of[User]
```

```scala
userId match {
  case TypeId.Nominal(name, owner, params, defKind, parents) =>
    s"Nominal type '$name' in ${owner.asString}"
  case TypeId.Alias(name, _, _, aliased) =>
    s"Alias '$name'"
  case TypeId.Opaque(name, _, _, repr, _) =>
    s"Opaque '$name'"
}
// res206: String = "Nominal type 'User' in repl.MdocSession.MdocApp205"
```

The extractors are:

| Extractor                                               | Matches                  |
|---------------------------------------------------------|--------------------------|
| `TypeId.Nominal(name, owner, params, defKind, parents)` | Classes, traits, objects |
| `TypeId.Alias(name, owner, params, aliased)`            | Type aliases             |
| `TypeId.Opaque(name, owner, params, repr, bounds)`      | Opaque types             |
| `TypeId.Sealed(name)`                                   | Sealed traits            |
| `TypeId.Enum(name, owner)`                              | Scala 3 enums            |

## TypeDefKind Reference

`TypeDefKind` classifies every type definition. Access it via the `defKind` property documented in [Core Operations](#type-classification).

The `defKind` property (documented in [Core Operations](#type-classification)) returns one of these variants. Use classification predicates like `isCaseClass`, `isSealed`, `isObject` for simple checks.

`TypeDefKind` has these variants:

| Variant                                              | Description                                  |
|------------------------------------------------------|----------------------------------------------|
| `Class(isFinal, isAbstract, isCase, isValue, bases)` | Class definitions                            |
| `Trait(isSealed, bases)`                             | Trait definitions                            |
| `Object(bases)`                                      | Singleton objects                            |
| `Enum(bases)`                                        | Scala 3 enums                                |
| `EnumCase(parentEnum, ordinal, isObjectCase)`        | Enum cases                                   |
| `TypeAlias`                                          | Type aliases (`type Foo = Bar`)              |
| `OpaqueType(publicBounds)`                           | Opaque types                                 |
| `AbstractType`                                       | Abstract type members                        |
| `Unknown`                                            | Unclassified or unresolvable type definition |

## Type Parameters and Generics

When you derive a TypeId for a generic type, the macro captures its type parameters (variance, bounds, kind) and any applied type arguments.

A **raw type constructor** is a generic type without any type arguments filled in. For example, `List` by itself (without `[Int]` or `[String]`) is a raw type constructor. Scala 3 supports deriving TypeIds directly for raw type constructors, but Scala 2 has restrictions due to its type system:

**Scala 3** allows you to work with raw type constructors directly:

```scala
// Scala 3 only
val listId = TypeId.of[List]  // Works: raw type constructor
```

**Scala 2** requires you to use a wildcard placeholder or implicit derivation since raw type constructors are not valid syntax:

```scala
// Scala 2 alternatives
val listId = TypeId.of[List[_]]  // Use wildcard type argument

// Or retrieve via implicit derivation
implicit val derived: TypeId[List[_]] = TypeId.of[List[_]]
```

This distinction matters when you need to capture the type constructor itself (for higher-kinded type operations) rather than concrete types like `List[Int]`.

### Inspecting Type Parameters

Define your own generic types and derive their TypeIds to see how type parameters are captured:

```scala

sealed trait Container[+A]
case class Box[+A](value: A) extends Container[A]

sealed trait Cache[K, +V]
case class LRUCache[K, +V](maxSize: Int) extends Cache[K, V]
```

A single-parameter type constructor:

```scala
val containerId = TypeId.of[Container]
// containerId: TypeId[[A >: Nothing <: Any] =>> Container[A]] = Container[+A]
containerId.typeParams
// res208: List[TypeParam] = List(
//   TypeParam(
//     name = "A",
//     index = 0,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
containerId.arity
// res209: Int = 1
```

A two-parameter type constructor with mixed variance (invariant `K`, covariant `V`):

```scala
val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
cacheId.typeParams
// res210: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheId.arity
// res211: Int = 2
```

Each `TypeParam` records the parameter's name, position, variance, bounds, and kind:

```scala
val containerParam = containerId.typeParams.head
// containerParam: TypeParam = TypeParam(
//   name = "A",
//   index = 0,
//   variance = Covariant,
//   bounds = TypeBounds(lower = None, upper = None),
//   kind = Type
// )
containerParam.name
// res212: String = "A"
containerParam.variance
// res213: Variance = Covariant
containerParam.kind
// res214: Kind = Type
containerParam.isCovariant
// res215: Boolean = true
```

```scala
val cacheParams = cacheId.typeParams
// cacheParams: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheParams.map(p => (p.name, p.variance))
// res216: List[Tuple2[String, Variance]] = List(
//   ("K", Invariant),
//   ("V", Covariant)
// )
```

### Inspecting Type Arguments

When you derive a TypeId for an *applied* type (a generic type with concrete arguments), the type arguments are captured:

```scala
val boxIntId = TypeId.of[Box[Int]]
// boxIntId: TypeId[Box[Int]] = Box[Int]
boxIntId.typeArgs
// res217: List[TypeRepr] = List(Ref(Int))
boxIntId.isApplied
// res218: Boolean = true
```

```scala
val cacheStringIntId = TypeId.of[LRUCache[String, Int]]
// cacheStringIntId: TypeId[LRUCache[String, Int]] = LRUCache[String, Int]
cacheStringIntId.typeArgs
// res219: List[TypeRepr] = List(Ref(String), Ref(Int))
```

### Variance

**Variance** describes how a type parameter's subtyping relationships are preserved. **Covariant** types (`+`) preserve subtyping (if `B <: A` then `Container[B] <: Container[A]`), **contravariant** types (`-`) reverse it, and **invariant** types preserve neither. For example, `Container[+A]` is covariant—a `Container[String]` can be used where `Container[Any]` is expected. In contrast, `Cache[K, V]` where `K` is invariant means `Cache[String, Int]` cannot substitute for `Cache[Any, Int]` even if `String <: Any`.

Variance matters for type safety, polymorphism, and API design. TypeId captures variance information, enabling runtime inspection and validation:

```scala

sealed trait Container[+A]
sealed trait Cache[K, +V]
```

```scala
val containerParams = TypeId.of[Container].typeParams
// containerParams: List[TypeParam] = List(
//   TypeParam(
//     name = "A",
//     index = 0,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
containerParams.map(p => (p.name, p.variance))
// res221: List[Tuple2[String, Variance]] = List(("A", Covariant))

val cacheParams = TypeId.of[Cache].typeParams
// cacheParams: List[TypeParam] = List(
//   TypeParam(
//     name = "K",
//     index = 0,
//     variance = Invariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   ),
//   TypeParam(
//     name = "V",
//     index = 1,
//     variance = Covariant,
//     bounds = TypeBounds(lower = None, upper = None),
//     kind = Type
//   )
// )
cacheParams.map(p => (p.name, p.variance))
// res222: List[Tuple2[String, Variance]] = List(
//   ("K", Invariant),
//   ("V", Covariant)
// )
```

You can also work with variance values directly:

```scala
Variance.Covariant.symbol
// res223: String = "+"
Variance.Contravariant.symbol
// res224: String = "-"
Variance.Invariant.symbol
// res225: String = ""
Variance.Covariant.flip
// res226: Variance = Contravariant
```

### Kind

**Kind** describes the "type of a type"—it captures whether something is a concrete type or a type constructor, and how many type parameters it requires. A **proper type** like `Int` or `Box[String]` has kind `*` (zero parameters). A **unary type constructor** like `List` has kind `* -> *` (takes one type parameter). A **binary type constructor** like `Map` has kind `* -> * -> *` (takes two). **Higher-kinded types** like `Runnable[F[_]]` have kinds like `(* -> *) -> *` (takes a type constructor as a parameter). Kind information is essential for generic programming, enforcing API contracts, and enabling advanced patterns like monad transformers.

TypeId captures kind information at runtime, allowing you to inspect and validate the structure of types:

```scala

sealed trait Container[+A]
case class Box[+A](value: A) extends Container[A]

sealed trait Cache[K, +V]
case class LRUCache[K, +V](maxSize: Int) extends Cache[K, V]

trait Runnable[F[_]] {
  def run[A](fa: F[A]): A
}
```

A proper type (`*`) — fully concrete with no type parameters:

```scala
val boxIntId = TypeId.of[Box[Int]]
// boxIntId: TypeId[Box[Int]] = Box[Int]
boxIntId.isApplied
// res228: Boolean = true
boxIntId.arity
// res229: Int = 1
```

A unary type constructor (`* -> *`) — takes one type parameter:

```scala
val containerId = TypeId.of[Container]
// containerId: TypeId[[A >: Nothing <: Any] =>> Container[A]] = Container[+A]
containerId.arity
// res230: Int = 1
containerId.typeParams.map(p => (p.name, p.kind))
// res231: List[Tuple2[String, Kind]] = List(("A", Type))
```

A binary type constructor (`* -> * -> *`) — takes two type parameters:

```scala
val cacheId = TypeId.of[Cache]
// cacheId: TypeId[[K >: Nothing <: Any, V >: Nothing <: Any] =>> Cache[K, V]] = Cache[K, +V]
cacheId.arity
// res232: Int = 2
cacheId.typeParams.map(p => (p.name, p.kind))
// res233: List[Tuple2[String, Kind]] = List(("K", Type), ("V", Type))
```

A higher-kinded type (`(* -> *) -> *`) — a type parameter that itself is a type constructor:

```scala
val runnableId = TypeId.of[Runnable]
// runnableId: TypeId[[F >: Nothing <: [_$4 >: Nothing <: Any] =>> Any] =>> Runnable[F]] = Runnable[F]
runnableId.typeParams.head.kind
// res234: Kind = Type
runnableId.typeParams.head.kind.arity
// res235: Int = 0
```

| Kind                      | Notation        | Arity | Examples              |
|---------------------------|-----------------|-------|-----------------------|
| `Kind.Type` / `Kind.Star` | `*`             | 0     | `Int`, `Box[Int]`     |
| `Kind.Star1`              | `* -> *`        | 1     | `Container`, `Option` |
| `Kind.Star2`              | `* -> * -> *`   | 2     | `Cache`, `Either`     |
| `Kind.HigherStar1`        | `(* -> *) -> *` | 1     | `Runnable`            |

## Subtype Relationships

**Subtype relationships** determine if one type is a subtype of another, enabling type-safe polymorphism and dispatch at runtime. This is essential for checking if a value of one type can be used where another type is expected. TypeId handles direct inheritance, transitive inheritance chains, sealed trait cases, and variance-aware subtyping for generic types.

Three key methods work together to express the full range of type relationships:

```scala

sealed trait Animal
sealed trait Mammal extends Animal
case class Dog(name: String) extends Mammal
case class Cat(name: String) extends Mammal
case class Fish(species: String) extends Animal
```

**`isSubtypeOf`** checks if this type is a subtype of another (direct or transitive):

```scala
val dogId     = TypeId.of[Dog]
// dogId: TypeId[Dog] = Dog
val mammalId  = TypeId.of[Mammal]
// mammalId: TypeId[Mammal] = Mammal
val animalId  = TypeId.of[Animal]
// animalId: TypeId[Animal] = Animal
val fishId    = TypeId.of[Fish]
// fishId: TypeId[Fish] = Fish

// Direct inheritance: Dog extends Mammal
dogId.isSubtypeOf(mammalId)
// res237: Boolean = true

// Transitive inheritance: Dog extends Mammal extends Animal
dogId.isSubtypeOf(animalId)
// res238: Boolean = true

// Not a subtype relationship
dogId.isSubtypeOf(fishId)
// res239: Boolean = false
fishId.isSubtypeOf(mammalId)
// res240: Boolean = false
```

**`isSupertypeOf`** is the reverse—checks if this type is a supertype (parent) of another:

```scala
mammalId.isSupertypeOf(dogId)
// res241: Boolean = true
animalId.isSupertypeOf(dogId)
// res242: Boolean = true
dogId.isSupertypeOf(animalId)
// res243: Boolean = false
```

**`isEquivalentTo`** checks if two types are exactly the same:

```scala
dogId.isEquivalentTo(dogId)
// res244: Boolean = true
dogId.isEquivalentTo(mammalId)
// res245: Boolean = false
animalId.isEquivalentTo(animalId)
// res246: Boolean = true
```

**Variance-aware subtyping** for generic types respects covariance and contravariance. Covariant type constructors like `List[+A]` preserve subtyping relationships:

```scala
val listDogId    = TypeId.of[List[Dog]]
// listDogId: TypeId[List[Dog]] = List[Dog]
val listMammalId = TypeId.of[List[Mammal]]
// listMammalId: TypeId[List[Mammal]] = List[Mammal]
val listAnimalId = TypeId.of[List[Animal]]
// listAnimalId: TypeId[List[Animal]] = List[Animal]

listDogId.isSubtypeOf(listMammalId)
// res247: Boolean = true
listDogId.isSubtypeOf(listAnimalId)
// res248: Boolean = true
listMammalId.isSubtypeOf(listAnimalId)
// res249: Boolean = true
```

These methods are essential for building type-safe registries, implementing generic serializers that dispatch based on type hierarchy, and validating API contracts that require specific type relationships.

## Annotations

**Annotations** are metadata attached to types at compile time. TypeId captures them at runtime, making this metadata available for introspection, validation, and dispatch logic. Annotations enable building smart serializers, validators, and code generators that adjust their behavior based on type-level metadata.

TypeId exposes each annotation as an `Annotation` object containing the annotation's type and its arguments. This is essential for frameworks that need to read compile-time metadata (like JPA, validation libraries, or custom serialization frameworks) but want to remain generic and support multiple annotation schemes:

```scala

@transient
case class ImportantData(id: Int, payload: String)

case class Plain(x: Int)
```

Inspect annotations on a type:

```scala
val importantId = TypeId.of[ImportantData]
// importantId: TypeId[ImportantData] = ImportantData
importantId.annotations
// res251: List[Annotation] = List(
//   Annotation(typeId = transient, args = List())
// )
importantId.annotations.map(_.name)
// res252: List[String] = List("transient")

TypeId.of[Plain].annotations
// res253: List[Annotation] = List()
```

Annotations can have arguments and parameters. Create a custom annotation to see how arguments are captured:

```scala

// Custom annotation with parameters
case class ApiEndpoint(version: Int, deprecated: Boolean = false) extends scala.annotation.StaticAnnotation

@ApiEndpoint(version = 2, deprecated = true)
case class UserV2(id: String, name: String)
```

Derive the TypeId and inspect annotation arguments:

```scala
val userV2Id = TypeId.of[UserV2]
// userV2Id: TypeId[UserV2] = UserV2
userV2Id.annotations
// res255: List[Annotation] = List(
//   Annotation(
//     typeId = ApiEndpoint,
//     args = List(
//       Named(name = "version", value = Const(2)),
//       Named(name = "deprecated", value = Const(true))
//     )
//   )
// )
userV2Id.annotations.head.args
// res256: List[AnnotationArg] = List(
//   Named(name = "version", value = Const(2)),
//   Named(name = "deprecated", value = Const(true))
// )
```

Annotations are represented internally as instances of the `Annotation` data class. The `Annotation` contains the annotation's `TypeId` and a list of `AnnotationArg` values representing the arguments:

| Type                                           | Description                                        |
|------------------------------------------------|----------------------------------------------------|
| `Annotation(typeId, args)`                     | An annotation instance with its type and arguments |
| `AnnotationArg.Const(value)`                   | A constant value argument                          |
| `AnnotationArg.Named(name, value)`             | A named parameter                                  |
| `AnnotationArg.ArrayArg(values)`               | An array of arguments                              |
| `AnnotationArg.Nested(annotation)`             | A nested annotation                                |
| `AnnotationArg.ClassOf(tpe)`                   | A `classOf[T]` argument                            |
| `AnnotationArg.EnumValue(enumType, valueName)` | An enum constant                                   |

**Use cases:** Annotations are commonly used to drive serialization strategies, enforce validation rules, mark types for code generation, configure persistence metadata, or enable framework-specific behavior without requiring explicit configuration objects.

## Namespaces and Owners

Every type has an `owner` — the hierarchical path that tells you where the type is defined (its package, enclosing object, or enclosing type). Owner is essential when you need to identify types by their origin, filter schemas by namespace, or build type-indexed registries that respect module boundaries.

When building cross-module systems (middleware, gateways, plugin registries, code generators), you often need to distinguish between types from different packages or modules. For example, you might want to:
- Apply different serialization strategies to types from `com.internal.domain` vs `com.external.api`
- Build a type registry keyed by both type name *and* origin package (to handle name collisions across modules)
- Validate that a deserialized type comes from a trusted package

Owner gives you the tools to make these decisions at runtime.

### Inspecting Owners

When you derive a TypeId, the `owner` property captures where the type is defined:

```scala

case class MyType(x: Int)
```

```scala
val myId = TypeId.of[MyType]
// myId: TypeId[MyType] = MyType
myId.owner
// res258: Owner = Owner(
//   List(Package("repl"), Term("MdocSession"), Type("MdocApp257"))
// )
myId.owner.asString
// res259: String = "repl.MdocSession.MdocApp257"
myId.fullName
// res260: String = "repl.MdocSession.MdocApp257.MyType"
```

Owner provides methods to inspect and navigate the hierarchy:

```scala
myId.owner.parent
// res261: Owner = Owner(List(Package("repl"), Term("MdocSession")))
myId.owner.lastName
// res262: String = "MdocApp257"
myId.owner.isRoot
// res263: Boolean = false
```

### Owner Structure: Packages, Terms, and Types

An Owner is a chain of segments: packages, terms (objects/values), and types. For a type defined as:

```scala
package com.example
object Outer {
  class Inner
}
```

The owner of `Inner` has three segments: `com`, `example` (packages), and `Outer` (term):

```scala

object ExampleModule {
  case class Config(timeout: Int)
}
```

```scala
val configId = TypeId.of[ExampleModule.Config]
// configId: TypeId[Config] = Config
configId.owner.asString
// res265: String = "repl.MdocSession.MdocApp264.ExampleModule"
```

### TermPath

`TermPath` represents paths to term values and is used in TypeRepr expressions for singleton types (like `obj.field.type`). Singleton type information exists at compile time but is **erased at runtime** by the JVM — both `TypeId.of[HttpStatus.OK.type]` and `TypeId.of[HttpStatus.NotFound.type]` resolve to the same underlying `Int` TypeId. TermPath is useful in type representation structures and code generators that need to capture the compile-time singleton distinction for metaprogramming.

When the macro encounters a singleton type (a `TermRef` in Scala's reflection API), it recursively walks the qualifier chain to build the term path and stores it as `TypeRepr.Singleton(path)` for use in type expressions.

Derive TypeIds for singleton values to see them resolve to their underlying type:

```scala

object HttpStatus {
  val OK = 200
  val NotFound = 404
}
```

```scala
val okSingletonId = TypeId.of[HttpStatus.OK.type]
// okSingletonId: TypeId[OK] = Int
okSingletonId.name
// res267: String = "Int"

val notFoundSingletonId = TypeId.of[HttpStatus.NotFound.type]
// notFoundSingletonId: TypeId[NotFound] = Int
notFoundSingletonId.name
// res268: String = "Int"

okSingletonId == notFoundSingletonId
// res269: Boolean = true
```

**When to use TermPath:** In TypeRepr expressions and code generators that need to represent the compile-time path to a value. While singleton types are erased at runtime, TermPath captures this distinction for reflection and metaprogramming scenarios.

## TypeRepr — Type Expressions

`TypeRepr` represents type expressions in the Scala type system. This is fundamentally different from `TypeId`: while `TypeId` identifies a specific type *definition* (like `List` as a class or `Person` as a case class), `TypeRepr` represents how types are *composed and expressed* at runtime — as type arguments, parent types, intersections, unions, functions, and more.

**The Key Distinction:**

A `TypeId` answers the question **"What is this type definition?"** — for example, `TypeId.of[List]` gives you metadata about the `List` class itself. But a `TypeRepr` answers **"How is this type used in context?"** — for example, when you inspect `TypeId.of[List[Int]].typeArgs`, you get a `TypeRepr.Applied(Ref(TypeId.list), List(Ref(TypeId.int)))`, which describes that `List` is applied to the type argument `Int`.

**Practical Examples of the Difference:**

- `TypeId.list` identifies the `List` class definition
- `TypeRepr.Ref(TypeId.list)` is the expression "use List as a type" (standalone)
- `TypeRepr.Applied(TypeId.list, args)` is the expression "List applied to type arguments" (e.g., `List[Int]`)
- `TypeRepr.Function` represents `(A, B) => C` as a first-class type expression (not a method signature)
- `TypeRepr.Union` represents `A | B` without requiring a union type definition to exist

TypeRepr variants like `Intersection`, `Tuple`, `Union`, `TypeLambda`, and `ContextFunction` allow you to represent type expressions that may not have their own `TypeId` definitions — they are *computed expressions* in the type system rather than named definitions.

You encounter `TypeRepr` values when inspecting `typeArgs`, parent types in `defKind`, and alias targets:

```scala

```

When you derive a TypeId for an applied type, the `typeArgs` are `TypeRepr` values representing the type arguments:

```scala
TypeId.of[Int & String].typeArgs
// res271: List[TypeRepr] = List()
TypeId.of[Map[String, Int]].typeArgs
// res272: List[TypeRepr] = List(Ref(String), Ref(Int))
```

Here is a reference of the different `TypeRepr` variants you may encounter when inspecting TypeIds:

| Category     | Variant                                          | Example                                     |
|--------------|--------------------------------------------------|---------------------------------------------|
| **Common**   | `Ref(id)`                                        | `Int`, `String` — reference to a named type |
|              | `ParamRef(param, depth)`                         | `A` — reference to a type parameter         |
|              | `Applied(tycon, args)`                           | `List[Int]` — parameterized type            |
| **Compound** | `Intersection(types)`                            | `A & B` (Scala 3) or `A with B` (Scala 2)   |
|              | `Union(types)`                                   | `A \| B` (Scala 3)                          |
|              | `Tuple(elems)`                                   | `(A, B, C)`, named tuples                   |
|              | `Function(params, result)`                       | `(A, B) => C`                               |
|              | `ContextFunction(params, result)`                | `(A, B) ?=> C` (Scala 3)                    |
| **Special**  | `Singleton(path)`                                | `x.type`                                    |
|              | `ThisType(owner)`                                | `this.type`                                 |
|              | `TypeProjection(qualifier, name)`                | `Outer#Inner`                               |
|              | `TypeSelect(qualifier, name)`                    | `qual.Member`                               |
|              | `Structural(parents, members)`                   | `{ def foo: Int }`                          |
| **Advanced** | `TypeLambda(params, body)`                       | `[X] =>> F[X]`                              |
|              | `Wildcard(bounds)`                               | `?`, `? <: Upper`                           |
|              | `ByName(underlying)`                             | `=> A`                                      |
|              | `Repeated(element)`                              | `A*`                                        |
|              | `Annotated(underlying, annotations)`             | `A @anno`                                   |
|              | `Constant.*`                                     | `42`, `"foo"`, `true` (literal types)       |
| **Builtins** | `AnyType`, `NothingType`, `NullType`, `UnitType` | Special types                               |

## Erased TypeId

For type-indexed collections where the type parameter doesn't matter, erase it:

```scala

```

```scala
val erased: TypeId.Erased = TypeId.of[Int].erased
// erased: TypeId[Unknown] = Int
erased
// res274: TypeId[Unknown] = Int
```

Erased TypeIds are the key to building type-indexed maps:

```scala
val registry: Map[TypeId.Erased, String] = Map(
  TypeId.of[Int].erased    -> "Integer type",
  TypeId.of[String].erased -> "String type"
)
// registry: Map[Erased, String] = Map(
//   Int -> "Integer type",
//   String -> "String type"
// )

registry.get(TypeId.of[Int].erased)
// res275: Option[String] = Some("Integer type")
registry.get(TypeId.of[Double].erased)
// res276: Option[String] = None
```

## Predefined TypeIds

TypeId provides instances for common types:

**Core Interfaces:** `TypeId.charSequence` (`java.lang`), `comparable` (`java.lang`), `serializable` (`java.io`)

**Primitives:** `TypeId.unit`, `boolean`, `byte`, `short`, `int`, `long`, `float`, `double`, `char`, `string`, `bigInt`, `bigDecimal`

**Collections:** `TypeId.option`, `some`, `none`, `list`, `vector`, `set`, `seq`, `indexedSeq`, `map`, `either`, `array`, `arraySeq`, `chunk`

**java.time:** `TypeId.dayOfWeek`, `duration`, `instant`, `localDate`, `localDateTime`, `localTime`, `month`, `monthDay`, `offsetDateTime`, `offsetTime`, `period`, `year`, `yearMonth`, `zoneId`, `zoneOffset`, `zonedDateTime`

**java.util:** `TypeId.currency`, `uuid`

**Scala 3 only:** `TypeId.iarray` — `IArray[T]`, the immutable array type.

## Integration with Schema

TypeId is central to ZIO Blocks' schema system. Every `Reflect` node carries an associated TypeId:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}
```

You can access the TypeId from a schema's reflection:

```scala
val reflect = Schema[Person].reflect
val typeId = reflect.typeId
typeId.name
typeId.isCaseClass
```

### Schema Transformations

TypeId is automatically attached when transforming schemas. The `transform` method takes an implicit `TypeId[B]` parameter, so the TypeId for the target type is resolved at compile time:

```scala
case class Email(value: String)

object Email {
  implicit val schema: Schema[Email] = Schema[String]
    .transform(Email(_), _.value)
    // TypeId[Email] is resolved implicitly — no extra call needed
}
```

### Schema Derivation

The `Deriver` trait receives a `TypeId` for each node in the schema. Methods like `deriveRecord` and `deriveVariant` include a `typeId: TypeId[A]` parameter alongside fields/cases, bindings, documentation, modifiers, and more. This lets you inspect the type's structure, annotations, and relationships when generating code.

For details on the full `Deriver` API and how to implement custom derivers, see the [Type Class Derivation](./type-class-derivation.md) reference.

## Comparison with Alternatives

TypeId occupies a different niche from the reflection and type-tagging mechanisms in the Scala ecosystem:

| Feature                           | `TypeId` | `ClassTag` | `TypeTag` (Scala 2) | `TypeTest` (Scala 3) | `Mirror` (Scala 3) |
|-----------------------------------|----------|------------|---------------------|----------------------|--------------------|
| Preserves generic type args       | Yes      | No         | Yes                 | No                   | No                 |
| Distinguishes opaque types        | Yes      | No         | No                  | No                   | No                 |
| Available on Scala.js             | Yes      | Partial    | No                  | Yes                  | Yes                |
| Cross-version (2 & 3)             | Yes      | Yes        | Scala 2 only        | Scala 3 only         | Scala 3 only       |
| Pure data (no runtime reflection) | Yes      | No         | No                  | No                   | Yes                |
| Captures annotations              | Yes      | No         | Yes                 | No                   | No                 |
| Captures variance & kind          | Yes      | No         | Yes                 | No                   | No                 |
| Subtype relationship checks       | Yes      | No         | Yes                 | Yes                  | No                 |

**When to migrate from `ClassTag`:** If you only need `ClassTag` to create arrays of the correct runtime type, keep using it — TypeId does not replace that functionality. If you are using `ClassTag` to identify or dispatch on types, TypeId provides strictly more information (generics, opaque types, annotations) and works identically on JVM and Scala.js.

**When to migrate from `TypeTag` / `WeakTypeTag`:** These are Scala 2-only, depend on `scala-reflect`, and are not available on Scala.js. TypeId captures comparable metadata (full name, type arguments, variance, annotations) as a pure data structure without runtime reflection, and works across Scala 2, Scala 3, JVM, and Scala.js.

**When to migrate from `TypeTest`:** `TypeTest` is a Scala 3 mechanism for safe pattern matching on types. It answers "is this value an instance of T?" but does not expose type structure, annotations, or generic arguments. Use TypeId when you need to inspect or serialize type metadata, not just test membership.

**When to migrate from `Mirror`:** `Mirror` provides structural information about products and sums for derivation in Scala 3. TypeId complements `Mirror` by adding namespace information (owner/package), annotations, opaque type support, and cross-version compatibility. In ZIO Blocks, the schema derivation system uses TypeId rather than `Mirror`.

## Running the Examples

All code from this guide is available as runnable examples in the `schema-examples` module.

**1. Clone the repository and navigate to the project:**

```bash
git clone https://github.com/zio/zio-blocks.git
cd zio-blocks
```

**2. Run individual examples with sbt:**

### Basic Usage

Demonstrates deriving TypeIds for case classes, accessing their properties (name, fullName, owner, arity), using predefined TypeIds for built-in types, and implicit derivation:

```scala title="schema-examples/src/main/scala/typeid/TypeIdBasicExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package typeid

/**
 * TypeId Basic Example
 *
 * Demonstrates deriving TypeIds for case classes, accessing their properties,
 * and using predefined TypeIds for built-in types.
 *
 * Run with: sbt "schema-examples/runMain typeid.TypeIdBasicExample"
 */

object TypeIdBasicExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("TypeId Basic Example")
  println("═══════════════════════════════════════════════════════════════\n")

  // Derive TypeId for a case class
  case class Person(name: String, age: Int)

  // Simple derivation
  val personId = TypeId.of[Person]

  println("--- Deriving TypeId for Person ---\n")

  // Accessing name and fullName
  println("TypeId.of[Person].name")
  show(personId.name)

  // Accessing fully qualified name
  println("TypeId.of[Person].fullName")
  show(personId.fullName)

  // Checking if it's a case class
  println("TypeId.of[Person].isCaseClass")
  show(personId.isCaseClass)

  // Accessing the owner (package/enclosing type)
  println("TypeId.of[Person].owner")
  show(personId.owner)

  println("\n--- Type Classification ---\n")

  // Check the kind (arity)
  println("TypeId.of[Person].arity")
  show(personId.arity)

  // Check it's a proper type
  println("TypeId.of[Person].isProperType")
  show(personId.isProperType)

  // Check it's not a type constructor
  println("TypeId.of[Person].isTypeConstructor")
  show(personId.isTypeConstructor)

  println("\n--- Predefined TypeIds ---\n")

  // Use predefined TypeIds for built-in types
  println("TypeId.int.fullName")
  show(TypeId.int.fullName)

  println("TypeId.string.fullName")
  show(TypeId.string.fullName)

  println("TypeId.list.name")
  show(TypeId.list.name)

  // List is a type constructor (arity = 1)
  println("TypeId.list.arity")
  show(TypeId.list.arity)

  println("TypeId.map.arity")
  show(TypeId.map.arity)

  println("\n--- Classification Predicates ---\n")

  // Check various classification predicates
  println("TypeId.option.isTypeConstructor")
  show(TypeId.option.isTypeConstructor)

  println("TypeId.either.arity")
  show(TypeId.either.arity)

  println("\n--- Implicit Derivation ---\n")

  // Implicit derivation
  val personIdImplicit: TypeId[Person] = implicitly[TypeId[Person]]

  println("implicitly[TypeId[Person]].name")
  show(personIdImplicit.name)

  println("\n═══════════════════════════════════════════════════════════════")

  show(
    TypeId.of[scala.util.Either].isSum
  )

  show(
    TypeId.of[Option[String]].isOption
  )
  show(
    TypeId.of[Either[String, Int]].isEither
  )

  show(
    TypeId.of[Option[String]].isSum
  )

  show(
    TypeId.of[Either[String, Int]].isSum
  )

  show(
    TypeId.of[(Int, String)].isProduct
  )

  show(
    TypeId.of[Tuple2[Int, String]].isProduct
  )

  show(
    TypeId.of[Product2[Int, String]].isProduct
  )

  show(
    TypeId.of[(Int, String, Boolean)].isTuple
  )

  sealed trait Animal

  sealed trait Mammal extends Animal

  case class Dog(name: String) extends Mammal

  case class Cat(name: String) extends Mammal

  case class Fish(species: String) extends Animal

  val dogId    = TypeId.of[Dog]
  val mammalId = TypeId.of[Mammal]
  val animalId = TypeId.of[Animal]
  val fishId   = TypeId.of[Fish]

  // Direct inheritance: Dog extends Mammal
  dogId.isSubtypeOf(mammalId)

  // Transitive inheritance: Dog extends Mammal extends Animal
  dogId.isSubtypeOf(animalId)

  // Not a subtype relationship
  dogId.isSubtypeOf(fishId)
  fishId.isSubtypeOf(mammalId)

  TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Mammal]])
  TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Animal]])

  show(
    TypeId.of[Mammal => String].isSupertypeOf(TypeId.of[Dog => String])
  )

  // A generic parent type with concrete type arguments
  case class StringList() extends scala.collection.mutable.ListBuffer[String]

  // A type with multiple generic parents
  case class Entry[K, V](key: K, value: V) extends scala.collection.Map[K, V] {
    def iterator = Iterator((key, value))

    def get(k: K) = if (k == key) Some(value) else None

    override def -(key: K): collection.Map[K, V] = ???

    override def -(key1: K, key2: K, keys: K*): collection.Map[K, V] = ???
  }

  val stringListId = TypeId.of[StringList]
  // StringList extends ListBuffer[String] - the type argument is captured
  show(
    stringListId.parents
  )

  val entryId = TypeId.of[Entry[String, Int]]
  // Entry[String, Int] extends Map[String, Int] - type arguments are preserved

  show(
    entryId.parents
  )

  // A trait can extend multiple traits
  trait Swimmer {
    def swim(): Unit = ()
  }

  trait Flyer {
    def fly(): Unit = ()
  }

  trait Duck extends Swimmer with Flyer

  // A case class can extend a trait
  case class MallardDuck() extends Duck

  show(
    TypeId.of[MallardDuck].parents
  )

  def isPrimitive(id: TypeId[?]): Boolean =
    id.classTag != scala.reflect.ClassTag.AnyRef

  show {
    isPrimitive(TypeId.of[Int])
    isPrimitive(TypeId.of[Double])
    isPrimitive(TypeId.of[Boolean])
    isPrimitive(TypeId.of[String])
    isPrimitive(TypeId.of[List[Int]])
  }

  def makeStorage(size: Int, id: TypeId[?]): Array[?] =
    id.classTag.newArray(size)

  show {
    makeStorage(100, TypeId.int).getClass.getComponentType
    makeStorage(100, TypeId.double).getClass.getComponentType
    makeStorage(100, TypeId.string).getClass.getComponentType
  }

}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/TypeIdBasicExample.scala))

```bash
sbt "schema-examples/runMain typeid.TypeIdBasicExample"
```

### Subtype Relationships

Demonstrates subtype checking with `isSubtypeOf`, `isSupertypeOf`, and `isEquivalentTo`, including direct inheritance, transitive inheritance, sealed trait cases, and variance-aware subtyping for applied types like `List[Dog] <: List[Animal]`:

```scala title="schema-examples/src/main/scala/typeid/TypeIdSubtypingExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package typeid

/**
 * TypeId Subtyping Example
 *
 * Demonstrates subtype relationships, checking inheritance, and variance-aware
 * subtyping for applied types.
 *
 * Run with: sbt "schema-examples/runMain typeid.TypeIdSubtypingExample"
 */

object TypeIdSubtypingExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("TypeId Subtyping Example")
  println("═══════════════════════════════════════════════════════════════\n")

  // Define a sealed trait hierarchy
  sealed trait Animal
  case class Dog(name: String) extends Animal
  case class Cat(name: String) extends Animal
  case object Bird             extends Animal

  println("--- Direct Inheritance ---\n")

  val dogId    = TypeId.of[Dog]
  val animalId = TypeId.of[Animal]

  // Check if Dog is a subtype of Animal
  println("TypeId.of[Dog].isSubtypeOf(TypeId.of[Animal])")
  show(dogId.isSubtypeOf(animalId))

  // Check if Animal is a supertype of Dog
  println("TypeId.of[Animal].isSupertypeOf(TypeId.of[Dog])")
  show(animalId.isSupertypeOf(dogId))

  // Check equivalence
  println("TypeId.of[Dog].isEquivalentTo(TypeId.of[Dog])")
  show(dogId.isEquivalentTo(dogId))

  println("TypeId.of[Dog].isEquivalentTo(TypeId.of[Animal])")
  show(dogId.isEquivalentTo(animalId))

  println("\n--- Transitive Inheritance ---\n")

  val catId = TypeId.of[Cat]

  // Both Dog and Cat are subtypes of Animal
  println("TypeId.of[Cat].isSubtypeOf(TypeId.of[Animal])")
  show(catId.isSubtypeOf(animalId))

  println("TypeId.of[Dog].isSubtypeOf(TypeId.of[Animal]) && TypeId.of[Cat].isSubtypeOf(TypeId.of[Animal])")
  show(dogId.isSubtypeOf(animalId) && catId.isSubtypeOf(animalId))

  println("\n--- Sealed Trait Cases ---\n")

  val birdId = TypeId.of[Bird.type]

  println("TypeId.of[Bird.type].isSubtypeOf(TypeId.of[Animal])")
  show(birdId.isSubtypeOf(animalId))

  println("\n--- Applied Type Subtyping (Covariance) ---\n")

  // List is covariant in its type parameter
  val listDogId    = TypeId.of[List[Dog]]
  val listAnimalId = TypeId.of[List[Animal]]

  // Due to covariance, List[Dog] is a subtype of List[Animal]
  println("TypeId.of[List[Dog]].isSubtypeOf(TypeId.of[List[Animal]])")
  show(listDogId.isSubtypeOf(listAnimalId))

  println("\n--- Applied Type Non-Subtyping (Invariance) ---\n")

  // Array is invariant in its type parameter (on Scala runtime)
  val arrayDogId    = TypeId.of[Array[Dog]]
  val arrayAnimalId = TypeId.of[Array[Animal]]

  println("TypeId.of[Array[Dog]].isSubtypeOf(TypeId.of[Array[Animal]])")
  show(arrayDogId.isSubtypeOf(arrayAnimalId))

  println("\n--- Type Parameters ---\n")

  // Check the type parameters of a generic type
  println("TypeId.of[List[Dog]].typeArgs")
  show(listDogId.typeArgs)

  println("TypeId.of[Map[String, Animal]].typeArgs")
  show(TypeId.of[Map[String, Animal]].typeArgs)

  println("\n═══════════════════════════════════════════════════════════════")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/TypeIdSubtypingExample.scala))

```bash
sbt "schema-examples/runMain typeid.TypeIdSubtypingExample"
```

### Normalization and Registries

Demonstrates type alias handling, normalization to underlying types, structural equality, and building type-indexed registries using erased TypeIds:

```scala title="schema-examples/src/main/scala/typeid/TypeIdNormalizationExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package typeid

/**
 * TypeId Normalization Example
 *
 * Demonstrates type alias handling, normalization, and type-indexed registries
 * using erased TypeIds.
 *
 * Run with: sbt "schema-examples/runMain typeid.TypeIdNormalizationExample"
 */

object TypeIdNormalizationExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("TypeId Normalization Example")
  println("═══════════════════════════════════════════════════════════════\n")

  // Define some type aliases
  type UserId = Long
  type Email  = String
  type Age    = Int

  println("--- Type Aliases ---\n")

  // Create TypeIds for the aliases
  val userIdAlias = TypeId.alias[UserId](
    name = "UserId",
    owner = Owner.Root,
    typeParams = Nil,
    aliased = TypeRepr.Ref(TypeId.long)
  )

  println("userIdAlias.name")
  show(userIdAlias.name)

  println("userIdAlias.fullName")
  show(userIdAlias.fullName)

  println("\n--- Normalization ---\n")

  // Normalize the alias to its underlying type
  val normalized = TypeId.normalize(userIdAlias)

  println("TypeId.normalize(userIdAlias).name")
  show(normalized.name)

  println("TypeId.normalize(userIdAlias).fullName")
  show(normalized.fullName)

  println("\n--- Equality with Normalization ---\n")

  // Structural equality (not considering aliases)
  val anotherUserIdAlias = TypeId.alias[UserId](
    name = "UserId",
    owner = Owner.Root,
    typeParams = Nil,
    aliased = TypeRepr.Ref(TypeId.long)
  )

  println("userIdAlias == anotherUserIdAlias")
  show(userIdAlias == anotherUserIdAlias)

  println("TypeId.normalize(userIdAlias) == TypeId.long")
  show(TypeId.normalize(userIdAlias) == TypeId.long)

  println("\n--- Erased TypeIds for Registries ---\n")

  // Erased TypeIds are useful for type-indexed maps
  val intErased: TypeId.Erased     = TypeId.int.erased
  val stringErased: TypeId.Erased  = TypeId.string.erased
  val listIntErased: TypeId.Erased = TypeId.of[List[Int]].erased

  println("TypeId.int.erased")
  show(intErased)

  println("TypeId.string.erased")
  show(stringErased)

  println("TypeId.of[List[Int]].erased")
  show(listIntErased)

  println("\n--- Type Registry Using Erased TypeIds ---\n")

  // Build a type registry
  val registry: Map[TypeId.Erased, String] = Map(
    TypeId.int.erased           -> "Integer type",
    TypeId.string.erased        -> "String type",
    TypeId.long.erased          -> "Long type",
    TypeId.of[List[Int]].erased -> "List of integers"
  )

  println("registry.get(TypeId.int.erased)")
  show(registry.get(TypeId.int.erased))

  println("registry.get(TypeId.string.erased)")
  show(registry.get(TypeId.string.erased))

  println("registry.get(TypeId.of[List[Int]].erased)")
  show(registry.get(TypeId.of[List[Int]].erased))

  println("\n--- Querying the Registry ---\n")

  // Look up types in the registry
  val intType    = TypeId.of[Int].erased
  val doubleType = TypeId.of[Double].erased

  println("registry.get(intType)")
  show(registry.get(intType))

  println("registry.get(doubleType)")
  show(registry.get(doubleType))

  println("\n═══════════════════════════════════════════════════════════════")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/TypeIdNormalizationExample.scala))

```bash
sbt "schema-examples/runMain typeid.TypeIdNormalizationExample"
```

### Opaque Types

Demonstrates how TypeId preserves the semantic distinction of opaque types, enabling runtime type safety that pure Scala reflection cannot provide. Shows building type-indexed validator registries keyed by opaque type identity:

```scala title="schema-examples/src/main/scala/typeid/OpaqueTypesExample.scala" 
/*
 * Copyright 2024-2026 John A. De Goes and the ZIO Contributors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package typeid

/**
 * Opaque Types Example
 *
 * Demonstrates how TypeId preserves the semantic distinction of opaque types,
 * enabling runtime type safety that pure Scala reflection cannot provide.
 *
 * Opaque types allow you to create distinct types that share the same
 * representation at runtime. TypeId captures this distinction, making it
 * possible to:
 *   - Distinguish between different opaque types wrapping the same base type
 *   - Build type-indexed registries that respect opaque type boundaries
 *   - Implement runtime validators specific to each opaque type
 *
 * Run with: sbt "schema-examples/runMain typeid.OpaqueTypesExample"
 */

object OpaqueTypesExample extends App {

  println("═══════════════════════════════════════════════════════════════")
  println("Opaque Types with TypeId")
  println("═══════════════════════════════════════════════════════════════\n")

  // Define domain-specific opaque types wrapping String
  // In a real application, these would enforce different validation rules
  opaque type UserId       = String
  opaque type Email        = String
  opaque type SessionToken = String

  println("--- Opaque Type Definitions ---\n")
  println("opaque type UserId = String")
  println("opaque type Email = String")
  println("opaque type SessionToken = String\n")

  // Derive TypeIds for each opaque type
  val userIdType       = TypeId.of[UserId]
  val emailType        = TypeId.of[Email]
  val sessionTokenType = TypeId.of[SessionToken]
  val stringType       = TypeId.string

  println("--- TypeId Derivation ---\n")

  println("TypeId.of[UserId].name")
  show(userIdType.name)

  println("TypeId.of[Email].name")
  show(emailType.name)

  println("TypeId.of[SessionToken].name")
  show(sessionTokenType.name)

  println("\n--- Opaque Types vs Base Type ---\n")

  // Key insight: opaque types are distinct from their representation type
  println("TypeId.of[UserId].isEquivalentTo(TypeId.string)")
  show(userIdType.isEquivalentTo(stringType))

  println("TypeId.of[Email].isEquivalentTo(TypeId.string)")
  show(emailType.isEquivalentTo(stringType))

  println("TypeId.of[SessionToken].isEquivalentTo(TypeId.string)")
  show(sessionTokenType.isEquivalentTo(stringType))

  println("\n--- Opaque Types are Distinct from Each Other ---\n")

  println("TypeId.of[UserId].isEquivalentTo(TypeId.of[Email])")
  show(userIdType.isEquivalentTo(emailType))

  println("TypeId.of[Email].isEquivalentTo(TypeId.of[SessionToken])")
  show(emailType.isEquivalentTo(sessionTokenType))

  println("TypeId.of[UserId].isEquivalentTo(TypeId.of[SessionToken])")
  show(userIdType.isEquivalentTo(sessionTokenType))

  println("\n--- Real-World Use Case: Type-Safe Registry ---\n")

  // Define validators for each opaque type
  trait Validator {
    def validate(value: String): Boolean
    def errorMessage: String
  }

  val userIdValidator = new Validator {
    def validate(value: String): Boolean = value.nonEmpty && value.forall(_.isDigit)
    def errorMessage                     = "UserId must be non-empty digits"
  }

  val emailValidator = new Validator {
    def validate(value: String): Boolean = value.contains("@") && value.contains(".")
    def errorMessage                     = "Email must contain @ and ."
  }

  val sessionTokenValidator = new Validator {
    def validate(value: String): Boolean = value.length >= 32
    def errorMessage                     = "SessionToken must be at least 32 characters"
  }

  // Build a type-indexed registry of validators
  // This demonstrates the power of TypeId: we can safely dispatch
  // to different validators based on opaque type identity
  val validatorRegistry: Map[TypeId.Erased, Validator] = Map(
    TypeId.of[UserId].erased       -> userIdValidator,
    TypeId.of[Email].erased        -> emailValidator,
    TypeId.of[SessionToken].erased -> sessionTokenValidator
  )

  println("Built validator registry keyed by opaque type")
  println("Validators can enforce different validation rules per type\n")

  // Demonstrate validation dispatch
  def validateString(value: String, typeId: TypeId[_]): Boolean =
    validatorRegistry
      .get(typeId.erased)
      .map(_.validate(value))
      .getOrElse {
        println(s"No validator found for type: ${typeId.fullName}")
        false
      }

  def getValidationError(typeId: TypeId[_]): String =
    validatorRegistry
      .get(typeId.erased)
      .map(_.errorMessage)
      .getOrElse("Unknown validator")

  println("--- Validation Examples ---\n")

  val testUserId = "12345"
  val testEmail  = "user@example.com"
  val testToken  = "a" * 32

  println(s"Validating UserId: '$testUserId'")
  if (validateString(testUserId, TypeId.of[UserId])) {
    println("✓ Valid UserId\n")
  } else {
    println(s"✗ Invalid: ${getValidationError(TypeId.of[UserId])}\n")
  }

  println(s"Validating Email: '$testEmail'")
  if (validateString(testEmail, TypeId.of[Email])) {
    println("✓ Valid Email\n")
  } else {
    println(s"✗ Invalid: ${getValidationError(TypeId.of[Email])}\n")
  }

  println(s"Validating SessionToken: '$testToken'")
  if (validateString(testToken, TypeId.of[SessionToken])) {
    println("✓ Valid SessionToken\n")
  } else {
    println(s"✗ Invalid: ${getValidationError(TypeId.of[SessionToken])}\n")
  }

  println("--- What Pure Scala Cannot Do ---\n")

  println("With pure Scala reflection:")
  println("- classOf[UserId] == classOf[String]  (erased at runtime)")
  println("- classOf[Email] == classOf[String]   (erased at runtime)")
  println("- You cannot distinguish opaque types from their base type\n")

  println("With TypeId:")
  println("- TypeId.of[UserId] != TypeId.of[String]  (preserved)")
  println("- TypeId.of[Email] != TypeId.of[String]   (preserved)")
  println("- You can build type-safe validators and registries\n")

  println("═══════════════════════════════════════════════════════════════")
}
```

([source](https://github.com/zio/zio-blocks/blob/main/schema-examples/src/main/scala/typeid/OpaqueTypesExample.scala))

```bash
sbt "schema-examples/runMain typeid.OpaqueTypesExample"
```

---

## Validation

`Validation` is a sealed trait in ZIO Blocks that represents declarative constraints on primitive values. Validations are attached to `PrimitiveType` instances and are checked during schema operations like decoding from `DynamicValue` or validating against a `DynamicSchema`.

## Overview

The validation system in ZIO Blocks provides:

- **Declarative constraints** on numeric and string values
- **Automatic enforcement** during schema-based decoding
- **Integration with wrapper types** via `transform` for custom validation logic
- **Schema error reporting** with path information for debugging via [`SchemaError`](./schema-error.md)

```
Validation[A]
 ├── Validation.None                    (no constraint)
 │
 ├── Validation.Numeric[A]              (numeric constraints)
 │    ├── Positive                      (> 0)
 │    ├── Negative                      (< 0)
 │    ├── NonPositive                   (<= 0)
 │    ├── NonNegative                   (>= 0)
 │    ├── Range[A](min, max)            (within bounds)
 │    └── Set[A](values)                (one of specific values)
 │
 └── Validation.String                  (string constraints)
      ├── NonEmpty                      (length > 0)
      ├── Empty                         (length == 0)
      ├── Blank                         (whitespace only)
      ├── NonBlank                      (has non-whitespace)
      ├── Length(min, max)              (length bounds)
      └── Pattern(regex)                (regex match)
```

## Built-in Validations

### Numeric Validations

Numeric validations apply to `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `BigInt`, and `BigDecimal`.

```scala

// Sign constraints
Validation.Numeric.Positive     // value > 0
Validation.Numeric.Negative     // value < 0
Validation.Numeric.NonPositive  // value <= 0
Validation.Numeric.NonNegative  // value >= 0

// Range constraint (inclusive bounds)
Validation.Numeric.Range(Some(1), Some(100))   // 1 <= value <= 100
Validation.Numeric.Range(Some(0), None)        // value >= 0 (no upper bound)
Validation.Numeric.Range(None, Some(1000))     // value <= 1000 (no lower bound)

// Set constraint (value must be one of the specified values)
Validation.Numeric.Set(Set(1, 2, 3, 5, 8, 13))
```

### String Validations

String validations apply to `String` primitive types.

```scala

// Content constraints
Validation.String.NonEmpty   // string.nonEmpty (length > 0)
Validation.String.Empty      // string.isEmpty (length == 0)
Validation.String.Blank      // string.trim.isEmpty (whitespace only)
Validation.String.NonBlank   // string.trim.nonEmpty (has non-whitespace)

// Length constraint (inclusive bounds)
Validation.String.Length(Some(1), Some(255))   // 1 <= length <= 255
Validation.String.Length(Some(3), None)        // length >= 3
Validation.String.Length(None, Some(100))      // length <= 100

// Pattern constraint (regex)
Validation.String.Pattern("^[a-z]+$")          // lowercase letters only
Validation.String.Pattern("^\\d{5}$")          // exactly 5 digits
```

## Validation with PrimitiveType

Validations are attached to `PrimitiveType` instances. Each primitive type carries its validation constraint:

```scala

// Int with no validation
val intType = PrimitiveType.Int(Validation.None)

// Positive Int
val positiveIntType = PrimitiveType.Int(Validation.Numeric.Positive)

// Non-empty String
val nonEmptyStringType = PrimitiveType.String(Validation.String.NonEmpty)

// String matching email pattern
val emailPattern = "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$"
val emailStringType = PrimitiveType.String(Validation.String.Pattern(emailPattern))

// Int in range 1-10
val rangeIntType = PrimitiveType.Int(Validation.Numeric.Range(Some(1), Some(10)))
```

## Error Handling with SchemaError

When validation fails, ZIO Blocks returns a `SchemaError` that provides detailed information about what went wrong and where.

### SchemaError Structure

`SchemaError` is an exception that contains one or more error details:

```scala

// SchemaError wraps a non-empty list of Single errors
final case class SchemaError(errors: ::[SchemaError.Single]) extends Exception

// Single error types
SchemaError.Single
 ├── ConversionFailed(source, details, cause)  // transformation/validation failure
 ├── MissingField(source, fieldName)           // required field not present
 ├── DuplicatedField(source, fieldName)        // field appears multiple times
 ├── ExpectationMismatch(source, expectation)  // type mismatch
 ├── UnknownCase(source, caseName)             // unknown variant case
 └── Message(source, details)                  // generic message
```

### Creating Validation Errors

Use the factory methods on `SchemaError` to create errors:

```scala

// For validation failures in transform
val error = SchemaError.validationFailed("must be positive")

// Generic message error
val msgError = SchemaError("Invalid input")

// With path information

val pathError = SchemaError.message("Value out of range", DynamicOptic.root.field("age"))

// Conversion failure with details
val convError = SchemaError.conversionFailed(Nil, "Expected ISO date format")
```

### Error Messages and Paths

`SchemaError` includes path information showing where in the data structure the error occurred:

```scala

val error: SchemaError = ???

// Get the full error message
val msg: String = error.message

// Add path context to errors
val atField = error.atField("name")    // prepend .name to path
val atIndex = error.atIndex(0)         // prepend [0] to path
val atCase = error.atCase("Some")      // prepend case context

// Combine multiple errors
val combined = error1 ++ error2
```

Example error message:
```
Validation failed: value must be positive at: $.user.age
```

## Integration with Wrapper Types

The most common way to use validation in ZIO Blocks is through `transform`, which creates a schema for a wrapper type with validation logic.

### Basic Wrapper with Validation

```scala

case class PositiveInt private (value: Int)

object PositiveInt {
  def unsafeMake(n: Int): PositiveInt =
    if (n > 0) PositiveInt(n)
    else throw SchemaError.validationFailed("must be positive")

  implicit val schema: Schema[PositiveInt] =
    Schema[Int].transform(unsafeMake, _.value)
}
```

### Email Type with Regex Validation

```scala

case class Email private (value: String)

object Email {
  private val EmailRegex = "^[A-Za-z0-9+_.-]+@[A-Za-z0-9.-]+$".r

  def unsafeMake(s: String): Email =
    s match {
      case EmailRegex(_*) => Email(s)
      case _ => throw SchemaError.validationFailed("Invalid email format")
    }

  implicit val schema: Schema[Email] =
    Schema[String].transform(unsafeMake, _.value).withTypeName[Email]
}
```

### NonEmptyString with Length Validation

```scala

case class NonEmptyString private (value: String)

object NonEmptyString {
  def unsafeMake(s: String): NonEmptyString =
    if (s.nonEmpty) NonEmptyString(s)
    else throw SchemaError.validationFailed("String must not be empty")

  implicit val schema: Schema[NonEmptyString] =
    Schema[String].transform(unsafeMake, _.value).withTypeName[NonEmptyString]
}
```

### Range-Bounded Integer

```scala

case class Percentage private (value: Int)

object Percentage {
  def unsafeMake(n: Int): Percentage =
    if (n >= 0 && n <= 100) Percentage(n)
    else throw SchemaError.validationFailed(s"Percentage must be 0-100, got $n")

  implicit val schema: Schema[Percentage] =
    Schema[Int].transform(unsafeMake, _.value).withTypeName[Percentage]
}
```

### Bidirectional Validation

Use the two-argument `transform` for cases where both encoding and decoding need validation:

```scala

case class BoundedValue(value: Int)

object BoundedValue {
  implicit val schema: Schema[BoundedValue] = Schema[Int].transform(
    wrap = n =>
      if (n >= 0 && n < 100) BoundedValue(n)
      else throw SchemaError.validationFailed("Value must be in [0, 100)"),
    unwrap = v =>
      if (v.value >= 0) v.value
      else throw SchemaError.validationFailed("Corrupted value")
  )
}
```

## Validation at Encode/Decode Time

### Decoding with Validation

When decoding from `DynamicValue` or JSON, validations in wrapper schemas are automatically enforced:

```scala

case class PositiveInt(value: Int)
object PositiveInt {
  def unsafeMake(n: Int): PositiveInt =
    if (n > 0) PositiveInt(n)
    else throw SchemaError.validationFailed("must be positive")

  implicit val schema: Schema[PositiveInt] =
    Schema[Int].transform(unsafeMake, _.value)
}

case class Order(quantity: PositiveInt, price: BigDecimal)
object Order {
  implicit val schema: Schema[Order] = Schema.derived
}

// JSON decoding will validate PositiveInt
val json = """{"quantity": -5, "price": 99.99}"""
val result = JsonDecoder[Order].decodeString(json)
// result: Left(SchemaError: must be positive at $.quantity)
```

### DynamicValue Validation

Use `DynamicSchema` to validate `DynamicValue` instances:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Create a DynamicSchema for validation
val dynamicSchema: DynamicSchema = Schema[Person].toDynamicSchema

// Create a DynamicValue to validate
val value = DynamicValue.Record(Vector(
  "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
  "age" -> DynamicValue.Primitive(PrimitiveValue.Int(30))
))

// Validate the value
val checkResult: Option[SchemaError] = dynamicSchema.check(value)
// None if valid, Some(error) if invalid

val isValid: Boolean = dynamicSchema.conforms(value)
// true if valid
```

### Converting DynamicSchema to Validating Schema

`DynamicSchema.toSchema` creates a `Schema[DynamicValue]` that rejects non-conforming values:

```scala

val dynamicSchema: DynamicSchema = Schema[Person].toDynamicSchema
val validatingSchema: Schema[DynamicValue] = dynamicSchema.toSchema

// Now any decoding through this schema will validate structure
val invalidValue = DynamicValue.Record(Vector(
  "name" -> DynamicValue.Primitive(PrimitiveValue.Int(42))  // wrong type!
))

val result = validatingSchema.fromDynamicValue(invalidValue)
// Left(SchemaError: Expected String, got Int at $.name)
```

## Validation in JSON Schema

When deriving JSON Schema from a ZIO Blocks schema, validations are reflected in the output:

```scala

// Numeric validations become JSON Schema constraints
// Validation.Numeric.Range(Some(0), Some(100)) → "minimum": 0, "maximum": 100

// String validations become JSON Schema constraints
// Validation.String.NonEmpty → "minLength": 1
// Validation.String.Length(Some(1), Some(255)) → "minLength": 1, "maxLength": 255
// Validation.String.Pattern("^[a-z]+$") → "pattern": "^[a-z]+$"
```

When parsing JSON Schema, these constraints are converted back to `Validation` instances.

## Composing Validations

The current `Validation` ADT does not support combining multiple validations on a single primitive (e.g., both `NonEmpty` and `Pattern`). For complex validation logic, use `transform`:

```scala

case class Username private (value: String)

object Username {
  private val UsernameRegex = "^[a-z][a-z0-9_]{2,19}$".r

  def unsafeMake(s: String): Username = {
    if (s.isEmpty)
      throw SchemaError.validationFailed("Username cannot be empty")
    else if (s.length < 3)
      throw SchemaError.validationFailed("Username must be at least 3 characters")
    else if (s.length > 20)
      throw SchemaError.validationFailed("Username cannot exceed 20 characters")
    else if (!s.matches(UsernameRegex.regex))
      throw SchemaError.validationFailed("Username must start with a letter and contain only lowercase letters, numbers, and underscores")
    else
      Username(s)
  }

  implicit val schema: Schema[Username] =
    Schema[String].transform(unsafeMake, _.value).withTypeName[Username]
}
```

## Best Practices

### 1. Use Wrapper Types for Domain Validation

Prefer creating dedicated wrapper types with `transform` over relying solely on `Validation` constraints:

```scala
// Good: Explicit domain type with validation
case class OrderId private (value: String)
object OrderId {
  def unsafeMake(s: String): OrderId =
    if (s.matches("^ORD-\\d{8}$")) OrderId(s)
    else throw SchemaError.validationFailed("Invalid order ID format")

  implicit val schema: Schema[OrderId] =
    Schema[String].transform(unsafeMake, _.value).withTypeName[OrderId]
}

// Less ideal: Raw String with separate validation
val orderIdString: String = ???
```

### 2. Provide Clear Error Messages

Include context in error messages to help users understand what went wrong:

```scala
// Good: Specific, actionable error message
throw SchemaError.validationFailed(
  s"Age must be between 0 and 150, got $age"
)

// Less helpful: Generic message
throw SchemaError.validationFailed("Invalid age")
```

### 3. Combine Errors When Possible

For multiple validation failures, combine them into a single `SchemaError`:

```scala
def validate(input: Input): Either[SchemaError, ValidInput] = {
  val errors = List.newBuilder[SchemaError]

  if (input.name.isEmpty)
    errors += SchemaError.validationFailed("name is required")
  if (input.age < 0)
    errors += SchemaError.validationFailed("age must be non-negative")

  val allErrors = errors.result()
  if (allErrors.isEmpty) Right(ValidInput(input))
  else Left(allErrors.reduce(_ ++ _))
}
```

---

## XML

`Xml` is a **sealed trait representing XML nodes**. It provides a type-safe, immutable representation of all valid XML document structures including elements, text nodes, CDATA sections, comments, and processing instructions.

```scala
sealed trait Xml {
  def xmlType: XmlType
  def is(xmlType: XmlType): Boolean
  def as(xmlType: XmlType): Option[xmlType.Type]
  def unwrap(xmlType: XmlType): Option[xmlType.Unwrap]
  def print: String
  def printPretty: String
  def select: XmlSelection
}
```

`Xml` supports:
- Type-safe representation of all XML node types
- Fluent navigation and querying with XmlSelection
- Schema-derived automatic codec generation
- Zero external dependencies
- Full cross-platform support (JVM and Scala.js)

## Overview

Zero-dependency XML codec for ZIO Blocks Schema with cross-platform support.

The schema-xml module provides automatic XML codec derivation for any type with a `Schema`. It includes a complete XML AST, fluent navigation API, and support for XML-specific features like attributes and namespaces.

Key features:

- **Zero Dependencies**: No external XML libraries required
- **Cross-Platform**: Full support for JVM and Scala.js
- **Schema-Based**: Automatic codec derivation from Schema definitions
- **XML AST**: Complete representation of XML documents
- **Fluent API**: Navigation and transformation with XmlSelection
- **Attributes**: First-class support for XML attributes via annotations
- **Namespaces**: XML namespace support with prefix handling

## Installation

To use the schema-xml module, add the following dependency to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-schema-xml" % "0.0.14"
```

## Basic Usage

Start by deriving an XML codec from your Schema definition:

### Deriving Codecs

To create an XML codec, use `Schema[A].derive(XmlFormat)`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Derive XML codec using the unified format API
val codec = Schema[Person].derive(XmlFormat)
```

### Encoding to XML

Encode your values to XML using the codec:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)
val person = Person("Alice", 30)
```

Encode to XML bytes:

```scala
val bytes: Array[Byte] = codec.encode(person)
// bytes: Array[Byte] = Array(
//   60,
//   80,
//   101,
//   114,
//   115,
//   111,
//   110,
//   62,
//   60,
//   110,
//   97,
//   109,
//   101,
//   62,
//   65,
//   108,
//   105,
//   99,
//   101,
//   60,
//   47,
//   110,
//   97,
//   109,
//   101,
//   62,
//   60,
//   97,
//   103,
//   101,
//   62,
//   51,
//   48,
//   60,
//   47,
//   97,
//   103,
//   101,
//   62,
//   60,
//   47,
//   80,
//   101,
//   114,
//   115,
//   111,
//   110,
//   62
// )
```

Encode to XML string:

```scala
val xmlString: String = codec.encodeToString(person)
// xmlString: String = "<Person><name>Alice</name><age>30</age></Person>"
```

Encode to pretty-printed XML:

```scala
val prettyXml = codec.encodeToString(person, WriterConfig.pretty)
// prettyXml: String = """<Person>
//   <name>Alice</name>
//   <age>30</age>
// </Person>"""
```

### Decoding from XML

Decode XML strings or bytes back to your typed values:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)

// Decode from XML string
val xml = "<Person><name>Alice</name><age>30</age></Person>"
```

Decode the XML string and see the result:

```scala
val result: Either[SchemaError, Person] = codec.decode(xml)
// result: Either[SchemaError, Person] = Right(
//   Person(name = "Alice", age = 30)
// )
```

You can also decode from bytes:

```scala
val bytes = xml.getBytes("UTF-8")
```

```scala
val fromBytes: Either[SchemaError, Person] = codec.decode(bytes)
// fromBytes: Either[SchemaError, Person] = Right(
//   Person(name = "Alice", age = 30)
// )
```

## XML AST

The `Xml` ADT represents all valid XML node types:

```
Xml
 ├── Xml.Element          (element with name, attributes, children)
 ├── Xml.Text             (character data)
 ├── Xml.CData            (unparsed character data)
 ├── Xml.Comment          (XML comment)
 └── Xml.ProcessingInstruction (processing instruction)
```

### Creating XML Nodes

Construct XML nodes directly using the case class constructors:

```scala

// Create elements
val simple = Xml.Element("person")
val withChildren = Xml.Element("person", Xml.Text("Alice"))

// Create with XmlName (for namespaces)
val namespaced = Xml.Element(
  XmlName("person", "http://example.com"),
  Chunk.empty,
  Chunk.empty
)

// Create text nodes
val text = Xml.Text("Hello, World!")
val cdata = Xml.CData("<script>...</script>")

// Create comments and processing instructions
val comment = Xml.Comment("This is a comment")
val pi = Xml.ProcessingInstruction("xml-stylesheet", "href=\"style.css\"")
```

### XmlName

`XmlName` represents an element or attribute name with optional namespace. Create instances with different namespace configurations:

```scala

// Local name only
val simple = XmlName("person")
simple.localName     // "person"
simple.namespace     // ""
simple.prefix        // ""

// With namespace URI
val ns = XmlName("person", "http://example.com/ns")

// With prefix (for prefixed elements like atom:feed)
val prefixed = XmlName("feed", Some("atom"), None)
prefixed.localName              // "feed"
prefixed.prefix.contains("atom") // true
```

## XmlBuilder

Construct XML documents programmatically with a fluent API:

```scala

// Build an element with attributes and children
val doc = XmlBuilder.element("person")
  .attr("id", "123")
  .attr("status", "active")
  .child(XmlBuilder.element("name").text("Alice").build)
  .child(XmlBuilder.element("age").text("30").build)
  .build

// Result:
// <person id="123" status="active">
//   <name>Alice</name>
//   <age>30</age>
// </person>

// Create other node types
val textNode = XmlBuilder.text("content")
val cdataNode = XmlBuilder.cdata("<![CDATA[raw content]]>")
val commentNode = XmlBuilder.comment("comment text")
```

## Configuration

The schema-xml module provides configuration options for both parsing and writing:

### WriterConfig

Use `WriterConfig` to control XML output formatting:

```scala

// Compact output (default)
val compact = WriterConfig.default
// <Person><name>Alice</name></Person>

// Pretty-printed with 2-space indentation
val pretty = WriterConfig.pretty
// <Person>
//   <name>Alice</name>
// </Person>

// With XML declaration
val withDecl = WriterConfig.withDeclaration
// <?xml version="1.0" encoding="UTF-8"?>
// <Person><name>Alice</name></Person>

// Custom configuration
val custom = WriterConfig(
  indentStep = 4,
  includeDeclaration = true,
  encoding = "UTF-8"
)
```

| Option | Default | Description |
|--------|---------|-------------|
| `indentStep` | `0` | Spaces per indentation level (0 = compact) |
| `includeDeclaration` | `false` | Include XML declaration |
| `encoding` | `"UTF-8"` | Character encoding in declaration |

### ReaderConfig

Controls XML parsing behavior and security limits:

```scala

// Default configuration
val default = ReaderConfig.default

// Custom limits
val custom = ReaderConfig(
  maxDepth = 100,
  maxAttributes = 50,
  maxTextLength = 1000000,
  preserveWhitespace = true
)
```

| Option | Default | Description |
|--------|---------|-------------|
| `maxDepth` | `1000` | Maximum element nesting depth |
| `maxAttributes` | `1000` | Maximum attributes per element |
| `maxTextLength` | `10000000` | Maximum text content length |
| `preserveWhitespace` | `false` | Preserve whitespace in text nodes |

## Attributes

Encode case class fields as XML attributes using the `@xmlAttribute` annotation:

```scala

case class Person(
  @xmlAttribute() id: String,
  @xmlAttribute("status") active: String,
  name: String,
  age: Int
)

object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)
val person = Person("123", "active", "Alice", 30)
```

Encode the person with attributes:

```scala
val xml = codec.encodeToString(person)
// xml: String = "<Person><id>123</id><active>active</active><name>Alice</name><age>30</age></Person>"
```

The `@xmlAttribute` annotation accepts an optional custom name:

- `@xmlAttribute()` - Uses the field name as the attribute name
- `@xmlAttribute("customName")` - Uses the provided name as the attribute name

## Namespaces

Support for XML namespaces with the `@xmlNamespace` annotation:

```scala

@xmlNamespace(uri = "http://www.w3.org/2005/Atom", prefix = "atom")
case class Feed(
  title: String,
  updated: String
)

object Feed {
  implicit val schema: Schema[Feed] = Schema.derived
}

val codec = Schema[Feed].derive(XmlFormat)
val feed = Feed("My Blog", "2024-01-01T00:00:00Z")
```

Encode the feed with a namespace prefix:

```scala
val xml = codec.encodeToString(feed)
// xml: String = "<Feed><title>My Blog</title><updated>2024-01-01T00:00:00Z</updated></Feed>"
```

Without a prefix (default namespace):

```scala
@xmlNamespace(uri = "http://www.w3.org/2005/Atom")
case class Feed(title: String)

object Feed {
  implicit val schema: Schema[Feed] = Schema.derived
}

val codec = Schema[Feed].derive(XmlFormat)
val feed = Feed("My Blog")
```

Encode the feed with default namespace:

```scala
val xml = codec.encodeToString(feed)
// xml: String = "<Feed><title>My Blog</title></Feed>"
```

## XmlSelection

`XmlSelection` provides a fluent API for navigating and querying XML structures:

### Navigation

Navigate to child elements, filter by type, and extract content:

```scala

val xml = XmlReader.read("""
<library>
  <books>
    <book id="1">
      <title>Functional Programming</title>
      <author>Alice</author>
    </book>
    <book id="2">
      <title>Advanced Scala</title>
      <author>Bob</author>
    </book>
  </books>
</library>
""")

// Navigate to child elements
val books = xml.select.get("library").get("books")

// Navigate by index
val firstBook = books.get("book")(0)
```

Extract text content from the first book:

```scala
val title: Either[SchemaError, String] = firstBook.get("title").text
// title: Either[SchemaError, String] = Left(
//   SchemaError(
//     List(
//       Message(
//         source = DynamicOptic(IndexedSeq()),
//         details = "Expected single value but got 0"
//       )
//     )
//   )
// )
```

You can also navigate descendants recursively:

```scala
val allTitles = xml.select.descendant("title")
// allTitles: XmlSelection = XmlSelection(
//   Right(
//     IndexedSeq(
//       Element(
//         name = XmlName(localName = "title", prefix = None, namespace = None),
//         attributes = IndexedSeq(),
//         children = IndexedSeq(Text("Functional Programming"))
//       ),
//       Element(
//         name = XmlName(localName = "title", prefix = None, namespace = None),
//         attributes = IndexedSeq(),
//         children = IndexedSeq(Text("Advanced Scala"))
//       )
//     )
//   )
// )
```

### Filtering

Filter selections by node type or custom predicates:

```scala

val selection: XmlSelection = ???

// Filter by type
val elements = selection.elements
val texts = selection.texts
val comments = selection.comments

// Custom filtering
val filtered = selection.filter(xml => xml.is(XmlType.Element))
```

### Terminal Operations

Execute a selection to extract values or convert to other formats:

```scala

val selection: XmlSelection = ???

// Get single value (fails if not exactly one)
val one: Either[SchemaError, Xml] = selection.one

// Get any value (first of many)
val any: Either[SchemaError, Xml] = selection.any

// Get all values as a single XML element
val all: Either[SchemaError, Xml] = selection.all

// Convert to chunk
val chunk = selection.toChunk

// Extract text content
val text: Either[SchemaError, String] = selection.text
val allText: String = selection.textContent
```

### Combinators

Combine and transform selections using monadic operations:

```scala

val selection1: XmlSelection = ???
val selection2: XmlSelection = ???

// Map over selections
val mapped = selection1.map(xml => xml)

// FlatMap for chaining
val nested = selection1.flatMap(xml => XmlSelection.succeed(xml))

// Combine selections
val combined = selection1 ++ selection2

// Alternative on failure
val withFallback = selection1.orElse(selection2)
```

## XmlPatch

`XmlPatch` provides composable XML modification operations:

### Creating Patches

Create patches for add, remove, replace, and attribute operations:

```scala

val path = p".library.books.book"

// Add content
val addPatch = XmlPatch.add(
  path,
  XmlBuilder.element("book").attr("id", "3").build,
  XmlPatch.Position.AppendChild
)

// Remove element
val removePatch = XmlPatch.remove(path)

// Replace element
val replacePatch = XmlPatch.replace(
  path,
  XmlBuilder.element("book").attr("id", "999").build
)

// Set attribute
val attrPatch = XmlPatch.setAttribute(path, "featured", "true")

// Remove attribute
val removeAttrPatch = XmlPatch.removeAttribute(path, "id")
```

### Position Options

Position options control where new content is inserted relative to the target:

```scala

Position.Before         // Insert before the target element
Position.After          // Insert after the target element
Position.PrependChild   // Insert as first child of target
Position.AppendChild    // Insert as last child of target
```

### Applying Patches

Apply a patch to an XML document to produce a modified result:

```scala

val xml: Xml = ???
val patch = XmlPatch.setAttribute(p".person", "active", "true")

// Apply the patch
val result: Either[SchemaError, Xml] = patch(xml)
```

### Composing Patches

Combine multiple patches to apply transformations in sequence:

```scala

val patch1 = XmlPatch.setAttribute(p".person", "id", "123")
val patch2 = XmlPatch.add(
  p".person",
  XmlBuilder.element("email").text("alice@example.com").build,
  XmlPatch.Position.AppendChild
)

// Compose patches - applies patch1, then patch2
val combined = patch1 ++ patch2
```

## Extension Syntax

When a `Schema` is in scope, use convenient extension methods:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val person = Person("Alice", 30)

// Encode to XML AST
val xml: Xml = person.toXml

// Encode to XML string
val xmlString: String = person.toXmlString
// <Person><name>Alice</name><age>30</age></Person>

// Encode to bytes
val bytes: Array[Byte] = person.toXmlBytes

// Decode from XML string
val parsed: Either[SchemaError, Person] = 
  "<Person><name>Bob</name><age>25</age></Person>".fromXml[Person]

// Decode from bytes
val fromBytes: Either[SchemaError, Person] = bytes.fromXml[Person]
```

## Printing XML

Format XML documents using compact or pretty-printed output. You can convert XML to string with different formatting options:

```scala

val xml = Xml.Element("person", Xml.Element("name", Xml.Text("Alice")))
```

Compact output:

```scala
val compact: String = xml.print
// compact: String = "<person><name>Alice</name></person>"
```

Pretty-printed output:

```scala
val pretty: String = xml.printPretty
// pretty: String = """<person>
//   <name>Alice</name>
// </person>"""
```

Custom configuration:

```scala
val custom: String = xml.print(WriterConfig(indentStep = 4))
```

## Type Testing and Access

Test and extract values from XML nodes using type guards and unwrapping:

```scala

val xml: Xml = Xml.Element("person")

// Type testing
xml.is(XmlType.Element)  // true
xml.is(XmlType.Text)     // false

// Type narrowing (returns Option)
val elem: Option[Xml.Element] = xml.as(XmlType.Element)

// Value extraction (returns Option)
val (name, attrs, children) = xml.unwrap(XmlType.Element).get
```

## Supported Types

All standard ZIO Blocks Schema types are supported:

**Numeric Types**:
- `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Char`
- `BigInt`, `BigDecimal`

**Text Types**:
- `String`

**Special Types**:
- `Unit`, `UUID`, `Currency`

**Java Time Types**:
- `Instant`, `LocalDate`, `LocalTime`, `LocalDateTime`
- `OffsetTime`, `OffsetDateTime`, `ZonedDateTime`
- `Duration`, `Period`
- `Year`, `YearMonth`, `MonthDay`
- `DayOfWeek`, `Month`
- `ZoneId`, `ZoneOffset`

**Composite Types**:
- Records (case classes)
- Variants (sealed traits)
- Sequences (`List`, `Vector`, `Set`, etc.)
- Maps (`Map[K, V]`)
- Options (`Option[A]`)
- Wrappers (newtypes)

## XmlCodec

`XmlCodec[A]` is the low-level codec interface that bridges Schema definitions with XML serialization. While usually derived automatically, you can work with it directly:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

// Get the underlying binary codec
val codec: XmlCodec[Person] = Schema[Person].derive(XmlCodecDeriver)

// Encode to Xml directly
val person = Person("Alice", 30)
val xml: Xml = codec.encodeValue(person)

// Decode from Xml directly
val decoded: Person = codec.decodeValue(xml)
```

**XmlCodec supports all Schema types:**
- Primitives (Int, String, Boolean, etc.)
- Java time types (Instant, LocalDate, Duration, etc.)
- Records (case classes with field-level configuration)
- Variants (sealed traits with discriminators)
- Collections (List, Vector, Map, etc.)
- Optional fields (Option[A])
- Custom wrappers and dynamic values

## Error Handling

All decoding operations return `Either[SchemaError, A]` or `Either[SchemaError, A]`. The `SchemaError` type provides detailed error information:

```scala

case class Person(name: String, age: Int)
object Person {
  implicit val schema: Schema[Person] = Schema.derived
}

val codec = Schema[Person].derive(XmlFormat)

// Decoding invalid XML
val invalid = "<Person><name>Alice</name></Person>"  // missing age
val result = codec.decode(invalid)

result match {
  case Right(person) => println(s"Decoded: $person")
  case Left(error) =>
    println(s"Error: ${error.getMessage}")
    // Error information includes parse location and context
}
```

`SchemaError` provides detailed error information for debugging:

```scala

val error = SchemaError("Parse failed")

// Error message
val message: String = error.getMessage

// Get error message for inspection
val errorMsg: String = error.getMessage
```

Error handling best practices:

```scala

case class Config(database: String, port: Int)
object Config {
  implicit val schema: Schema[Config] = Schema.derived
}

def loadConfig(xml: String): Either[String, Config] = {
  val codec = Schema[Config].derive(XmlFormat)
  codec.decode(xml).left.map { error =>
    s"Configuration error: ${error.getMessage}\nCheck XML format and required fields."
  }
}
```

## Cross-Platform Support

The XML module works across all platforms:

- **JVM** - Full functionality
- **Scala.js** - Browser and Node.js

All features including parsing, writing, navigation, and patching work identically on both platforms.

## Examples

These examples demonstrate common use cases and patterns with the XML module:

### Complete Example with Attributes and Namespaces

Define a schema with attributes and namespaces, then encode and decode:

```scala

@xmlNamespace(uri = "http://www.w3.org/2005/Atom", prefix = "atom")
case class Entry(
  @xmlAttribute() id: String,
  title: String,
  updated: String,
  author: Author
)

case class Author(name: String, email: String)

object Entry {
  implicit val authorSchema: Schema[Author] = Schema.derived
  implicit val schema: Schema[Entry] = Schema.derived
}

val codec = Schema[Entry].derive(XmlFormat)

val entry = Entry(
  id = "entry-1",
  title = "First Post",
  updated = "2024-01-01T00:00:00Z",
  author = Author("Alice", "alice@example.com")
)
```

Encode the entry with pretty printing:

```scala
val xmlStr = codec.encodeToString(entry, WriterConfig.pretty)
// xmlStr: String = """<Entry>
//   <id>entry-1</id>
//   <title>First Post</title>
//   <updated>2024-01-01T00:00:00Z</updated>
//   <author>
//     <name>Alice</name>
//     <email>alice@example.com</email>
//   </author>
// </Entry>"""
```

Decode the XML back to a typed value:

```scala
val decoded = codec.decode(xmlStr)
// decoded: Either[SchemaError, Entry] = Right(
//   Entry(
//     id = "entry-1",
//     title = "First Post",
//     updated = "2024-01-01T00:00:00Z",
//     author = Author(name = "Alice", email = "alice@example.com")
//   )
// )
```

### Navigation and Transformation

Find elements, extract data, and apply patches to modify XML:

```scala

val xmlString = """
<library>
  <books>
    <book id="1">
      <title>Functional Programming</title>
      <price>49.99</price>
    </book>
    <book id="2">
      <title>Advanced Scala</title>
      <price>59.99</price>
    </book>
  </books>
</library>
"""

val xml = XmlReader.read(xmlString)

// Find all books
val books = xml.select.get("library").get("books").get("book")

// Extract all titles
val titles = books.get("title").toChunk.map { titleElem =>
  titleElem.asInstanceOf[Xml.Element].children.head match {
    case Xml.Text(text) => text
    case _ => ""
  }
}
// Chunk("Functional Programming", "Advanced Scala")

// Apply a patch to add a new book
val patch = XmlPatch.add(
  p".library.books",
  XmlBuilder.element("book")
    .attr("id", "3")
    .child(XmlBuilder.element("title").text("ZIO Essentials").build)
    .child(XmlBuilder.element("price").text("39.99").build)
    .build,
  XmlPatch.Position.AppendChild
)

val updated = patch(xml)
```

## Real-World Examples

Learn by examining practical examples of XML codecs in action:

### RSS Feed Parsing

Parse RSS feeds by defining a schema and decoding XML:

```scala

@xmlNamespace(uri = "http://www.rss.org/", prefix = "rss")
case class Item(
  @xmlAttribute() guid: String,
  title: String,
  link: String,
  pubDate: String,
  description: String
)

@xmlNamespace(uri = "http://www.rss.org/", prefix = "rss")
case class Channel(
  title: String,
  link: String,
  description: String,
  items: List[Item]
)

object Channel {
  implicit val itemSchema: Schema[Item] = Schema.derived
  implicit val schema: Schema[Channel] = Schema.derived
}

// Parse RSS feed from string
val feedXml = """<rss:Channel xmlns:rss="http://www.rss.org/">
  <title>Tech Blog</title>
  <link>https://example.com</link>
  <description>Latest tech articles</description>
  <items>
    <Item guid="1">
      <title>Functional Programming in Scala</title>
      <link>https://example.com/fp</link>
      <pubDate>2024-01-01</pubDate>
      <description>Deep dive into FP concepts</description>
    </Item>
  </items>
</rss:Channel>"""

val codec = Schema[Channel].derive(XmlFormat)
val result: Either[SchemaError, Channel] = codec.decode(feedXml)
```

### Atom Feed with Advanced Features

Work with Atom feeds using attributes, multiple entries, and custom configurations:

```scala

@xmlNamespace(uri = "http://www.w3.org/2005/Atom", prefix = "atom")
case class Entry(
  @xmlAttribute() id: String,
  title: String,
  author: String,
  updated: String,
  @xmlAttribute("href") link: String
)

@xmlNamespace(uri = "http://www.w3.org/2005/Atom", prefix = "atom")
case class Feed(
  @xmlAttribute() id: String,
  title: String,
  updated: String,
  entries: List[Entry]
)

object Feed {
  implicit val entrySchema: Schema[Entry] = Schema.derived
  implicit val schema: Schema[Feed] = Schema.derived
}

// Encode feed to XML with custom formatting
val feed = Feed(
  id = "urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6",
  title = "Example Feed",
  updated = "2024-01-01T18:30:02Z",
  entries = List(
    Entry(
      id = "urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a",
      title = "Atom-Powered Robots Run Amok",
      author = "John Doe",
      updated = "2024-01-01T18:30:02Z",
      link = "http://example.org/2024/01/entry"
    )
  )
)

val codec = Schema[Feed].derive(XmlFormat)
val xmlOutput = codec.encodeToString(feed, WriterConfig.pretty)
```

### Sitemap XML

Generate sitemap XML with URLs and optional metadata fields:

```scala

case class Url(
  loc: String,
  lastmod: Option[String],
  changefreq: Option[String],
  priority: Option[Double]
)

case class Urlset(
  urls: List[Url]
)

object Urlset {
  implicit val urlSchema: Schema[Url] = Schema.derived
  implicit val schema: Schema[Urlset] = Schema.derived
}

// Build and encode sitemap
val sitemap = Urlset(List(
  Url("https://example.com", Some("2024-01-01"), Some("monthly"), Some(1.0)),
  Url("https://example.com/about", Some("2024-01-01"), Some("monthly"), Some(0.8)),
  Url("https://example.com/contact", None, Some("monthly"), Some(0.5))
))

val codec = Schema[Urlset].derive(XmlFormat)
val sitemapXml = codec.encodeToString(sitemap, WriterConfig(
  indentStep = 2,
  includeDeclaration = true
))
```

---

## Ring Buffer

# ZIO Blocks — Ring Buffer (`zio.blocks.ringbuffer`)

`zio.blocks.ringbuffer` is a family of **high-performance, bounded ring buffers** for the JVM (and Scala.js). Each variant is optimized for a specific producer/consumer threading pattern—pick the one that matches your use case and get the fastest possible inter-thread communication with zero dependencies.

All variants expose `offer` (returns `false` if full) and `take` (returns `null` if empty)—both non-blocking. Capacity must be a **power of two** (enables bitwise masking instead of modulo). Elements must be **non-null reference types** (`A <: AnyRef`).

## Ring buffer variants

| Type | Producers | Consumers | Algorithm |
|------|-----------|-----------|-----------|
| `SpscRingBuffer` | 1 | 1 | FastFlow (null/non-null signaling) |
| `SpmcRingBuffer` | 1 | Many | Index-based capacity + CAS consumers |
| `MpscRingBuffer` | Many | 1 | CAS producers + relaxed poll |
| `MpmcRingBuffer` | Many | Many | Vyukov/Dmitry sequence buffer |

**Naming convention:** `S` = single, `M` = multi, `p` = producer, `c` = consumer.

---

## Installation

```scala
libraryDependencies += "dev.zio" %% "zio-blocks-ringbuffer" % "0.0.33"
```

---

## Quick start

```scala

val buf = SpscRingBuffer[String](1024) // capacity must be a power of 2

// Producer thread
buf.offer("hello") // true if inserted, false if full

// Consumer thread
val msg: String = buf.take() // element or null if empty
```

---

## API

Every ring buffer provides:

```scala
def offer(a: A): Boolean   // insert; returns false if full
def take(): A               // remove; returns null if empty
def size: Int               // approximate element count
def isEmpty: Boolean        // approximate emptiness check
def isFull: Boolean         // approximate fullness check
```

`SpscRingBuffer` additionally provides batch operations:

```scala
def drain(consumer: A => Unit, limit: Int): Int  // drain up to limit elements
def fill(supplier: () => A, limit: Int): Int      // fill up to limit slots
```

All `size`/`isEmpty`/`isFull` values are **approximate** under concurrency—they are snapshots that may be stale by the time the caller acts on them.

---

## Choosing a variant

Use the most constrained variant that fits your threading model:

| Scenario | Recommended |
|----------|-------------|
| Dedicated pipeline: one writer thread, one reader thread | `SpscRingBuffer` |
| Fan-in: many writers, one reader (e.g., logging, event aggregation) | `MpscRingBuffer` |
| Fan-out: one writer, many readers (e.g., work distribution) | `SpmcRingBuffer` |
| General purpose: any number of writers and readers | `MpmcRingBuffer` |

---

## Usage examples

### SPSC with drain/fill

```scala

val buf = SpscRingBuffer[java.lang.Integer](64)

// Producer: batch-fill from a data source
var seq = 0
val filled = buf.fill(() => { seq += 1; Integer.valueOf(seq) }, 32)
println(s"Filled $filled elements")

// Consumer: batch-drain into a processor
val drained = buf.drain(e => println(s"Processing $e"), 32)
println(s"Drained $drained elements")
```

### MPSC fan-in (multiple producers, single consumer)

```scala

val buf = MpscRingBuffer[String](256)

// Multiple producer threads
for (i <- 0 until 4) {
  new Thread(() => {
    for (j <- 0 until 100)
      buf.offer(s"producer-$i: message-$j")
  }).start()
}

// Single consumer thread
new Thread(() => {
  var msg = buf.take()
  while (msg != null) {
    println(msg)
    msg = buf.take()
  }
}).start()
```

### SPMC fan-out (single producer, multiple consumers)

```scala

val buf = SpmcRingBuffer[String](256)

// Single producer thread
new Thread(() => {
  for (i <- 0 until 400)
    while (!buf.offer(s"task-$i")) {} // retry if full
}).start()

// Multiple consumer (worker) threads
for (w <- 0 until 4) {
  new Thread(() => {
    var msg = buf.take()
    while (msg != null) {
      println(s"worker-$w: $msg")
      msg = buf.take()
    }
  }).start()
}
```

### Non-blocking try-once with fallback

```scala

val buf = MpmcRingBuffer[String](64)

// Try to offer without blocking; handle backpressure yourself
if (!buf.offer("data")) {
  // buffer is full — drop, log, or retry later
  println("Buffer full, applying backpressure")
}

// Try to take without blocking
val result = buf.take()
if (result != null) {
  println(s"Got: $result")
} else {
  // buffer is empty — do other work
}
```

---

## Design notes

### FastFlow pattern (SPSC)

`SpscRingBuffer` uses the FastFlow algorithm: the **null/non-null state of an array slot** is the synchronization signal. The producer never reads `consumerIndex`; the consumer never reads `producerIndex`. This minimizes cross-core cache traffic to a single cache line per operation.

A **look-ahead step** (`capacity/4`, capped at 4096) lets the producer batch-check multiple future slots at once, further reducing the frequency of slow-path reads.

### Vyukov/Dmitry sequence buffer (MPMC)

`MpmcRingBuffer` uses a parallel `Long` sequence buffer alongside the data array. Each slot carries a stamp indicating whether it is available for writing or reading. Both `producerIndex` and `consumerIndex` are advanced via CAS, allowing any number of threads on both sides. The minimum capacity is 2 (the algorithm requires at least 2 slots to distinguish written from consumed).

### CAS-based producers (MPSC)

`MpscRingBuffer` follows the JCTools `MpscArrayQueue` design: producers claim a slot via CAS on `producerIndex`, then write the element with release semantics. A cached `producerLimit` avoids reading `consumerIndex` on every offer, reducing cross-core traffic. The consumer side uses relaxed poll semantics—a `null` slot means either empty or a producer mid-write.

### Index-based SPMC

`SpmcRingBuffer` uses index-based capacity checking on the producer side (no CAS needed for a single producer). Consumers use a CAS loop on `consumerIndex`. Consumers read the element *before* the CAS to avoid a race with the producer overwriting the slot. Consumers do not null array slots after reading—the producer overwrites them on the next lap.

### Cache-line padding

All ring buffers use **128-byte padding regions** (16 `Long` fields) between producer and consumer fields. This prevents false sharing on all architectures, including Apple Silicon which uses 128-byte cache lines (most x86 CPUs use 64-byte lines).

The padding is implemented via a class hierarchy:

```
Pad0 → ProducerFields → Pad1 → ConsumerFields → Pad2
```

### VarHandle for memory ordering

All JVM implementations use `java.lang.invoke.VarHandle` (Java 9+) for acquire/release semantics instead of `sun.misc.Unsafe`. This is the recommended modern approach for lock-free data structures on the JVM.

### Power-of-two masking

Capacity must be a power of two. This allows `index & (capacity - 1)` instead of `index % capacity`, which is significantly faster because bitwise AND compiles to a single CPU instruction.

---

## Thread-safety contract

Violating the threading contract (e.g., calling `take` from multiple threads on an `SpscRingBuffer`) results in **undefined behavior**. No runtime check is performed—this is enforced by contract for maximum performance.

| Type | `offer` | `take` |
|------|---------|--------|
| `SpscRingBuffer` | Single producer thread only | Single consumer thread only |
| `SpmcRingBuffer` | Single producer thread only | Any number of consumer threads |
| `MpscRingBuffer` | Any number of producer threads | Single consumer thread only |
| `MpmcRingBuffer` | Any number of producer threads | Any number of consumer threads |

---

## Performance characteristics

| Operation | Complexity |
|-----------|-----------|
| `offer` | Lock-free (SPSC/SPMC: wait-free) |
| `take` | Lock-free (SPSC/MPSC: wait-free) |

**SPSC** is the fastest: no CAS, no locks, minimal cache-line traffic. Use it whenever your threading model allows a dedicated producer-consumer pair.

---

## Cross-platform support

On **Scala.js**, all ring buffer types compile and provide the same API surface. Since Scala.js is single-threaded, the JS implementations use plain reads and writes with no memory ordering primitives.

| Platform | Support |
|----------|---------|
| JVM | Full concurrency support |
| Scala.js | Sequential (same API) |

---

## Docs Critique Subagent Implementation Plan


> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add a maker-critic agent workflow that automatically reviews documentation for content quality, technical accuracy, completeness, and consistency.

**Architecture:** A pure-coordinator skill (`docs-critique`) spawns a maker agent to run a doc creation skill, then spawns a fresh critic agent to review the output. The orchestrator passes critique back to the maker via `SendMessage`. The maker fixes its own work. Critic is freshly spawned each review round.

**Tech Stack:** Claude Code skills, Claude Code agent definitions, Agent tool, SendMessage

**Spec:** `docs/superpowers/specs/2026-03-19-docs-critique-subagent-design.md`

---

## File Structure

| File | Responsibility |
|------|---------------|
| `.claude/agents/docs-critic.md` | Critic agent definition — persona, review dimensions, severity rubric, report format. Read-only tools. |
| `.claude/skills/docs-critique/SKILL.md` | Orchestrating skill — pure coordinator that spawns maker, spawns critic, passes messages, manages iteration loop. Never edits files. |

---

### Task 1: Create the Critic Agent Definition

**Files:**
- Create: `.claude/agents/docs-critic.md`

- [ ] **Step 1: Create the `.claude/agents/` directory**

Run: `mkdir -p /home/milad/sources/scala/zio-blocks-new/.claude/agents`

- [ ] **Step 2: Write the critic agent definition**

Create `.claude/agents/docs-critic.md` with this exact content:

```markdown
---
name: docs-critic
description: Reviews ZIO Blocks documentation for content quality, technical accuracy, completeness, and consistency. Returns a structured report with severity-rated findings. Read-only — never modifies files.
tools: Read, Glob, Grep
model: sonnet
color: purple
---

You are a senior technical writer and Scala developer reviewing ZIO Blocks documentation. You are skeptical by default — assume the document has problems and find them.

## Inputs

You will receive:
1. A documentation file path to review
2. A list of relevant Scala source file paths (read them yourself)
3. A list of related documentation file paths (read them yourself)
4. (Optional) Results from mechanical checks already performed — skip those areas

## Review Dimensions

Evaluate the document across four dimensions:

### Content Quality
- Is there motivation before code? Does the reader understand *why* before *how*?
- Are examples realistic (not toy `foo`/`bar` examples)?
- Is the narrative arc logical — does each section build on the previous?
- Is the writing appropriate for the target audience?
- Is the prose clear and concise?

### Technical Accuracy
- Do API signatures in the doc match the actual source code? (Read the source files to verify.)
- Are code examples correct beyond just compiling? Would they produce the described output?
- Does the described behavior match the actual implementation?
- Are type parameters, return types, and method names accurate?

**Note:** You cannot compile code. Your accuracy checks are static text comparisons against source files. Flag anything you cannot verify with certainty.

### Completeness
- Are all required sections present for this doc type?
  - **Reference pages** (`docs/reference/`): Overview, Construction, Predefined Instances, Operators, Comparison, Advanced Usage
  - **How-to guides** (`docs/guides/`): Prerequisites, Steps, Verification, Troubleshooting
  - **Tutorials** (`docs/tutorials/`): Introduction, Prerequisites, Steps, Summary, Next Steps
  - If the doc type cannot be determined from its path, skip required-sections check and note this in your report.
- Are edge cases and error scenarios mentioned?
- Are cross-references to related types/pages adequate?

### Consistency
- Does terminology match related documentation pages? (Read the related docs to verify.)
- Are there contradictions with other pages?
- Is the tone consistent with the rest of the documentation?

## Severity Rubric

Rate each finding:

- **HIGH**: Factually wrong, misleading, or missing critical content. A reader following this doc would be confused or write buggy code.
- **MEDIUM**: Incomplete, unclear, or inconsistent. A reader could figure it out but shouldn't have to.
- **LOW**: Stylistic nit or minor improvement. A reader wouldn't notice.

## Report Format

You MUST structure your response exactly like this:

## Docs Critic Report: <filename>

### Summary
<1-2 sentence overall assessment>

### Findings

#### [HIGH/dimension] <title>
**Location:** <section name or line range>
**Issue:** <what's wrong>
**Evidence:** <quote from source code or related doc that proves it>
**Suggested fix:** <concrete suggestion>

#### [MEDIUM/dimension] <title>
**Location:** <section name or line range>
**Issue:** <what's wrong>
**Evidence:** <supporting evidence>
**Suggested fix:** <concrete suggestion>

#### [LOW/dimension] <title>
**Location:** <section name or line range>
**Issue:** <what's wrong>
**Suggested fix:** <concrete suggestion>

### Verdict
<APPROVED | ITERATE — N high, M medium issues remain>

## Rules

- Always read the source files before making accuracy claims. Never guess.
- Always read related docs before making consistency claims.
- If you find no issues, return APPROVED with an empty Findings section.
- Never suggest fixes that require information you don't have.
- Never modify any files. You are read-only.
```

- [ ] **Step 3: Verify the file was created correctly**

Run: `head -5 /home/milad/sources/scala/zio-blocks-new/.claude/agents/docs-critic.md`
Expected: The YAML frontmatter starting with `---` and `name: docs-critic`

- [ ] **Step 4: Commit**

```bash
git add .claude/agents/docs-critic.md
git commit -m "feat: add docs-critic agent definition

Read-only agent that reviews documentation for content quality,
technical accuracy, completeness, and consistency. Returns structured
reports with severity-rated findings."
```

---

### Task 2: Create the Orchestrating Skill

**Files:**
- Create: `.claude/skills/docs-critique/SKILL.md`

- [ ] **Step 1: Create the skill directory**

Run: `mkdir -p /home/milad/sources/scala/zio-blocks-new/.claude/skills/docs-critique`

- [ ] **Step 2: Write the orchestrating skill**

Create `.claude/skills/docs-critique/SKILL.md` with this exact content:

````markdown
---
name: docs-critique
description: >
  Run a documentation creation skill with automatic maker-critic review loop.
  Spawns a maker agent to run the skill, then a critic agent to review the output.
  The maker receives critique and fixes its own work. Iterates until approved or
  max 3 rounds. Pure coordinator — never edits files itself.
argument-hint: "<skill-name> <skill-args>"
allowed-tools: Agent, Glob, Grep, Read, SendMessage
---

# Documentation Critique Loop

## Arguments

1. **skill-name** — The documentation skill to run (e.g., `docs-data-type-ref`, `docs-how-to-guide`, `docs-tutorial`, `docs-document-pr`, `docs-enrich-section`, `docs-add-missing-section`)
2. **skill-args** — Arguments to pass to the skill (e.g., `Schema`, `TypeId`)

Example invocation: `/docs-critique docs-data-type-ref Schema`

## Role

You are a **pure coordinator**. You NEVER read, write, or edit documentation files yourself. You ONLY:
1. Spawn agents
2. Pass messages between agents
3. Parse critic reports to decide next action
4. Report final status to the user

## Phase 1: Spawn Maker Agent

Spawn a general-purpose agent via the `Agent` tool:

```
Agent(
  description: "Run doc creation skill",
  prompt: "Run /<skill-name> <skill-args>. Complete all steps of the skill.
           When done, report the absolute path of the generated/modified
           documentation file as the LAST line of your response, in the format:
           DOC_PATH: <absolute-path>"
)
```

Parse the maker's response to extract the doc file path from the `DOC_PATH:` line.

**Error handling:** If the maker does not return a `DOC_PATH:` line, ask the user which file was generated and use that path.

Save the maker's agent ID for later `SendMessage` calls. The `Agent` tool returns an `agentId` in its result — store this value. You will use it as the `to` field in `SendMessage` to route critique back to the maker.

## Phase 2: Gather Critic Context

Using the doc file path from Phase 1, gather context for the critic. You MAY use `Glob` and `Grep` for this phase only — this is the one exception to the "never read files" rule, because you need file paths (not content) to pass to the critic.

1. **Source files** — Extract the type name from the doc path (e.g., `docs/reference/schema.md` → `Schema`). Find source files:
   ```
   Glob: **/<TypeName>.scala
   Grep: "class <TypeName>" or "trait <TypeName>" or "object <TypeName>"
   ```
   Also find test files:
   ```
   Glob: **/<TypeName>Spec.scala or **/<TypeName>Test.scala
   ```

2. **Related docs** — Find sibling pages using two methods:
   - **sidebars.js** (preferred): Read `sidebars.js` and find the array containing the doc's ID. Extract sibling page IDs from the same array. Map IDs to file paths.
   - **Fallback glob** (if sidebars.js parsing fails): Glob the parent directory:
     ```
     Glob: docs/reference/*.md (for reference pages)
     Glob: docs/guides/*.md (for guides)
     Glob: docs/tutorials/*.md (for tutorials)
     ```

3. Collect all found paths into two lists: `source_files` and `related_docs`.

## Phase 3: Spawn Critic Agent

Spawn the `docs-critic` agent:

```
Agent(
  description: "Review documentation",
  subagent_type: "docs-critic",
  prompt: "Review the following documentation file for content quality,
           technical accuracy, completeness, and consistency.

           Documentation file: <doc-path>

           Source files to check accuracy against:
           <list of source_files, one per line>

           Related documentation to check consistency against:
           <list of related_docs, one per line>

           Read each file yourself using the Read tool. Return your
           structured report."
)
```

**Error handling:** If the critic's response does not contain a `### Findings` section or a `### Verdict` line, treat it as an agent failure. Retry by spawning a fresh critic with the same prompt. If the second attempt also fails, report the raw response to the user and stop.

## Phase 4: Triage

Parse the critic's `### Verdict` line:

- **`APPROVED`** → Report success to user. Done.
- **`ITERATE`** with HIGH or MEDIUM findings → Enter Phase 5.
- Only LOW findings → Send LOWs to maker for a single-pass fix:
  ```
  SendMessage(
    to: <maker-agent-id>,
    message: "The documentation critic found minor issues. Fix them if easy,
              skip if not. One commit per fix.
              Commit format: docs(<file-stem>): fix LOW/<dimension> — <description>

              <paste LOW findings here>"
  )
  ```
  Done after maker responds.

## Phase 5: Fix Loop

**Maximum 3 rounds.** Track the current round number.

**Severity-based iteration rules:**
- **HIGH** findings: iterate until fixed (up to round 3)
- **MEDIUM** findings: iterate at most once — if a MEDIUM finding persists after round 1, do not iterate further for it
- After round 1, only HIGH findings drive further iteration

### Each Round:

**Step A — Send critique to maker:**

For round 1, send all HIGH and MEDIUM findings:
```
SendMessage(
  to: <maker-agent-id>,
  message: "The documentation critic found issues that need fixing.
            Fix ALL HIGH and MEDIUM findings below. For each fix:
            - Make a separate git commit
            - Commit format: docs(<file-stem>): fix <SEVERITY>/<dimension> — <description>
            - If multiple findings target the same paragraph, combine into one commit
              using the highest severity level

            <paste HIGH and MEDIUM findings here>"
)
```

For rounds 2+, send only HIGH findings (MEDIUM issues have had their one iteration).

Wait for the maker to respond confirming fixes are done.

**Step B — Spawn fresh critic:**

Spawn a NEW `docs-critic` agent (do NOT reuse the previous one — fresh eyes each round):

```
Agent(
  description: "Re-review documentation round N",
  subagent_type: "docs-critic",
  prompt: <same prompt as Phase 3, identical>
)
```

**Step C — Check verdict:**

- `APPROVED` → Report success to user. Done.
- `ITERATE` with only MEDIUM findings remaining (no HIGH) → Done. MEDIUM had its one iteration.
- `ITERATE` with HIGH findings and round < 3 → Go to next round.
- `ITERATE` and round = 3 → Report remaining issues to user:
  "The documentation was reviewed 3 times. These issues remain unresolved:
   <paste remaining findings>
   Please review manually."

## Output

When done, report to the user:
- Whether the doc was APPROVED or has remaining issues
- How many rounds were needed
- Summary of findings fixed (count by severity)
````

- [ ] **Step 3: Verify the file was created correctly**

Run: `head -10 /home/milad/sources/scala/zio-blocks-new/.claude/skills/docs-critique/SKILL.md`
Expected: The YAML frontmatter with `name: docs-critique`

- [ ] **Step 4: Commit**

```bash
git add .claude/skills/docs-critique/SKILL.md
git commit -m "feat: add docs-critique orchestrating skill

Pure coordinator that spawns a maker agent to run any doc creation
skill, then spawns a fresh critic agent each round to review. Passes
critique back to maker via SendMessage. Severity-gated iteration
with max 3 rounds."
```

---

### Task 3: Manual Smoke Test

**Files:**
- None (testing only)

- [ ] **Step 1: Verify the critic agent is recognized**

Run: `ls -la /home/milad/sources/scala/zio-blocks-new/.claude/agents/docs-critic.md`
Expected: File exists with correct permissions

- [ ] **Step 2: Verify the skill is recognized**

Run: `ls -la /home/milad/sources/scala/zio-blocks-new/.claude/skills/docs-critique/SKILL.md`
Expected: File exists with correct permissions

- [ ] **Step 3: Test invocation with an existing doc**

In a new Claude Code session, run:
```
/docs-critique docs-data-type-ref TypeId
```

Verify:
- The orchestrator spawns a maker agent that runs `/docs-data-type-ref TypeId`
- After the maker finishes, the orchestrator gathers source file and related doc paths
- The orchestrator spawns a `docs-critic` agent with the gathered context
- The critic returns a structured report with `### Findings` and `### Verdict`
- If ITERATE, the orchestrator sends findings to the maker via SendMessage
- The maker fixes issues and commits
- The loop repeats until APPROVED or 3 rounds

- [ ] **Step 4: Verify error handling — malformed critic report**

If the critic returns a malformed report (no `### Verdict`), verify:
- The orchestrator retries once
- If still malformed, it reports the raw response and stops

- [ ] **Step 5: Commit any fixes discovered during smoke test**

If any issues are found in the agent definition or skill during testing, fix them and commit each fix separately.

---

## Design: Subagent-Based Documentation Review


**Date:** 2026-03-19
**Status:** Draft
**Scope:** Add a maker-critic workflow to all documentation creation skills

## Problem

The existing documentation pipeline has layered mechanical checks (writing style, mdoc conventions, compilation gates) but lacks a content-level reviewer. No system verifies that documentation is clear, technically accurate against source code, complete in coverage, or consistent with related pages. These gaps are caught only by human review — if at all.

## Decisions

| Decision | Choice |
|----------|--------|
| Critique level | Full spectrum — content quality, technical accuracy, completeness, consistency |
| Trigger scope | All 6 creation skills automatically |
| Iteration model | Severity-gated — iterate for HIGH/MEDIUM, single pass for LOW |
| Architecture | Maker-critic agent pair — maker produces and fixes, critic reviews, orchestrator coordinates |
| Output format | Structured report + commit-per-fix |
| Critic context | Doc file + source code + related docs |
| Who fixes | The maker agent — it receives critique and fixes its own work |
| Agent lifecycles | Maker stays alive across rounds; critic freshly spawned each round |

## Architecture: Maker Agent + Critic Agent + Orchestrator

Three components:

1. **`.claude/agents/docs-critic.md`** — Reusable agent definition with persona, review dimensions, severity rubric, and report format. Read-only tools (`Read`, `Glob`, `Grep`).
2. **`.claude/skills/docs-critique/SKILL.md`** — Pure coordinator that spawns the maker, spawns the critic, passes messages between them, and manages the iteration loop. Never edits files itself.
3. **Maker agent** — A general-purpose agent spawned by the orchestrator to run the doc creation skill. Stays alive via `SendMessage` to receive critique and fix its own work.

## Agent Definition: `docs-critic`

**Persona:** Senior technical writer and Scala developer. Skeptical by default — assumes the doc has problems and looks for them.

**Tools:** `Read`, `Glob`, `Grep` only. No write access.

**Review Dimensions:**

| Dimension | What it checks |
|-----------|---------------|
| Content Quality | Clarity, narrative flow, example realism, audience fit, motivation before code |
| Technical Accuracy | API signatures match source (static text comparison — critic cannot compile), examples correct beyond compilation, described behavior matches implementation |
| Completeness | Required sections present (detect doc type from path/frontmatter, compare against appropriate section list: `docs-data-type-ref` for reference pages, `docs-how-to-guide` for guides, `docs-tutorial` for tutorials; if doc type cannot be determined, skip required-sections check and note in report), edge cases mentioned, error scenarios covered, cross-references adequate |
| Consistency | Matches terminology/tone of related docs, no contradictions with other pages |

**Severity Rubric:**

| Severity | Definition | Iteration? |
|----------|-----------|------------|
| HIGH | Factually wrong, misleading, or missing critical content. Reader would be confused or write buggy code. | Yes — must iterate until fixed |
| MEDIUM | Incomplete, unclear, or inconsistent. Reader could figure it out but shouldn't have to. | Yes — iterate once |
| LOW | Stylistic nit, minor improvement. Reader wouldn't notice. | No — single pass, fix if easy |

**Report Format:**

```
## Docs Critic Report: <filename>

### Summary
<1-2 sentence overall assessment>

### Findings

#### [HIGH/accuracy] <title>
**Location:** <section name or line range>
**Issue:** <what's wrong>
**Evidence:** <quote from source code or related doc that proves it>
**Suggested fix:** <concrete suggestion>

#### [MEDIUM/completeness] <title>
...

### Verdict
<APPROVED | ITERATE — N high, M medium issues remain>
```

## Orchestrating Skill: `docs-critique`

**Invocation:** `/docs-critique <skill-name> <skill-args>`

Example: `/docs-critique docs-data-type-ref Schema`

The orchestrator is a pure coordinator. It never reads, writes, or edits documentation files. It only spawns agents and passes messages between them.

### Phase 1: Spawn Maker Agent

Spawn a general-purpose agent via the `Agent` tool:

```
"Run /docs-data-type-ref for Schema. Complete all steps of the skill.
 Report the path of the generated documentation file when done."
```

The maker agent runs the full creation skill (research, write, verify, format, integrate) and returns the doc file path.

### Phase 2: Gather Critic Context

Using the doc path returned by the maker, the orchestrator prepares context for the critic:

1. Extract type names from the doc path/filename
2. Find corresponding Scala source files and tests (via `Glob`/`Grep`)
3. Find related doc pages (scan `sidebars.js` for siblings, scan doc for cross-reference links)
4. Collect these as a list of file paths

### Phase 3: Spawn Critic Agent

Spawn `docs-critic` agent with a prompt containing:

- The doc file path (agent reads it via `Read` tool for live content)
- List of relevant source file paths (agent reads them itself)
- List of related doc file paths (agent reads them itself)

Passing paths rather than inline content keeps the prompt small and ensures the agent sees live file state.

**Error handling:** If the agent returns a response without a `### Findings` section or without a `### Verdict` line, treat it as an agent failure. Retry once. If the second attempt also fails, report the raw response to the user and skip the fix loop.

### Phase 4: Triage

Parse the critic's report. Sort findings by severity.

- Any HIGH or MEDIUM → enter fix loop (Phase 5)
- Only LOW → send LOWs to maker for single-pass fix, done
- No findings → APPROVED, done

### Phase 5: Fix Loop (severity-gated, max 3 rounds)

```
Round 1:
  Orchestrator → SendMessage to Maker Agent:
    "The critic found these issues. Fix all HIGH and MEDIUM findings.
     One commit per fix (co-located issues may share a commit).
     Commit format: docs(<file-stem>): fix <severity>/<dimension> — <description>"
  Maker fixes and commits
  Orchestrator → Spawn fresh Critic Agent (fresh eyes)
  Critic re-reviews → returns new report
  If new HIGH/MEDIUM → Round 2

Round 2:
  Orchestrator → SendMessage to Maker Agent with new findings
  Maker fixes and commits
  Orchestrator → Spawn fresh Critic Agent
  Critic re-reviews
  If still HIGH/MEDIUM → Round 3 (final)

Round 3 (cap):
  Orchestrator → SendMessage to Maker Agent with remaining findings
  Maker fixes what it can
  Orchestrator reports any remaining issues to user
```

**Why fresh critic each round:** The critic gets true fresh eyes on each re-review — no anchoring to its prior findings or assumptions. This catches regressions that a persistent critic might overlook because it "already checked that."

**Co-located issues:** When multiple findings target the same paragraph or sentence, the maker combines them into a single commit with the highest severity level. Example: `docs(schema): fix HIGH/accuracy+completeness — correct API and add missing context in Construction`

## Data Flow

```
User invokes: /docs-critique docs-data-type-ref Schema
        │
        ▼
  Orchestrator (pure coordinator, never edits files)
        │
        ├──► Phase 1: Spawn Maker Agent
        │      Maker runs /docs-data-type-ref Schema
        │      Produces doc, commits, returns doc path
        │
        ├──► Phase 2: Gather critic context
        │      Find source files, related docs
        │
        ├──► Phase 3: Spawn Critic Agent (fresh)
        │      Critic reads doc + sources + related docs
        │      Returns structured report
        │
        ├──► Phase 4: Triage by severity
        │      HIGH/MEDIUM found?
        │      │
        │  ┌───┴────┐
        │  │ yes    │ no
        │  ▼        ▼
        │  Phase 5  Single-pass LOWs → Done
        │  │
        │  ├──► SendMessage → Maker: "Fix these issues"
        │  │      Maker fixes, commits (one per fix)
        │  │
        │  ├──► Spawn fresh Critic Agent
        │  │      Critic re-reviews
        │  │      APPROVED? → Done
        │  │      ITERATE? → next round (max 3)
        │  │
        │  └──► Round cap reached → report remaining to user
        │
        └──► Final state: APPROVED or remaining issues reported
```

## Integration Into Creation Skills

**No modifications to existing skills.** The orchestrator wraps them — the maker agent runs the skill as-is, then stays alive to receive critique. This is less invasive and keeps existing skills clean.

The user invokes `/docs-critique <skill-name> <args>` instead of invoking the creation skill directly. The orchestrator handles the rest.

## File Inventory

**New files (2):**

- `.claude/agents/docs-critic.md` — critic agent definition
- `.claude/skills/docs-critique/SKILL.md` — orchestrating skill (pure coordinator)

**Modified files (0):**

No existing skills are modified.

## Token Cost Estimate

- Maker agent (creation + alive across rounds): ~30-50K tokens
- Critic agent per spawn: ~15-25K tokens
- Orchestrator overhead: ~2-5K tokens

Typical scenarios:
- APPROVED on first review: ~50-80K tokens (maker + 1 critic)
- 1 round of fixes: ~70-110K tokens (maker + 2 critics)
- 3 rounds (worst case): ~110-150K tokens (maker + 3 critics)

---

## Documentation Coverage Report


Comprehensive analysis of documentation gaps in ZIO Blocks, combining automated scanning with manual source-code review.

## Summary

| Metric | Count |
|--------|-------|
| Total public types found | 552 |
| Types with documentation | 302 |
| Types lacking documentation | 250 |
| Documentation coverage | 54% |
| Existing reference pages | 24 |
| Missing methods in existing pages | ~44 |
| Missing examples in existing pages | ~39 |
| Conceptual docs (guides, how-tos) | 0 |

---

## Critical: Missing Reference Pages

These are core public API types that users interact with directly. Each needs a dedicated reference page or a substantial new section in an existing page.

- [ ] **`MediaType`** (module `mediatype`) — Public API for media type parsing and matching; essential for content negotiation. The entire `zio.blocks.mediatype` package has zero documentation. **Scope: new page**. Source: `mediatype/shared/src/main/scala/zio/blocks/mediatype/MediaType.scala`
- [ ] **`MediaTypes`** (module `mediatype`) — Predefined media type instances (application/json, text/html, etc.). Should be part of the `MediaType` page. **Scope: new section**. Source: `mediatype/shared/src/main/scala/zio/blocks/mediatype/MediaTypes.scala`
- [ ] **`SchemaExpr`** (module `schema`) — Core trait for expression evaluation on schema-described types; central to the validation DSL. **Scope: new page**. Source: `schema/shared/src/main/scala/zio/blocks/schema/SchemaExpr.scala`
- [ ] **`SchemaError`** (module `schema`) — Primary error type returned from schema operations (ConversionFailed, MissingField, DuplicatedField, ExpectationMismatch, UnknownCase). Users handle these constantly. **Scope: new page or major section in schema.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/SchemaError.scala`
- [ ] **`Into`** (module `schema`) — Core conversion type class appearing in all cross-type transformations. **Scope: new page**. Source: `schema/shared/src/main/scala/zio/blocks/schema/Into.scala`
- [ ] **`JsonPatch`** (module `schema`) — Main public API for JSON patching and diffing operations. **Scope: new section in json.md or patch.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/json/JsonPatch.scala`

---

## High Priority: Incomplete Coverage

### Types needing at least a dedicated section in an existing page

**schema module — Binding subsystem:**
- [ ] **`BindingResolver`** — Essential for type binding resolution; critical for schema rebinding. Referenced in 6 source files. **Scope: new section in binding.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/binding/BindingResolver.scala`
- [ ] **`MapConstructor` / `MapDeconstructor`** — Public traits for customizing map handling in bindings. Referenced in 7 files each. **Scope: new section in binding.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/binding/MapConstructor.scala`
- [ ] **`ConstantConstructor` / `ConstantDeconstructor`** — Binding helpers for constant values. Referenced in 11 files each. **Scope: brief section in binding.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/binding/Constructor.scala`
- [ ] **`RegisterType`** — Type-safe register representation. Referenced in 5 files. **Scope: brief section in registers.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/binding/RegisterType.scala`

**schema module — Derivation subsystem:**
- [ ] **`InstanceOverride`** — Used for customizing schema derivation. Referenced in 8 files. **Scope: new section in type-class-derivation.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/derive/InstanceOverride.scala`
- [ ] **`ModifierOverride`** — Used for overriding modifiers during derivation. Referenced in 3 files. **Scope: new section in type-class-derivation.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/derive/ModifierOverride.scala`

**schema module — Type classes:**
- [ ] **`IsNumeric`** — Type class for arithmetic operations. Referenced in 3 files. **Scope: brief section in schema.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/IsNumeric.scala`
- [ ] **`IsCollection`** — Type class for collection operations. **Scope: brief section in schema.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/IsCollection.scala`
- [ ] **`IsMap`** — Type class for map operations. **Scope: brief section in schema.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/IsMap.scala`
- [ ] **`Reflectable`** — Trait for types with reflectable modifiers. Referenced in 3 files. **Scope: brief section in reflect.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/Reflectable.scala`
- [ ] **`ToStructural`** — Converts nominal to structural schemas. **Scope: brief section in reflect.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/ToStructural.scala`

**schema module — JSON subsystem:**
- [ ] **`Keyable`** — Type class for JSON key support. Referenced in 5 files. **Scope: new section in json.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/json/Keyable.scala`
- [ ] **`NameMapper`** (`CamelCase`, `KebabCase`, `PascalCase`) — Public field name transformations. **Scope: new section in json.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/json/NameMapper.scala`
- [ ] **`JsonCodecError`** — Custom exception for JSON codec errors. Referenced in 7 files. **Scope: brief section in json.md**. Source: `schema/shared/src/main/scala/zio/blocks/schema/json/JsonCodecError.scala`

**typeid module:**
- [ ] **`TypeRepr`** — Central type representation (24 subtypes: ThisType, TypeLambda, Singleton, Repeated, etc.). Referenced across 4-8 files each. **Scope: new section in typeid.md**. Source: `typeid/shared/src/main/scala/zio/blocks/typeid/TypeRepr.scala`
- [ ] **`TypeDefKind`** — Type definition classifier (AbstractType, EnumCase, TypeAlias, etc.). **Scope: new section in typeid.md**. Source: `typeid/shared/src/main/scala/zio/blocks/typeid/TypeDefKind.scala`
- [ ] **`Kind`** and **`Arrow`** — Higher-kinded type expressions. **Scope: new section in typeid.md**. Source: `typeid/shared/src/main/scala/zio/blocks/typeid/Kind.scala`
- [ ] **`Annotation`** — Type annotation metadata (ArrayArg, ClassOf, EnumValue). **Scope: new section in typeid.md**. Source: `typeid/shared/src/main/scala/zio/blocks/typeid/Annotation.scala`
- [ ] **`Owner`** and **`Segment`** — Type ownership path. **Scope: new section in typeid.md**. Source: `typeid/shared/src/main/scala/zio/blocks/typeid/Owner.scala`

**context module:**
- [ ] **`IsNominalType`** — Type class for nominal type extraction. Referenced in 6 files. **Scope: new section in context.md**. Source: `context/shared/src/main/scala/zio/blocks/context/IsNominalType.scala`
- [ ] **`IsNominalIntersection`** — Type class for intersection type handling. Referenced in 4 files. **Scope: new section in context.md**. Source: `context/shared/src/main/scala/zio/blocks/context/IsNominalIntersection.scala`

**Format codec modules:**
- [ ] **`BsonEncoder` / `BsonDecoder` / `BsonCodec`** (schema-bson) — Core encoding/decoding types for BSON. **Scope: expand formats.md BSON section**. Source: `schema-bson/src/main/scala/zio/blocks/schema/bson/BsonTypes.scala`
- [ ] **`MessagePackCodec`** (schema-messagepack) — Public codec for MessagePack. **Scope: expand formats.md MessagePack section**. Source: `schema-messagepack/src/main/scala/zio/blocks/schema/msgpack/MessagePackCodec.scala`
- [ ] **`ThriftCodec`** (schema-thrift) — Public codec for Thrift. **Scope: expand formats.md Thrift section**. Source: `schema-thrift/src/main/scala/zio/blocks/schema/thrift/ThriftCodec.scala`
- [ ] **`ToonReader` / `ToonWriter`** (schema-toon) — Public codec for TOON. Referenced in 9-10 files. **Scope: expand formats.md TOON section**. Source: `schema-toon/src/main/scala/zio/blocks/schema/toon/`

**markdown module:**
- [ ] **`Block`** subtypes (`BlockQuote`, `BulletList`, `CodeBlock`, `HtmlBlock`, `OrderedList`, `ThematicBreak`) — Core markdown AST elements. **Scope: new section in docs.md**. Source: `markdown/shared/src/main/scala/zio/blocks/docs/Block.scala`
- [ ] **`Inline`** subtypes (`Autolink`, `HtmlInline`, `Image`, `HardBreak`, `SoftBreak`) — Core markdown AST elements. **Scope: new section in docs.md**. Source: `markdown/shared/src/main/scala/zio/blocks/docs/Inline.scala`

---

## Medium Priority: Brief Mentions Needed

Types that should be mentioned in related pages but don't need dedicated sections.

**schema module:**
- [ ] `SchemaMetadata` / `Folder` — metadata traversal infrastructure. **Mention in: schema.md**
- [ ] `FromBinding` — type class for binding conversion. **Mention in: binding.md**
- [ ] `UnapplySeq` / `UnapplyMap` — implicit evidence for seq/map operations. **Mention in: binding.md**
- [ ] `OpticCheck` subtypes (`EmptyMap`, `MissingKey`, `SequenceIndexOutOfBounds`, `WrappingError`) — validation results from optic operations. **Mention in: optics.md**
- [ ] `RebindException` — thrown during schema rebinding. **Mention in: binding.md**
- [ ] `JsonDiffer` — JSON diffing utility. **Mention in: json.md**
- [ ] `JsonSchemaType` / `SchemaType` — JSON Schema type system. **Mention in: json-schema.md**
- [ ] `ContextDetector` parsing states — JSON interpolator internals. **Mention in: json.md**
- [ ] `TypeIdSchemas` — hand-rolled schema instances for TypeId. **Mention in: typeid.md**

**chunk module:**
- [ ] `ChunkIterator` — streaming iterator for chunks. **Mention in: chunk.md**
- [ ] `IsText` — type class for chunk-to-string conversion. **Mention in: chunk.md**

**scope module:**
- [ ] `InStack` — trait for stack-like containment. **Mention in: scope.md**

**markdown module:**
- [ ] `Alignment` / `Center` — table column alignment. **Mention in: docs.md**
- [ ] `HeadingLevel` (`H4`, `H5`) — heading levels. **Mention in: docs.md**
- [ ] `TerminalRenderer` — ANSI terminal rendering. **Mention in: docs.md**
- [ ] `MdInterpolatorRuntime` — runtime support for `md"..."`. **Mention in: docs.md**

**schema-toon module:**
- [ ] `Delimiter` / `Comma` / `Pipe` / `Tab` — array delimiters. **Mention in: formats.md**
- [ ] `ArrayFormat` — array encoding strategy. **Mention in: formats.md**
- [ ] `KeyFolding` / `PathExpansion` — TOON reader/writer config. **Mention in: formats.md**

**schema-bson module:**
- [ ] `DiscriminatorField` / `NoDiscriminator` / `WrapperWithClassNameField` — sum type encoding strategies. **Mention in: formats.md**

---

## Documentation Depth Issues

Existing pages that need updates — missing methods, examples, or cross-references.

### Schema (`schema.md`)
- [ ] Missing examples of schema serialization to JSON Schema and back
- [ ] Missing example of caching behavior with `derive(format)`
- [ ] Missing example of using modifiers on schemas
- [ ] No mention of `TypeId` integration in schema derivation
- [ ] Missing reference to schema validation with `DynamicSchema`

### Reflect (`reflect.md`)
- [ ] `Reflect#noBinding` — critical for understanding serialization, not documented
- [ ] `Reflect#transform` — used extensively but not documented
- [ ] `Reflect.Extractors` — pattern matching helpers not documented
- [ ] Missing example of working with `Reflect.Unbound` for serialization
- [ ] Missing example of recursive type handling with `Deferred`
- [ ] Should reference `Binding` more clearly for each reflect type

### Binding (`binding.md`)
- [ ] Missing documentation on `Register` and `RegisterOffset` API — critical for zero-allocation architecture
- [ ] Missing `SpecializedIndexed` documentation for array-based performance
- [ ] No detailed example of `RegisterOffset` calculation
- [ ] No example of implementing custom `SeqConstructor` for new collection types
- [ ] No example of `MapConstructor` / `MapDeconstructor` implementation
- [ ] Missing explanation of why specialized constructors exist

### Chunk (`chunk.md`)
- [ ] `Chunk.BitChunk` and bit operations barely documented
- [ ] No example of the `BitChunk` operations with endianness
- [ ] Missing performance comparison with `Vector` for various operations
- [ ] `Chunk.materialize` automatic triggering conditions are vague
- [ ] Missing `NonEmptyChunk#flatMap` documentation

### JSON (`json.md`)
- [ ] Missing `Keyable` typeclass documentation
- [ ] `Json#diff` operation types not documented
- [ ] No example of complex querying with predicates
- [ ] No example of recursive transformation with `transformDown` vs `transformUp`
- [ ] No example of merging strategies in detail
- [ ] Should reference `JsonSchema` more thoroughly

### Codec (`codec.md`)
- [ ] Missing explanation of `Format` trait interface
- [ ] Missing documentation of `Deriver` pattern for custom codecs
- [ ] No example of implementing a custom `Format`
- [ ] No example of `DerivationBuilder` advanced usage
- [ ] Codec instance caching mechanism not explained
- [ ] JSON codec configuration incomplete — missing validation options

### Patch (`patch.md`)
- [ ] `DynamicPatch` operations not fully documented
- [ ] No example of `modifyKey` on maps with type parameters
- [ ] No example of composing patches across different types
- [ ] Missing example of serialization and storage of patches
- [ ] Should explain relationship to `DynamicValue.diff`

### TypeId (`typeid.md`)
- [ ] `TypeRepr` subtypes barely documented (24 variants)
- [ ] `TypeDefKind` variants incomplete
- [ ] `Member` API — parameter modifiers underdocumented
- [ ] `TermPath` and singleton types barely covered
- [ ] No mention of Scala 2 vs Scala 3 differences in TypeId derivation

### Context (`context.md`)
- [ ] `IsNominalType` typeclass not explained
- [ ] No mention of performance characteristics
- [ ] Missing explanation of why only nominal types are supported
- [ ] Missing example of context composition in larger applications

### Validation (`validation.md`)
- [ ] Composition/chaining methods not documented
- [ ] No example of using validations directly on `PrimitiveType`
- [ ] No example of validation in format derivers
- [ ] Missing guidance on validation composition workarounds
- [ ] Missing integration examples with JSON Schema

---

## Conceptual Gaps

Missing guides, overviews, and tutorials — none of these exist today.

- [ ] **Getting Started Guide** — No quick-start for new users. Should cover: adding dependencies, defining a case class, deriving a schema, encoding/decoding JSON. **Scope: new page `docs/getting-started.md`**
- [ ] **Architecture Overview** — No high-level design document. Should cover: module dependency graph, register-based zero-allocation architecture, the Reflect/Binding/Schema layering, and the Deriver pattern. **Scope: new page `docs/architecture.md`**
- [ ] **How-To: Custom Codec** — No guide for implementing a custom `Format` and its `Deriver`. **Scope: new page `docs/how-to-custom-codec.md`**
- [ ] **How-To: Schema Derivation** — No guide walking through `Schema.derived` vs manual schema construction. **Scope: new section in schema.md or new page**
- [ ] **How-To: Working with DynamicValue** — No guide on converting between typed and dynamic representations. **Scope: new section in dynamic-value.md**
- [ ] **End-to-End Pipeline Example** — No documentation showing the full Schema -> Codec -> Encoding -> Decoding -> Validation pipeline. **Scope: new page or section in index.md**
- [ ] **Migration Guide** — No version migration docs (may not be needed yet if pre-1.0, but placeholder is useful)
- [ ] **Performance Guide** — No guidance on when to use `materialize` on Chunks, binding register allocation strategies, caching behavior in schema derivation, or JSON encoder/decoder performance characteristics

---

## Low Priority / Skip

Internal types that don't need documentation.

| Type | Module | Reason |
|------|--------|--------|
| `LittleEndian` | chunk | Endianness marker for bit operations; specialized internal |
| `ChunkMapBuilder` | chunk | Internal builder behind `ChunkMap.newBuilder` |
| `PlatformSpecific` | schema | Platform-specific trait for JVM/JS split |
| `Extractors` | schema | Internal pattern matching helpers on `Reflect` |
| `AsLowPriorityImplicits` | schema | Implicit resolution priority helper |
| `IntoPrimitiveInstances` / `IntoContainerInstances` | schema | Implicit instance providers (infrastructure) |
| `HasInstances` | schema | Derivation infrastructure |
| `UnapplySeqLowPriority` | schema | Implicit priority helper |
| `Leaf` (in Doc) | schema | Internal documentation node type |
| `Folder` | schema | Internal metadata traversal helper |
| All `*Delta` / `*Dummy` types in `DynamicPatch` | schema | Internal patch operation representations (BigDecimalDelta, ByteDelta, DoubleDelta, DurationDelta, DurationDummy, FloatDelta, InstantDelta, IntDelta, LocalDateDelta, LocalDateTimeDelta, LongDelta, PeriodDelta, PeriodDummy, ShortDelta) |
| All `PathParser` error types | schema | Internal parser errors (EmptyChar, InvalidEscape, InvalidIdentifier, InvalidSyntax, IntegerOverflow, MultiCharLiteral, UnexpectedChar, UnexpectedEnd, UnterminatedChar, UnterminatedString) |
| `ContextDetector` / parsing states | schema | Internal JSON interpolator state machine (AfterValue, ExpectingColon, ExpectingKey, ExpectingValue, InString, TopLevel) |
| `JsonSchemaToReflect` helpers | schema | Internal conversion types (FieldVariant, KeyVariant, MapShape, OptionOf, PrimKind) |
| `CaseInfo` / `EnumInfo` | schema | Internal JSON codec deriver helpers |
| `DynamicValueMergeStrategy` / `KeepLeft` | schema | Internal merge strategy implementation |
| `DynamicValueSelection` | schema | Internal selection helper |
| `NoBinding` | schema | Internal marker type |
| `ReflectPrinter` | schema | Internal debug printing |
| `Registry` / `Registry.Entry` | schema | Internal binding resolver storage |
| `ObjectIdSupport` | schema-bson | Internal BSON ObjectId helper |
| `BsonBuilder` / `BsonTrace` / `EncoderContext` / `BsonDecoderContext` | schema-bson | Internal codec implementation |
| `MessagePackCodecDeriver` | schema-messagepack | Internal deriver |
| `MessagePackReader` / `MessagePackWriter` | schema-messagepack | Internal binary readers/writers |
| `Mixed` / `UniformRecords` | schema-toon | Internal codec strategy types |
| `ArrayHeader` | schema-toon | Internal reader state |
| `Off` | schema-toon | Internal config value |
| All `scope/internal/*` types | scope | Internal error rendering (Colors, DepNode, DepStatus, ErrorMessages, Found, Missing, Pending, ProviderInfo) |
| `Destroyed` / `Uninitialized` | scope | Internal resource lifecycle states |
| `FlatMap` / `ZSink` / `ZSinkFiber` / `ZSource` / `ZSourceFiber` | streams | Experimental/WIP module |
| `TypeIdInstances` | typeid | Internal implicit provider trait |
| `TypeIdPrinter` | typeid | Internal rendering utility |
| `Owners` | typeid | Internal namespace helper |
| All `*Const` types in `TypeRepr` | typeid | Internal literal type representations (BooleanConst, CharConst, ClassOfConst, DoubleConst, FloatConst, IntConst, LongConst, NullConst, StringConst, UnitConst) |
| `AnyKindType` / `AnyType` / `NothingType` / `NullType` / `UnitType` | typeid | Internal special type representations |

---

## Suggested Actions

Ordered TODO checklist grouped by module, with estimated scope.

### Conceptual Documentation (highest impact)

1. - [ ] Write **Getting Started Guide** — `docs/getting-started.md` — *new page*
2. - [ ] Write **Architecture Overview** — `docs/architecture.md` — *new page*
3. - [ ] Write **End-to-End Pipeline Example** — Schema -> Codec -> Encode -> Decode -> Validate — *new page or section*

### Module: `mediatype` (entirely undocumented)

4. - [ ] Write **MediaType reference page** — `docs/reference/media-type.md` — *new page*

### Module: `schema` (core gaps)

5. - [ ] Write **SchemaError reference** — error types, handling patterns — *new page or new section in schema.md*
6. - [ ] Write **SchemaExpr reference** — expression DSL, validation expressions — *new page*
7. - [ ] Write **Into reference** — cross-type conversions — *new page*
8. - [ ] Add **BindingResolver, MapConstructor, MapDeconstructor** sections to `binding.md` — *update existing*
9. - [ ] Add **InstanceOverride, ModifierOverride** sections to `type-class-derivation.md` — *update existing*
10. - [ ] Add **IsNumeric, IsCollection, IsMap** section to `schema.md` — *update existing*
11. - [ ] Add **Keyable, NameMapper** section to `json.md` — *update existing*
12. - [ ] Add **JsonPatch operations** section to `json.md` or `patch.md` — *update existing*
13. - [ ] Add **Reflectable, ToStructural** sections to `reflect.md` — *update existing*
14. - [ ] Expand **Register/RegisterOffset** explanation in `binding.md` and `registers.md` — *update existing*

### Module: `typeid` (many subtypes undocumented)

15. - [ ] Add **TypeRepr** section with key subtypes to `typeid.md` — *update existing*
16. - [ ] Add **TypeDefKind** section to `typeid.md` — *update existing*
17. - [ ] Add **Kind/Arrow** section to `typeid.md` — *update existing*
18. - [ ] Add **Annotation** subtypes section to `typeid.md` — *update existing*
19. - [ ] Add **Owner/Segment** section to `typeid.md` — *update existing*

### Module: `context`

20. - [ ] Add **IsNominalType, IsNominalIntersection** section to `context.md` — *update existing*

### Module: `markdown`

21. - [ ] Add **Block subtypes** reference section to `docs.md` — *update existing*
22. - [ ] Add **Inline subtypes** reference section to `docs.md` — *update existing*
23. - [ ] Add **Alignment, HeadingLevel, TerminalRenderer** mentions to `docs.md` — *update existing*

### Module: `chunk`

24. - [ ] Add **BitChunk operations** section to `chunk.md` — *update existing*
25. - [ ] Add **ChunkIterator** mention to `chunk.md` — *update existing*

### Format modules (expand `formats.md`)

26. - [ ] Expand **BSON section** in `formats.md` with BsonEncoder/BsonDecoder API — *update existing*
27. - [ ] Expand **MessagePack section** in `formats.md` with MessagePackCodec API — *update existing*
28. - [ ] Expand **Thrift section** in `formats.md` with ThriftCodec API — *update existing*
29. - [ ] Expand **TOON section** in `formats.md` with ToonReader/ToonWriter, Delimiter, config — *update existing*

### Depth improvements (existing pages)

30. - [ ] Add missing **Reflect#transform** and **Reflect#noBinding** to `reflect.md`
31. - [ ] Add missing **Format trait interface** and **custom Format** example to `codec.md`
32. - [ ] Add missing **DynamicPatch** operations and **patch serialization** example to `patch.md`
33. - [ ] Add missing **Validation composition** guidance and **JSON Schema integration** to `validation.md`
34. - [ ] Add missing **TypeRepr pattern matching** and **Scala 2/3 differences** to `typeid.md`
35. - [ ] Add missing **Cache behavior** and **intersection type** examples to `context.md`

---

*Report generated on 2026-02-13. Scan performed by `scan-undocumented.sh`; manual enrichment by AI review of 250 undocumented types across 12 modules and 10 existing reference pages.*

---

## Getting Started with ZIO Bson

[ZIO Bson](https://github.com/zio/zio-bson) is BSON library with tight ZIO integration.

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-bson/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-bson_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-bson_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-bson_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-bson_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-bson-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-bson-docs_2.13) [![ZIO Bson](https://img.shields.io/github/stars/zio/zio-bson?style=social)](https://github.com/zio/zio-bson)

## Introduction

The goal of this project is to create the best all-round BSON library for Scala:

- **Native BSON support** to avoid intermediate JSON conversions and support BSON types.
- **Future-Proof**, prepared for Scala 3 and next-generation Java.
- **Simple** small codebase, concise documentation that covers everything.
- **Helpful errors** are readable by humans and machines.
- **ZIO Integration** so nothing more is required.

## Installation

In order to use this library, we need to add the following lines in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-bson" % "1.0.11"
libraryDependencies += "dev.zio" %% "zio-bson-magnolia" % "1.0.11"
```

## zio-schema support

`zio-bson-magnolia` projects provides typeclass derivation only for `scala` `2.13`.
You can use [zio-schema-bson](https://github.com/zio/zio-schema/) instead to get typeclass derivation on `scala` `2.12`, `2.13` and `3`.

## Example

All the following code snippets assume that the following imports have been declared

```scala

```

### Declaring codecs

Use `DeriveBsonCodec.derive` to get a codec for your case class or sealed trait:

```scala

case class Banana(curvature: Double)

object Banana {
  implicit val codec: BsonCodec[Banana] = DeriveBsonCodec.derive
}
```

### Converting to BsonValue

To use values in `Filter` of `Update` expressions you can convert them to `BsonValue` this way:

```scala
"str".toBsonValue

Banana(0.2).toBsonValue

def method[T: BsonEncoder](value: T): BsonDocument = doc("key" -> value.toBsonValue)
```

### Creating CodecProvider

To get `CodecProvider` for your `BsonCodec` use `zioBsonCodecProvider`:

```scala
val codecProvider = zioBsonCodecProvider[Banana]
```

### Parsing BsonValue

In general `CodecProvider` should parse your case class without intermediate `BsonValue` representation.
But you can parse `BsonValue` any way:

```scala

val bsonVal: BsonValue = doc("curvature" -> double(0.2))

bsonVal.as[Banana]
```

## Type Mappings

See [Type Mappings](types.md) for the full list of supported Scala and Java types, their primary BSON types, and decoder fallbacks.

## Errors

Bad BSON will produce an error with path and contextual information

```
scala> doc("curvature" -> Array[Byte](1, 2, 3).toBsonValue).as[Banana]
val res: Either[String,Banana] = Left(.curvature: Expected BSON type Double, but got BINARY.)
```

## Configuration

You can configure typeclass derivation with annotations.

```scala

sealed trait Fruit

object Fruit {
  case class Banana(curvature: Double) extends Fruit
  case class Apple(poison: Boolean) extends Fruit
  
  implicit val codec: BsonCodec[Fruit] = DeriveBsonCodec.derive
}

val bsonFruit = doc( "Banana" -> doc( "curvature" -> double(0.5) ))

bsonFruit.as[Fruit]
//Right(Banana(0.5))

@bsonDiscriminator("$type")
sealed trait FruitConfigured

object FruitConfigured {
  case class Banana(curvature: Double) extends FruitConfigured
  @bsonHint("custom_type_name")
  case class Apple(@bsonField("is_poisoned") poison: Boolean) extends FruitConfigured

  implicit val codec: BsonCodec[FruitConfigured] = DeriveBsonCodec.derive
}

val bsonFruitConfigured = doc(
  "$type" -> str("custom_type_name"),
  "is_poisoned" -> bool(true)
)

bsonFruitConfigured.as[FruitConfigured]
//Right(Apple(true))
```

---

## Type Mappings

## Scala / Java to BSON Type Mappings

ZIO Bson provides `BsonEncoder` and `BsonDecoder` typeclass instances for a wide range of Scala and Java types.
The encoder determines the primary BSON type a value is written as.
The decoder accepts the primary type and, where it makes sense, additional fallback BSON types.

| Scala / Java Type | Encoder → BSON | Decoder fallbacks |
|---|---|---|
| `Boolean` | `BOOLEAN` | — |
| `Byte` | `INT32` | `INT64` (range check), `DOUBLE` (lossless), `DECIMAL128` (lossless), `STRING` |
| `Short` | `INT32` | `INT64` (range check), `DOUBLE` (lossless), `DECIMAL128` (lossless), `STRING` |
| `Int` | `INT32` | `INT64` (range check), `DOUBLE` (lossless), `DECIMAL128` (lossless), `STRING` |
| `Long` | `INT64` | `INT32`, `DOUBLE` (lossless), `DECIMAL128` (lossless), `STRING` |
| `Float` | `DOUBLE` | `INT32`, `INT64`, `DECIMAL128` (lossless), `STRING` |
| `Double` | `DOUBLE` | `INT32`, `INT64`, `DECIMAL128` (lossless), `STRING` |
| `java.math.BigDecimal` | `DECIMAL128` | `INT32`, `INT64`, `DOUBLE`, `STRING` |
| `BigDecimal` | `DECIMAL128` | `INT32`, `INT64`, `DOUBLE`, `STRING` |
| `java.math.BigInteger` | `STRING` | `INT32`, `INT64`, `DOUBLE` (lossless), `DECIMAL128` (lossless) |
| `BigInt` | `STRING` | `INT32`, `INT64`, `DOUBLE` (lossless), `DECIMAL128` (lossless) |
| `String` | `STRING` | `SYMBOL` |
| `Char` | `STRING` | `SYMBOL` |
| `scala.Symbol` | `STRING` | `SYMBOL` |
| `UUID` | `STRING` | — |
| `java.util.Currency` | `STRING` | — |
| `DayOfWeek` | `STRING` | — |
| `Month` | `STRING` | — |
| `MonthDay` | `STRING` | — |
| `Year` | `INT32` | `INT64` (range check), `DOUBLE` (lossless), `DECIMAL128` (lossless), `STRING` |
| `YearMonth` | `STRING` | — |
| `ZoneId` | `STRING` | — |
| `ZoneOffset` | `STRING` | — |
| `Duration` | `STRING` | — |
| `Period` | `STRING` | — |
| `Instant` | `DATE_TIME` | — |
| `LocalDate` | `DATE_TIME` | — |
| `LocalDateTime` | `DATE_TIME` | — |
| `LocalTime` | `DATE_TIME` | — |
| `OffsetDateTime` | `DOCUMENT` | — |
| `OffsetTime` | `DOCUMENT` | — |
| `ZonedDateTime` | `DOCUMENT` | — |
| `org.bson.types.ObjectId` | `OBJECT_ID` | — |
| `Option[A]` | same as `A` / `NULL` | — |
| `Array[Byte]` | `BINARY` | — |
| `Iterable[Byte]` subtype | `BINARY` | — |
| `NonEmptyChunk[Byte]` | `BINARY` | — |
| `Array[A]` (non-byte) | `ARRAY` | — |
| `Iterable[A]` subtype (non-byte) | `ARRAY` | — |
| `NonEmptyChunk[A]` (non-byte) | `ARRAY` | — |
| `Map[K, V]` | `DOCUMENT` | — |
| `T <: BsonValue` | any (pass-through) | — |

## Special Cases

- **`Byte` and `Short`** are decoded via the `Int` decoder followed by a range check using `validateIntRange`. All BSON numeric types and `STRING` are accepted, but the decoded value must fit within the type's range, otherwise a decoder error is returned.
- **`BigInt` and `java.math.BigInteger`** encode to `STRING` but decode from all numeric BSON types. Conversion from `DOUBLE` and `DECIMAL128` requires the value to be a lossless integer (no fractional part).
- **`String`** decoder also accepts `SYMBOL` for backward compatibility with legacy BSON documents.
- **`Char`** decoder requires the string to be exactly one character long.
- **`OffsetDateTime`** and **`OffsetTime`** encode as a `DOCUMENT` with two fields: `date_time` (`DATE_TIME`) and `offset` (`STRING`).
- **`ZonedDateTime`** encodes as a `DOCUMENT` with two fields: `date_time` (`DATE_TIME`) and `zone_id` (`STRING`).
- **`Option[A]`** encodes `None` as `NULL` and `Some(a)` using the encoder for `A`. The decoder returns `None` for `NULL` and `Some(a)` for any value accepted by the decoder for `A`.

## BSON Types Without Scala Mappings

The following BSON types have no dedicated Scala or Java codec in this library. They are accessible only via the `T <: BsonValue` pass-through encoder/decoder using the underlying MongoDB Java driver:

| BSON Type | Notes |
|---|---|
| `SYMBOL` | Deprecated BSON type; accepted by the `String` decoder for backward compatibility, but no encoder exists |
| `UNDEFINED` | Deprecated BSON type |
| `REGULAR_EXPRESSION` | — |
| `DB_POINTER` | Deprecated BSON type |
| `JAVASCRIPT` | — |
| `JAVASCRIPT_WITH_SCOPE` | — |
| `TIMESTAMP` | MongoDB internal replication timestamp |
| `MIN_KEY` | — |
| `MAX_KEY` | — |
| `END_OF_DOCUMENT` | Internal sentinel used by the reader loop; never a value |

---

## Cache Statistics

ZIO Cache automatically tracks various statistics associated with the cache, such as the number of cache hits and misses and the current size of the cache, to allow you to assess the effectiveness of the cache. You can access these statistics by using the `cacheStats` operator on `Cache`.

```scala

trait CacheStats {
  def hits: Long
  def misses: Long
  def size: Int
}

trait Cache[-Key, +Error, +Value] {
  def cacheStats: UIO[CacheStats]
}
```

The cache statistics represent a snapshot as of a point in time. Note that the cache statistics are designed for aggregate analysis and may not be fully consistent.

More cache statistics will be added over time.

---

## Cache

A cache is defined in terms of a lookup function, a capacity, and a time to live.

```scala

trait Lookup[-Key, -Environment, +Error, +Value]

trait Cache[-Key, +Error, +Value] {
  def get(k: Key): IO[Error, Value]
}

object Cache {

  def make[Key, Environment, Error, Value](
    capacity: Int,
    timeToLive: Duration,
    lookup: Lookup[Key, Environment, Error, Value]
  ): ZIO[Environment, Nothing, Cache[Key, Error, Value]] =
    ???
}
```

The most fundamental operator on a cache is `get`, which either returns the value in the cache if it exists or else computes a new value with the lookup function, puts it in the cache, and returns it.

## Concurrent Access

The cache is guaranteed to be safe for concurrent access. In addition, getting a value from the cache is efficient under concurrency.

If two concurrent processes attempt to get the same value and it does not exist in the cache, the value will be computed once and provided to both concurent processes as soon as it is available. The concurrent processes will semantically block for the value to become available but no underlying operating system threads will block as a result of this operation.

In the event that the computation of the lookup function fails or is interrupted that will automatically be propagated to any concurrent processes waiting for the same value.

Failures will be cached and made available to subsequent calls to `get` to avoid repeatedly computing the same failed value. In the event of interruption the key will be removed from the cache so subsequent calls to `get` will attempt to compute the value again.

## Capacity

A cache is constructed with a specified capacity. When the cache is at capacity the least recently accessed values will be removed first.

Note that the size of the cache may slightly exceed the specified capacity between operations.

## Time To Live (TTL)

A cache also allows specifying a time to live. The cache guarantees that no value will be returned from the cache if its age is greater than or equal to the specified time to live.

The age is calculated based on the interval between when the value was loaded in the cache and when it is accessed.

## Operators

In addition to `get`, `Cache` provides a variety of other operators.

```scala
trait Cache[-Key, +Error, +Value] {
  def cacheStats: UIO[CacheStats]
  def contains(key: Key): UIO[Boolean]
  def entryStats(key: Key): UIO[Option[EntryStats]]
  def invalidate(key: Key): UIO[Unit]
  def invalidateAll: UIO[Unit]
  def refresh(key: Key): IO[Error, Unit]
  def size: UIO[Int]
}
```
The `refresh` operator is similar to `get`. The difference is `refresh` triggers a re-computation of the value without invalidating it. This allows any request to the associated key to be served while the value is being re-computed / retrieved by the lookup function. Note: `refresh` always triggers the lookup function, disregarding the last `Error`. 

The `size` operator returns the current size of the cache. Under concurrent access the size should be regarded as only approximate since by the time we observe a given size the cache may have a different size based on concurrent insertions or removals.

Similarly, the `contains` operator returns whether a value associated with the specified key exists in the cache. Again, under concurrent access `contains` is guaranteed to return whether the cache contains a value associated with the specified key as of a given point in time but that value may be concurrently added or removed immediately after that.

There are also the `cacheStats` and `entryStats` operators which allow obtaining a snapshot of statistics either for the cache itself or for a specified entry. See the sections on cache statistics and entry statistics for further discussion of this functionality.

The `invalidate` and `invalidateAll` operators can be used to evict a value associated with a specified key and evict all values, respectively.

---

## Entry Statistics

ZIO Cache also tracks statistics associated with each entry, such as when the entry was last accessed. You can access the statistics for a specified entry using the `entryStats` operator on `Cache`.

```scala

trait EntryStats {
  def loaded: Instant
}

trait Cache[-Key, +Error, +Value] {
  def entryStats: UIO[EntryStats]
}
```

More entry statistics will be added over time.

---

## Introduction to ZIO Cache

ZIO Cache is a library that makes it easy to optimize the performance of our application by caching values.

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-cache/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-cache_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-cache_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-cache_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-cache_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-cache-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-cache-docs_2.13) [![ZIO Cache](https://img.shields.io/github/stars/zio/zio-cache?style=social)](https://github.com/zio/zio-cache)

## Introduction

Sometimes we may call or receive requests to do overlapping work. Assume we are writing a service that is going to handle all incoming requests. We don't want to handle duplicate requests. Using ZIO Cache we can make our application to be more **performant** by preventing duplicated works.

Some key features of ZIO Cache:

- **Compositionality** — If we want our applications to be **compositional**, different parts of our application may do overlapping work. ZIO Cache helps us to stay benefit from compositionality while using caching.

- **Unification of Synchronous and Asynchronous Caches** — Compositional definition of cache in terms of _lookup function_ unifies synchronous and asynchronous caches. So the lookup function can compute value either synchronously or asynchronously.

- **Deep ZIO Integration** — ZIO Cache is a ZIO native solution. So without losing the power of ZIO it includes support for _concurrent lookups_, _failure_, and _interruption_.

- **Caching Policy** — Using caching policy, the ZIO Cache can determine when values should/may be removed from the cache. So, if we want to build something more complex and custom we have a lot of flexibility. The caching policy has two parts and together they define a whole caching policy:

    - **Priority (Optional Removal)** — When we are running out of space, it defines the order that the existing values **might** be removed from the cache to make more space.

    - **Evict (Mandatory Removal)** — Regardless of space when we **must** remove existing values because they are no longer valid anymore. They might be invalid because they do not satisfy business requirements (e.g., maybe it's too old). This is a function that determines whether an entry is valid based on the entry and the current time.

- **Composition Caching Policy** — We can define much more complicated caching policies out of much simpler ones.

- **Cache/Entry Statistics** — ZIO Cache maintains some good statistic metrics, such as entries, memory size, hits, misses, loads, evictions, and total load time. So we can look at how our cache is doing and decide where we should change our caching policy to improve caching metrics.

## How to Define a Cache?

A cache is defined in terms of a lookup function that describes how to compute the value associated with a key if a value is not already in the cache.

```scala

trait Lookup[-Key, -Environment, +Error, +Value] {
  def lookup(key: Key): ZIO[Environment, Error, Value]
}
```

The lookup function takes a key of type `Key` and returns a `ZIO` effect that requires an environment of type `Environment` and can fail with an error of type `Error` or succeed with a value of type `Value`. Because the lookup function returns a `ZIO` effect it can describe both synchronous and asynchronous workflows.

We construct a cache using a lookup function as well as a maximum size and a time to live.

```scala
trait Cache[-Key, +Error, +Value] {
  def get(k: Key): IO[Error, Value]
}

object Cache {

  def make[Key, Environment, Error, Value](
    capacity: Int,
    timeToLive: Duration,
    lookup: Lookup[Key, Environment, Error, Value]
  ): ZIO[Environment, Nothing, Cache[Key, Error, Value]] =
    ???
}
```

Once we have created a cache the most idiomatic way to work with it is the `get` operator. The `get` operator will return the current value in the cache if it exists or else compute a new value, put it in the cache, and return it.

If multiple concurrent processes get the value at the same time the value will only be computed once, with all of the other processes receiving the computed value as soon as it is available. All of this will be done using ZIO's fiber based concurrency model without ever blocking any underlying operating system threads.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-cache" % "0.2.8"
```

## Example

In this example, we are calling `timeConsumingEffect` three times in parallel with the same key. The ZIO Cache runs this effect only once. So the concurrent lookups will suspend until the value being computed is available:

```scala

object ZIOCacheExample extends ZIOAppDefault {
  def timeConsumingEffect(key: String) =
    ZIO.sleep(5.seconds).as(key.hashCode)

  def run =
    for {
      cache <- Cache.make(
        capacity = 100,
        timeToLive = Duration.Infinity,
        lookup = Lookup(timeConsumingEffect)
      )
      result <- cache
        .get("key1")
        .zipPar(cache.get("key1"))
        .zipPar(cache.get("key1"))
      _ <- ZIO.debug(
        s"Result of parallel execution of three effects with the same key: $result"
      )

      hits <- cache.cacheStats.map(_.hits)
      misses <- cache.cacheStats.map(_.misses)
      _ <- ZIO.debug(s"Number of cache hits: $hits")
      _ <- ZIO.debug(s"Number of cache misses: $misses")
    } yield ()

}
```

The output of this program should be as follows:

```
Result of parallel execution three effects with the same key: ((3288498,3288498),3288498)
Number of cache hits: 2
Number of cache misses: 1
```

## Resources

- [Compositional Caching](https://www.youtube.com/watch?v=iFeTUhYpPLs) by Adam Fraser (December 2020) — In this talk, Adam will introduce ZIO Cache, a new library in the ZIO ecosystem that provides a drop-in caching solution for ZIO applications. We will see how ZIO’s support for asynchrony and concurrent lets us implement a cache in terms of a single lookup function and how we get many other things such as typed errors and compositional caching policies for free. See how easy it can be to add caching to your ZIO application!

---

## Lookup

A `Lookup` is a lookup function that, given a key of type `Key`, knows how to compute a value of type `Value`, requiring an environment of type `Environment` and potentially failing with an error of type `Error`.

```scala

trait Lookup[-Key, -Environment, +Error, +Value] {
  def lookup(key: Key): ZIO[Environment, Error, Value]
}
```

You can think of a key as essentially an effectual function to compute a value. And in fact you can construct a `Lookup` from any effectual function using the `apply` method on `Lookup`.

```scala
object Lookup {

  def apply[Key, Environment, Error, Value](
    f: Key => ZIO[Environment, Error, Value]
  ): Lookup[Key, Environment, Error, Value] =
    ???
}
```

Since the lookup function can return a `ZIO` effect it can either return its result synchronously or asynchronously. It can also use an environment and potentially fail with an error.

In short, if you can describe it with a ZIO effect you can use it as the lookup function for a cache.

---

## Args

The `Args` data type models command-line arguments. Arguments are name-less, position-based parameters passed to a command-line program that modify its behavior. For example, the Git CLI has a command named `git clone` that creates a copy of an existing repository. It has an argument called "repository" that represents the existing repository's path. To clone the **ZIO CLI** repository, it can be called this way:
```
git clone https://github.com/zio/zio-cli.git
```
where `https://github.com/zio/zio-cli.git` is the path of the repository.

In **ZIO CLI**, Arguments are represented by instances of class `Args[_]`. `Args[A]` is a description of the process of constructing an instance of `A` from a valid input of the CLI. It is not yet a specified argument for the CLI. In other words, an `Args[A]` defines a collection of valid arguments for a command and a way to construct a value `A` from them.

## Construction of basic Args
**ZIO CLI** offers a variety of methods to create basic `Args` that takes a string from the user as input and transform it into a value of the corresponding type. The benefit of using these methods is that it is automatically checked if they describe a value of the given type and, in that case, are transformed into it. In case of an invalid input, a `ValidationError` will be thrown. Note that each type has its own validation function.

An `Args` instance of a basic type carries a name attribute. This can be specified when being created, but it can also be skipped. In this last case, the name will be that of the type. An example with Boolean type:
```scala

val name = "name"

Args.bool(name) // Boolean Args with name
Args.bool       // Boolean Args named "boolean"
```

### Boolean Args
Produces an `Args[Boolean]`. It accepts as input `true` or `false`, while other pieces of text are not valid.
```scala
Args.bool(name)
```
### Path Args
They are used to produce an `Args[Path]`.
- Produces `Args[java.nio.file.Path]`.
```scala
Args.file(name) // Used for a file
```
- Produces `Args[java.nio.file.Path]`.
```scala
Args.directory(name) // Used for a directory
```
By default, the `CliApp` accepts paths pointing to files or directories that might or not exist. If you need to work with one of them exclusively you can use the trait `Exists`:
```scala
// Path can point to both existing or non-existing files or directories
Args.file(name, exists = Exists.Either)
Args.directory(name, exists = Exists.Either)

// Path must point to existing files or directories
Args.file(name, exists = Exists.Yes)
Args.directory(name, exists = Exists.Yes)

// Path must point to non-existing files or directories
Args.file(name, exists = Exists.Yes)
Args.directory(name, exists = Exists.Yes)
```
### Text Args
Produces an `Args[String]`.
```scala
Args.text(name)
```
### Numeric Args
Numeric `Args` produce `BigInt` and `BigDecimal`, so they can represent numbers of arbitrary size and precision. Note that the CLI checks that the input really represents a number.
- Produces `Args[BigDecimal]`.
```scala
Args.decimal(name)
```

- Produces `Args[BigInt]`.
```scala
Args.integer(name)
```

### Date/Time Args
The following methods produce `Args` whose type parameter is a Date/Time type of the `java.time` library.
- Produces `Args[java.time.Duration]`. The input must be a time-based amount of time in the ISO-8601 format, such as 'P1DT2H3M'.
```scala
Args.duration(name)
```
- Produces `Args[java.time.Instant]`. The input must be an instant in time in UTC format, such as 2007-12-03T10:15:30.00Z.
```scala
Args.instant(name)
```
- Produces `Args[java.time.LocalDate]`. The input must be a date in ISO_LOCAL_DATE format, such as 2007-12-03.
```scala
Args.localDate(name)
```
- Produces `Args[java.time.LocalDateTime]`. The input must be a date-time without a time-zone in the ISO-8601 format, such as 2007-12-03T10:15:30.
```scala
Args.localDateTime(name)
```
- Produces `Args[java.time.LocalTime]`. The input must be a time without a time-zone in the ISO-8601 format, such as 10:15:30.
```scala
Args.localTime(name)
```
- Produces `Args[java.time.MonthDay]`. The input must be a month-day in the ISO-8601 format such as 12-03.
```scala
Args.monthDay(name)
```
- Produces `Args[java.time.OffsetDateTime]`. The input must be a date-time with an offset from UTC/Greenwich in the ISO-8601 format, such as 2007-12-03T10:15:30+01:00.
```scala
Args.offsetDateTime(name)
```
- Produces `Args[java.time.OffsetTime]`. The input must be a time with an offset from UTC/Greenwich in the ISO-8601 format, such as 10:15:30+01:00.
```scala
Args.offsetTime(name)
```
- Produces `Args[java.time.Period]`. The input must be a date-based amount of time in the ISO-8601 format, such as 'P1Y2M3D'.
```scala
Args.period(name)
```
- Produces `Args[java.time.Year]`. The input must be a year in the ISO-8601 format, such as 2007.
```scala
Args.year(name)
```
- Produces `Args[java.time.YearMonth]`. The input must be a year-month in the ISO-8601 format, such as 2007-12.
```scala
Args.yearMonth(name)
```
- Produces `Args[java.time.ZonedDateTime]`. The input must be a date-time with a time-zone in the ISO-8601 format, such as 2007-12-03T10:15:30+01:00 Europe/Paris.
```scala
Args.zonedDateTime(name)
```
- Produces `Args[java.time.ZoneId]`. The input must be a time-zone ID, such as Europe/Paris.
```scala
Args.zoneId(name)
```
- Produces `Args[java.time.ZoneOffset]`. The input must be a time-zone offset from Greenwich/UTC, such as +02:00.
```scala
Args.zoneOffset(name)
```

## Combining and transforming Args
When constructing a command, you can specify only one argument. Thus, to create more complex `Args` it is necessary to use the following methods of the trait `Args`:
```scala
trait Args[A] {
  def ++[B](that: Args[B]): Args[(A, B)]              // Zip two args
  def +[A1 >: A]: Args[::[A1]]                        // Requires a non-empty list of Args of type A1
  def * : Args[List[A]]                               // Requires a list of Args of same type
  def atLeast(min: Int): Args[List[A]]                // Requires a list of Args with at least min elements
  def atMost(max: Int): Args[List[A]]                 // Requires a list of Args with at most max elements
  def between(min: Int, max: Int): Args[List[A]]      // Requires a list of Args with a bound on the number of elements
  def map[B](f: A => B): Args[B]                      // Applies a function f to the result of the Args
  def ??(that: String): Args[A]                       // Adds a string to the HelpDoc of the Args
}
```

### Adding Args
Operator `++` can be used to zip two arguments. It can be use to chain two arguments in a tuple. For example, `git clone` command has arguments `<repository>` and `<directory>`. The argument of this command can be created in the following manner:
```scala
val cloneArgs = Args.text("repository") ++ Args.text("directory")
```

The output of the CLI help will be:
```
COMMANDS

  clone <repository> <directory>
```

### Repeating Args
If we need an argument a repeated number of times, we can use the following operators:
- Method `*`

It creates a new `Args` that accepts a list of arguments of the same type. There are no restrictions on length. Take into account that there is no limit, so if there are arguments or options after using this methods, the CLI app will not read them unless they are not valid!
```scala
// Accepts a list, possibly empty, of texts.
Args.text.*
```
- Method `+`

It works as method `*` but the list of arguments cannot be empty. Do not mistake with method `++`!

- Method `between`

It creates an argument accepting a list of arguments of the type before whose length must be between `min` and `max` parameters.
```scala
Args.text.between(2,5) // Creates an arguments accepting a list of String of length between 2 and 5.
```
- Method `atLeast`

It creates an argument accepting a list of arguments of the type before whose length must be more than `min` parameter.
```scala
Args.text.atLeast(2) // Creates an arguments accepting a list of String of length more than 2.
```

- Method `atMost`

It creates an argument accepting a list of arguments of the type before whose length must be less than `max`parameter
```scala
Args.text.atMost(5) // Creates an arguments accepting a list of String of length less than 5.
```

### Transforming Args
Method `map` allows to transform the type parameter of `Args[A]`. It takes a function `f: A => B` as parameter that is applied when processing a user's input in a CLI app and returns `Args[B]`. This makes it easier to implement the business logic of a CLI app. For example, we will construct an `Args` that asks for a list of 12 decimals (one for each month) and a year. Then, it will create a new `Args` that store the year and the mean value in a custom type.
```scala

val data: Args[BigDecimal] = Args.decimal
val args: Args[(List[BigDecimal], BigInt)] = data.between(12, 12) ++ Args.integer

case class YearAndMean(year: BigInt, mean: BigDecimal)

val mappedArgs: Args[YearAndMean] = args.map {
  case (months, year) => YearAndMean(year, months.sum/12)
}
```

### Adding help
Method `??` allows adding information about an argument. The string is added after the current `HelpDoc` of the `Args`. We are going to create the `<repository>` argument of `git clone` to observe the effect of using `??`.

```scala
val repository = Args.text("repository")
  
/* HelpDoc of repository:
 *
 * <repository>
 *   A user-defined piece of text.
 * 
 */

```
Now we add a description of the argument:

```scala
val repositoryWithHelp = repository ?? "Path of the repository to be cloned."

/* HelpDoc of repositoryWithHelp:
 *
 * <repository>
 *   A user-defined piece of text.
 * 
 *   Path of the repository to be cloned.
 */
```

---

## Authorization with OAuth2

A command-line application interacting with an external API might require some form of authentication, so that access is restricted for unauthorized users. One widely employed authorization and authentication protocol is OAuth2. **ZIO CLI** can perform interactions with APIs requiring OAuth2. Although it is possible to construct a custom OAuth2 provider easily, there are already tailored OAuth2 providers for GitHub, Google and Facebook within **ZIO CLI**. We will see how we can add `OAuth2` to our CLI application and how we can construct a custom OAuth2 provider to interact with other APIs.

## Using OAuth2

 OAuth2 is added to a CLI App as an `Options[OAuth2Token]`. The token can be stored and the user can specify the path, so it is not necessary to repeat authentication. We can create it as with other options:
```scala

val clientId = "clientId"

val provider: OAuth2Provider = OAuth2Provider.Github(clientId)
val scope: List[String] = List("repo")
val oauth2 = Options.oauth2(provider, scope)
```

## ZIO Sample providers
Currently the supported OAuth2 providers are GitHub, Google and Facebook.

### Github
To create a GitHub provider, you must specify the `clientId` and the scope of the authorization in a list. The ClientId is the client ID received from GitHub during registering.

```scala
val githubOAuth = Options.oauth2(OAuth2Provider.Github(clientId), List("repo"))
```
### Google
Google requires Client ID and a Client Secret generated by Google API Console. They are known both to Google and your application.
```scala

val clientSecret = "clientSecret"

val googleOAuth = Options.oauth2(OAuth2Provider.Google(clientId, clientSecret), Nil)

```

### Facebook
Facebook requires an App Id and Client Token. They can be obtained through a Facebook developer account.
```scala

val appId = "appId"
val clientToken = "clientToken"

val facebookOAuth = Options.oauth2(OAuth2Provider.Facebook(appId, clientToken), Nil)

```

## Construction of custom OAuth2 provider
To create a custom OAuth2 provider, it suffices to extend the trait `OAuth2Provider`.
The methods that need to be overrided are the following:
```scala

trait OAuth2Provider {
  
  def name: String

  def clientIdentifier: String

  def authorizationRequest(scope: List[String]): HttpRequest

  def accessTokenRequest(authorization: AuthorizationResponse): HttpRequest

  def refreshTokenRequest(refreshToken: String): Option[HttpRequest]
}
``` 
Two other methods that might need to be overrided depending on the particular provider are 
```scala

trait OAuth2Provider {
  
  def decodeAuthorizationResponse(body: String): Either[String, AuthorizationResponse]

  def decodeAccessTokenResponse(body: String): Either[String, AccessTokenResponse]

}
``` 
### Description of methods
- Method `name`

It is the name of the provider.
- Method `clientIdentifier`

It is a public client identifier as provided after registration on authorization
server. It is used for generating default file name, which holds access token.

- Method `authorizationRequest`

It generates the HTTP request for authorization request.

- Method `accessTokenRequest`

It generates the HTTP request for access token request.

- Method `refreshTokenRequest`

It generates the HTTP request for refresh token request. It must return `None` if this operation is not supported by the provider.

- Method `decodeAuthorizationResponse`

It converts textual response of authorization request into `AuthorizationResponse`. It defaults to decoding from standard JSON format.

- Method `decodeAccessTokenResponse`

It converts textual response of access token request into `AccessTokenResponse`.
It defaults to decoding from standard JSON format.

### Construction of GitHub OAuth2 provider
The construction of an OAuth2 provider will be dependent on the particular API that we would like to access. The first step is to define `name` and `clientIdentifier`. The value `clientIdentifier` can be obtained as a field of the case class representing our Provider. Then we construct the core of the Provider. Observe that the methods `authorizationRequest` and `accessTokenRequest` construct an `HttpRequest` from the library **ZIO Http**. They represent a POST request to GitHub API.

```scala

final case class GithubExample(clientId: String) extends OAuth2Provider {
    override val name = "Github"

    override val clientIdentifier = clientId

    // Core logic of Provider
    override def authorizationRequest(scope: List[String]): HttpRequest =
      HttpRequest
        .newBuilder()
        .uri(URI.create(s"https://github.com/login/device/code?client_id=$clientId&scope=${scope.mkString(",")}"))
        .header("Accept", "application/json")
        .POST(HttpRequest.BodyPublishers.noBody())
        .build()

    override def accessTokenRequest(authorization: AuthorizationResponse): HttpRequest =
      HttpRequest
        .newBuilder()
        .uri(
          URI.create(
            s"https://github.com/login/oauth/access_token?client_id=$clientId&device_code=${authorization.deviceCode}&grant_type=urn:ietf:params:oauth:grant-type:device_code"
          )
        )
        .header("Accept", "application/json")
        .POST(HttpRequest.BodyPublishers.noBody())
        .build()

    override def refreshTokenRequest(refreshToken: String): Option[HttpRequest] = None
  }
```

## Integrating OAuth2
This example shows how to integrate OAuth2 in a ZIO `CliApp`. We are going to make a CLI App that interacts with Github and uploads a file to GitHub (This needs OAuth2).

The first step is to define the `Options` that provides an `OAuth2Token
```scala

val githubOAuth: Options[OAuth2Token] = Options.oauth2(OAuth2Provider.Github("sampleId"), List("repo"))
```
Then, we add the token to an `Options` providing the path of a file. We construct a command `upload` usign this `Options`.

```scala
val options = Options.file("path") ++ githubOAuth
val upload = Command("upload", options)
```
Finally, a `CliApp` is specified using `upload` command.
```scala
val cliApp = CliApp.make(
    name = "OAuth2 Example",
    version = "0.0.1",
    summary = text("Example of CliApp with OAuth2"),
    command = upload) {
    // Implement logic of CliApp
      case file => printLine("Upload file to GitHub")
    }
```

---

## Bash and Zsh Completion

## Overview

**ZIO CLI** supports a mechanism for performing tab completion of command line
options and arguments in bash and zsh. The approach that **ZIO CLI** uses to
communicate with the shell tooling for performing tab completion is heavily
inspired by the excellent Haskell
[optparse-applicative](https://github.com/pcapriotti/optparse-applicative#bash-zsh-and-fish-completions)
library. Every `CliApp` is extended with a few hidden built-in options for
providing tab completions to shell environments.

In what follows, pretend that your CLI application (called `my-cli-app`) has been
installed into a stable location in your path (such as the `~/.local/bin`
directory favored by the **ZIO CLI** installer script).

## Generating a completion shell script
The `--shell-completion-script` and `--shell-type` built-in options produce a
shell script that enables tab completion. In the example below, we generate a
completion script (called `completion-script.sh`):
```bash
my-cli-app                                       \
    --shell-completion-script `which my-cli-app` \
    --shell-type bash > completion-script.sh
```
After generating the script, you can quickly enable tab completion via:
```bash
source completion-script.sh
```
Unfortunately, the tab completion will only be enabled within the current shell
session. Normally, the output of `--shell-completion-script` should be shipped
with the program and copied to the appropriate directory (e.g.,
`/etc/bash_completion.d/`) during program installation.

## How Bash and Zsh Completions are Generated
The shell completion scripts register an event handler that fires whenever
`my-cli-app` is the first term at the terminal prompt and the tab key is
pressed. This event handler sends information about the terminal contents and
cursor position back to `my-cli-app` using another built-in option called
`--shell-completion-index` and some special environment variables
(`COMP_WORD_0`, `COMP_WORD_1`, ...).

When `my-cli-app` receives these values, it runs a completion algorithm and
prints the completion terms to the console (one line per completion term). The
console output feeds back into the shell machinery, which renders the completion
results in the terminal.

For example, when the user types the following in the terminal
```
$ my-cli-app foo bar baz
```
and then moves the cursor over "foo" and hits the tab key, `my-cli-app` is called
as follows:
```bash
COMP_WORD_0=my-cli-app     \
COMP_WORD_1=foo            \
COMP_WORD_2=bar            \
COMP_WORD_3=baz            \
my-cli-app                 \
--shell-completion-index 1 \
--shell-type bash
```

The `COMP_WORD_` prefix of these environment variables is directly inspired by
the `COMP_WORD` array-valued Bash variable that is part of its
[programmable completion system](https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html).
Unfortunately, array-valued variables cannot be used as environment variables,
so our approach instead uses one variable per term in the array.

## Further Reading
The [optparse-applicative documentation](https://github.com/pcapriotti/optparse-applicative#bash-zsh-and-fish-completions)
is an excellent resource that may help to clarify the implementation above.

---

## Built-in commands

**ZIO CLI** automatically constructs some common commands:
- `--help` or `-h` shows a description of the CLI's commands.
- `--wizard` executes wizard mode.
- `--shell-completion-index`, `--shell-completion-script` and `--shell-type` are used for Bash and Zsh Completion.

These commands are automatically generated also for all subcommands of a command.
```scala

val cliApp = CliApp.make(
    name = "SampleCliApp",
    version = "0.0.1",
    summary = HelpDoc.Span.text("Sample CLI application"),
    command = Command("command")
) {
    case _ => ???
}

// Prints help of the cliApp command
cliApp.run(List("-h"))

// Prints help of subcommand
cliApp.run(List("subcommand", "-h"))
```

## Examples
```scala

// Construction of basic commands and HelpDoc
val helpGit = "This is command git."
val helpGitAdd = "Stages changes in the working repository."
val helpGitClone = "Creates a copy of an existing repository."

val git: Command[Unit] = Command("git").withHelp(helpGit)
val gitAdd: Command[Unit] = Command("add").withHelp(helpGitAdd)
val gitClone: Command[Path] = Command("clone", Options.directory("directory")).withHelp(helpGitClone)

// Adds subcommands add and clone to command git
val finalCommand: Command[Any] = git.subcommands(gitAdd, gitClone)

```
### Help
The output produced by command `--help` or `-h` is
```
USAGE

  $ git <command>

DESCRIPTION

  This is command git.

COMMANDS

  - add                          Stages changes in the working repository.
  - clone --directory directory  Creates a copy of an existing repository.
```
If we desire to consult the help of a subcommand, like `clone`, we would get the following.
```
USAGE

  $ git clone --directory directory

DESCRIPTION

  Creates a copy of an existing repository.

OPTIONS

  --directory directory
    A directory.
```

### Wizard
Wizard mode allows to construct a command by choosing between parameters offered by the application. The output would be the following.
```
Command(add|clone): 
```
If we input `clone`, we will get:
```
--directory (directory): 
```

---

## Cli Configuration

It is possible to tweak the behavior of our `CliApp` specifying a custom `CliConfig` in the parameter `cliConfig` of method `CliApp.make`. By default, `CliApp.make` uses `CliConfig.default`. `CliConfig` specifies how a `CliApp`
determines the valid commands from the `command: Command[Model]` parameter.

## Parameters

You can construct directly a `CliConfig`:
```scala
final case class CliConfig(
  caseSensitive: Boolean,
  autoCorrectLimit: Int,
  finalCheckBuiltIn: Boolean = true,
  showAllNames: Boolean = true,
  showTypes: Boolean = true
)
```

`CliConfig` allows to control case sensitivity, autocorrection behaviour, command processing and
help appearance.
### Case sensitivity
It is controlled by field `caseSensitive`. If it is `true`, then a `CliApp` will determine as distinct uppercase and lowercase versions of a letter in a command. On the other hand, `caseSensitive = false` implies that the `CliApp` will treat uppercase and lowercase letters as the same. In the Git example, we would have:

- `caseSensitive = true`
```
git clone  
GIT cloNE
```
The first will be detected as the `git clone` command while the second will trigger an error.

- `caseSensitive = false`
```
git clone   // Detected by CLI as "git clone" command
GIT cloNE   // Detected by CLI as "git clone" command
```
Both commands will be detected as the `git clone` command.

### Autocorrection
It is controlled by the field `autoCorrectLimit`. It is the number of mistakes that can be corrected when parsing the name of an option introduced by a user. If the CLI detects that the user has written an incorrect name for the option and the number of mistakes is less, it will suggest the correct option. If `autoCorrectLimit=2` and the user inputs 
`git status --bran nameOfBranch` instead of `git status --branch nameOfBranch`, the output produced by the CLI app will be 
```
The flag "--bran" is not recognized. Did you mean --branch?
```
On the other hand, if the user writes `git status --bra nameOfBranch`, the CLI app will not be able to detect the 3 mistakes and will produce
```
Expected to find --branch option.
```

## Command processing
`finalCheckBuiltIn` controls whether after an invalid command is entered by the user, there is a final check searching for a flag like `--help`, `-h` or `--wizard`. In this case, the corresponding Help or Wizard Mode of the parent command is triggered. Note that this can only trigger the parent command predefined option because the entered command is not valid, so it is an "emergency" check.

## Help appearance
`showAllNames` controls whether all the names of an option are shown in the usage synopsis of a command:
```
command (-o, --option text)      # showAllNames = true
```
`showTypes` controls whether the type of the option is shown in the usage synopsis of a command.
```
command (-o, --option text)      # showAllNames = true,  showTypes = true
command --option text            # showAllNames = false, showTypes = true
command (-o, --option )          # showAllNames = true,  showTypes = false
command --option                 # showAllNames = false, showTypes = false
```

## Default configuration

The default configuration is given by
```scala
object CliConfig {
  val default: CliConfig = CliConfig(caseSensitive = false, autoCorrectLimit = 2)
}
```
This means that a `CliApp` that does not specify any `CliConfig` and uses `CliConfig.default` will:
- ignore if the letters of a command are written in uppercase or lowercase,
- correct automatically up to two mistakes when writing the name of an option in a command of `CliApp`,
- trigger Help or Wizard Mode if the corresponding option is found after an invalid command was entered and
- show full usage synopsis of commands.

---

## Commands

A command is a piece of text representing a directive for a CLI application. This allows the user to communicate easily which task must perform the application. A popular CLI app is the Version Control System called Git. Commonly used commands of Git are among the following.
```
git clone // Creates a copy of a repository
git add   // Adds modified or new files that will be committed after using git commit
```
**ZIO CLI** uses class `Command[Model]` as a description of a CLI command. When using `CliApp.make`, it is needed to specify a `Command[Model]` instance. This parameter is parsed by `CliApp` to produce a collection of possible commands and transform a valid input from a user into an instance of type `Model`. Then you can implement the functionality of the CLI App using pattern matching on instances of `Model`.

### Subcommands
A subcommand is a command that belongs to a larger command. Thus, it is possible for a command to have different subcommands. Subcommands can be used to better organize and design the functionality of a CLI app. Different subcommands can represent different types of task that a CLI can perform and give them a descriptive name. In the Git example above, `git` is the name of the command while `clone` or `add` are subcommands of `git`. As the functionality of `git clone` and `git add` are very different, it is better to use different subcommands. In the same way, it would be possible for a subcommand to have other subcommands.

### Basic construction

It's possible to create a `Command` using the `apply` method. You must always specify the name of the command that will be used
in the CLI, but you can also specify options and/or arguments:

```scala

object Command {
  def apply(name: String) = ???
  def apply[ArgsType](name: String, args: Args[ArgsType]) = ???
  def apply[OptionsType](name: String, options: Options[OptionsType]) = ???
  def apply[OptionsType, ArgsType](name: String, options: Options[OptionsType], args: Args[ArgsType]) = ???
}
```
## Combining commands and transformations
**ZIO CLI** also allows to combine different commands in a functional manner. This can be used for a cleaner design of the commands of a CLI app.

```scala
trait Command[+A] {
  def |[A1 >: A](that: Command[A1]): Command[A1]
  def orElse[A1 >: A](that: Command[A1]): Command[A1]
  def orElseEither[B](that: Command[B]): Command[Either[A, B]]
  def withHelp(help: HelpDoc): Command[A]
  def withHelp(help: String): Command[A]
  def subcommands[B](that: Command[B])
  def subcommands[B](c1: Command[B], c2: Command[B], cs: Command[B]*)
  def map[B](f: A => B): Command[B]
}
```

### Choosing between commands
If we have more than one command, we can create a new `Command` that allows to choose between two commands using methods `|`, `orElse` and `orElseEither` as with `Options`. The difference between these three methods lies in the type parameter of the new `Command`. The user will be able to enter only the command that he wants, triggering the desired functionality.
- Method `orElse`

`|` is just an alias for `orElse`. In the Git CLI, it is possible to choose between different commands. This can be realized using `orElse`.
```scala

val gitAdd: Command[Unit] = Command("add")
val gitClone: Command[Path] = Command("clone", Options.directory("directory"))

val newCommand: Command[Any] = gitAdd orElse gitClone 
```

- Method `orElseEither`

This method wraps the types in an `Either` class.

```scala

val gitAdd: Command[Unit] = Command("add")
val gitClone: Command[Path] = Command("clone", Options.directory("directory"))

val newCommand: Command[Either[Unit, Path]] = gitAdd orElseEither gitClone 
```

### Adding Help Documentation
Method `withHelp` returns a new command with the parameter `HelpDoc` or `String` as the `helpDoc` of the command.

```scala

val helpGit = "This is command git."
val helpGitAdd = "Stages changes in the working repository."
val helpGitClone = "Creates a copy of an existing repository."

// Adding HelpDoc to Commands
val gitCommand: Command[Unit] = Command("git").withHelp(helpGit)
val gitAdd: Command[Unit] = Command("add").withHelp(helpGitAdd)
val gitClone: Command[Path] = Command("clone", Options.directory("directory")).withHelp(helpGitClone)
```
The output that this HelpDoc produces is the following.
```
DESCRIPTION

  This is command git.

COMMANDS

  - add                          Stages changes in the working repository.
  - clone --directory directory  Creates a copy of an existing repository.
```

### Adding subcommands
Method `subcommands` returns a new command with a new set of child commands.

```scala
// Adds subcommands add and clone to command git
val git: Command[Any] = gitCommand.subcommands(gitAdd, gitClone)
```

### Changing the type parameter
`Command[_]` is a generic class. We can apply `map` to construct commands returning any custom type that helps us implement business logic. In this way, we can avoid using types as `Unit`, `Path`, `Any` or `Either` that are not very informative. For example, we can modify the type parameter of `git` to the trait `Git`. 
```scala
sealed trait Git
case class Add() extends Git
case class Clone(directory: Path) extends Git

val customCommand: Command[Git] =
  git.map {
    case () => Add()
    case directory: Path => Clone(directory)
  }
```

This makes possible to implement the business logic of our CLI app using directly a custom type. This can be of great use if we have employed various (`orElseEither`). We are going to implement another subcommand of Git using `orElseEither`.
```scala
// new Git subcommand
val gitBranch = Command("branch", Args.text("branch-name"))

// new Git subclass
case class Branch(branchName: String) extends Git

val eitherCommand: Command[Either[Either[Unit, Path], String]] = gitAdd orElseEither gitClone orElseEither gitBranch
val customEitherCommand: Command[Git] =
  eitherCommand.map {
    case Left(Left(_)) => Add()
    case Left(Right(directory)) => Clone(directory)
    case Right(branchName) => Branch(branchName)
  }
```
In this example, we have transformed a rather cumbersome `Command[Either[Either[Unit, Path], String]]` into a `Command[Git]`.

---

## Advanced example

This example implements some git commands. The workflow is the one used before:
- Define options and arguments
- Construct a command
- Transform command using `map` to change the type parameter
- Construct CLI app using `CliApp.make`
```scala

object GitExample extends ZIOCliDefault {

  sealed trait Subcommand extends Product with Serializable
  object Subcommand {
    final case class Add(modified: Boolean, directory: JPath) extends Subcommand
    final case class Remote(verbose: Boolean)                 extends Subcommand
    object Remote {
      sealed trait RemoteSubcommand extends Subcommand
      final case class Add(name: String, url: String) extends RemoteSubcommand
      final case class Remove(name: String)           extends RemoteSubcommand
    }
  }

  val modifiedFlag: Options[Boolean] = Options.boolean("m")

  val addHelp: HelpDoc = HelpDoc.p("Add subcommand description")
  val add =
    Command("add", modifiedFlag, Args.directory("directory")).withHelp(addHelp).map { case (modified, directory) =>
      Subcommand.Add(modified, directory)
    }

  val verboseFlag: Options[Boolean] = Options.boolean("verbose").alias("v")
  val configPath: Options[Path]     = Options.directory("c", Exists.Yes)

  val remoteAdd = {
    val remoteAddHelp: HelpDoc = HelpDoc.p("Add remote subcommand description")
    Command("add", Options.text("name") ++ Options.text("url")).withHelp(remoteAddHelp).map { case (name, url) =>
      Subcommand.Remote.Add(name, url)
    }
  }

  val remoteRemove = {
    val remoteRemoveHelp: HelpDoc = HelpDoc.p("Remove remote subcommand description")
    Command("remove", Args.text("name")).withHelp(remoteRemoveHelp).map(Subcommand.Remote.Remove)
  }

  val remoteHelp: HelpDoc = HelpDoc.p("Remote subcommand description")
  val remote = {
    Command("remote", verboseFlag)
      .withHelp(remoteHelp)
      .map(Subcommand.Remote(_))
      .subcommands(remoteAdd, remoteRemove)
      .map(_._2)
  }

  val git: Command[Subcommand] =
    Command("git", Options.none, Args.none).subcommands(add, remote)

  val cliApp = CliApp.make(
    name = "Git Version Control",
    version = "0.9.2",
    summary = text("a client for the git dvcs protocol"),
    command = git
  ) {
    case Subcommand.Add(modified, directory) =>
      printLine(s"Executing `git add $directory` with modified flag set to $modified")
    case Subcommand.Remote.Add(name, url) =>
      printLine(s"Executing `git remote add $name $url`")
    case Subcommand.Remote.Remove(name) =>
      printLine(s"Executing `git remote remove $name`")
    case Subcommand.Remote(verbose) =>
      printLine(s"Executing `git remote` with verbose flag set to $verbose")

  }
}

object Example2 extends scala.App {
  println(GitExample.git.helpDoc.toPlaintext())
}
```

Output:

```scala
 COMMANDS

  - add [-m] <directory>      Add subcommand description
  - remote [(-v, --verbose)]  Remote subcommand description
```

---

## Help Documentation

`HelpDoc` is a description of the documentation of a CLI App. They can be added to any `Command`, `Options` or `Args`.
`HelpDoc` is composed of a list of `HelpDoc` items that can be headers, paragraphs, description lists, sequences and enumerations.

## Building blocks
The most basic forms of `HelpDoc` are headers and paragraphs. Method `HelpDoc.p` can create paragraphs from text, while methods `HelpDoc.h1`, `HelpDoc.h2` and `HelpDoc.h3` create headers of different levels.

```scala

val t = "text"

val element: HelpDoc  = HelpDoc.empty             // HelpDoc without content
val header1: HelpDoc  = HelpDoc.h1("Level 1")     // Header of level 1
val header2: HelpDoc  = HelpDoc.h2("Level 2")     // Header of level 2
val header3: HelpDoc  = HelpDoc.h3(t: String)     // Header of level 3
val p: HelpDoc        = HelpDoc.p(t: String)      // Paragraph
```

It is possible to construct a more complex `HelpDoc` by combining them. We can use enumerations, description lists and blocks. In each of the following methods, it is possible to use as many `HelpDoc` arguments as desired.
- Enumeration
```scala
// Enumeration of HelpDocs
HelpDoc.enumeration(element: HelpDoc): HelpDoc
HelpDoc.enumeration(header1 + p, header2 + p)

```
The output of `HelpDoc.enumeration(header1 + p, header2 + p)` is:
```
- 
LEVEL 1

  text
  -
LEVEL 2

  text
```

- Description List
```scala
  
// Creates a list with Span as header
val definition1 = (Span.text("Span1"), HelpDoc.p("Description 1"))
val definition2 = (Span.text("Span2"), HelpDoc.p("Description 12"))
HelpDoc.descriptionList(definition1: (Span, HelpDoc)): HelpDoc
HelpDoc.descriptionList(definition1: (Span, HelpDoc), definition2: (Span, HelpDoc))

```
The output of `HelpDoc.descriptionList(definition1: (Span, HelpDoc), definition2: (Span, HelpDoc))` is:
```
Span1
  Description 1

Span2
  Description 2
```

- Blocks
```scala
// Stacks HelpDocs
HelpDoc.blocks(element: HelpDoc): HelpDoc
HelpDoc.blocks(header1 + p, header2 + p)

```
The output of `HelpDoc.blocks(header1 + p, header2 + p)` is:
```
LEVEL 1

  text

LEVEL 2

  text
```

### Span
The data of a HelpDoc is not stored as text, rather as `Span` type, that also contains information about the type of information. It is possible to use methods `h1`, `h2`, `h3` and `p` with `Span` instead of `String`. You can create `Span` instances using the following methods:
```scala

val span = Span.code(t: String)
Span.empty
Span.error(span: Span)
Span.error(t: String)
Span.space
Span.spans(span: Span)                // You can add more than one span
Span.spans(span, span)
Span.spans(List(span))
Span.strong(span: Span)
Span.strong(t: String)
Span.text(t: String)
Span.uri(java.net.URI.create("https://zio.dev/"))
Span.weak(span: Span)
Span.weak(t: String)
```
You can also obtain its text value using `span.text` or concatenate `Span` using `span1 + span2`.

## Transformation methods
A `HelpDoc` can be converted into plaintext and HTML:
```scala
trait HelpDoc {
  def toHTML: String
  def toPlaintext(columnWidth: Int = 100, color: Boolean = true): String
}
```

## Combination methods
The following methods allow to combine `HelpDoc`:
```scala
trait HelpDoc {
  def +(that: HelpDoc): HelpDoc     // Concatenate HelpDocs in successive levels
  def |(that: HelpDoc): HelpDoc
}
```
- Method `+`

It concatenates `HelpDoc`:
```scala

HelpDoc.h1("Header 1") + HelpDoc.p("paragraph content")
```
It shows:
```
Header 1

  paragraph content
```

- Method `|`

It shows the second `HelpDoc` only if the first one is empty. It could be used to show a backup `HelpDoc`.

## Examples
The more common use case is through the operators `??` and `withHelp`.
- `??` can be applied to `Options` and `Args`. It adds a string to the current description.
```scala

trait Options[A] {
  def ??(that: String): Options[A] // or Args[A]
}

val optionsWithHelp = Options.text("sample") ?? "description of options"
```

- `withHelp` is applied to `Command`. It overwrites the current help of the command, so use it cautiously! On the other hand, you need to use it to add your desired `HelpDoc`
```scala

trait Options[A] {
  def withHelp(that: String): Command[A] // that is converted into a paragraph
  def withHelp(that: HelpDoc): Command[A]
}

val optionsWithHelp = Options.text("sample") ?? "description of options"
val commandWithHelp = Command("command", optionsWithHelp).withHelp("description of command")
```
When `withHelp` is used with a command that has parent and children subcommands, it is applied only to the parent command.

If a more complex use is desired, it is possible to combine headers and paragraphs using the methods before.
```scala
// Create a HelpDoc with a header depending on an integer
val paragraph = HelpDoc.p("paragraph")
def header(n: Int) = HelpDoc.h1("Header: " + n)
def completeDoc(n: Int) = header(n) + paragraph

// Create an enumeration of HelpDocs
val complexDoc = HelpDoc.enumeration(completeDoc(7), completeDoc(-1), completeDoc(3))
```

---

## Introduction to ZIO CLI

Rapidly build powerful command-line applications powered by ZIO

[![Experimental](https://img.shields.io/badge/Project%20Stage-Experimental-yellowgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-cli/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-cli_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-cli_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-cli_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-cli_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-cli-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-cli-docs_2.13) [![ZIO CLI](https://img.shields.io/github/stars/zio/zio-cli?style=social)](https://github.com/zio/zio-cli)

## Installation

To use **ZIO CLI**, we need to add the following to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-cli" % "0.7.4"
```
## Getting Started
**ZIO CLI** allows to easily construct a CLI application. A CLI or Command-Line Interface is an application that allows the user to give instructions by means of pieces of text called commands. A command has the following structure
```
command param1 param2 ... paramN
```
where `command` is the name of the command and `param1`, `param2`, ..., `paramN` form a list of parameters depending on the command that determines the precise instruction to the CLI application.

Given the case, a command itself might contain subcommands. This allows a better design of the command-line application and a more comfortable user experience.

A command might include arguments and options.
- Arguments are name-less and position-based parameters that the user specifies just by the position in the command. As an example, we can consider the widely used command-line application Git. One subcommand is `clone`. It creates a copy of an existing repository. An argument of `git clone` is `repository`. If the repository name is `https://github.com/zio/zio-cli.git`, we will use it in the following manner:
```
git clone https://github.com/zio/zio-cli.git
```

- Options are named and position-independent parameters that are specified by writing the content after the name. The name is preceded by `--`. An option may have a shorter form called an alias. When the alias is used instead of the full name, only `-` is needed. An option of command `git clone` is `local`. It is a boolean option, so it is not necessary to write true or false after it: it will be true only if it appears. It is used in the following manner:
```
git clone --local
```
It also has an alias `-l`:
```
git clone -l
```

The description of the command `git clone`, taking only into account option `local` and argument `repository` will be
```
git clone [-l] <repository>
```
where `[]` implies that the option is optional and `<>` indicates an argument.

### Difference between Args and Options
Arguments and options are different due to the way the user specifies them. Arguments are not specified using its name, only by the position inside the command. On the other hand, options must be preceded by its name and `--` indicating that it is the name of an option. 

Furthermore, a command-line application will represent them in different ways. Argument's name will be inside `<>` while an option will be preceded by `--`. In case that the option has a short form or alias, this will be preceded by `-`.

### First ZIO CLI example

 This is done by defining `cliApp` value from `ZIOCliDefault` using `CliApp.make` and specifying a `Command` as parameter. A `Command[Model]` is a description of the commands of a CLI application that allows to specify which commands are valid and how to transform the input into an instance of `Model`. Then it is possible to implement the logic of the CLI application in terms of `Model`. As a sample we are going to create a command of Git. We are going to implement only command `git clone` with argument `repository` and option `local`.

```scala

// object of your app must extend ZIOCliDefault
object Sample extends ZIOCliDefault {

  /**
   * First we define the commands of the Cli. To do that we need:
   *    - Create command options
   *    - Create command arguments
   *    - Create help (HelpDoc) 
   */
  val options: Options[Boolean] = Options.boolean("local").alias("l")
  val arguments: Args[String] = Args.text("repository")
  val help: HelpDoc = HelpDoc.p("Creates a copy of an existing repository")
  
  val command: Command[(Boolean, String)] = Command("clone").subcommands(Command("clone", options, arguments).withHelp(help))
  
  // Define val cliApp using CliApp.make
  val cliApp = CliApp.make(
    name = "Sample Git",
    version = "1.1.0",
    summary = text("Sample implementation of git clone"),
    command = command
  ) {
    // Implement logic of CliApp
    case _ => printLine("executing git clone")
  }
}
```
The output will be
```
   _____@       @           @        @   __@     @       @  ______@   _ @  __ @
  / ___/@ ____ _@  ____ ___ @   ____ @  / /@ ___ @       @ / ____/@  (_)@ / /_@
  \__ \ @/ __ `/@ / __ `__ \@  / __ \@ / / @/ _ \@       @/ / __  @ / / @/ __/@
 ___/ / / /_/ / @/ / / / / /@ / /_/ /@/ /  /  __/@       / /_/ /  @/ /  / /_  @
/____/  \__,_/  /_/ /_/ /_/ @/ .___/ /_/   \___/ @       \____/   /_/   \__/  @
        @       @           /_/      @     @     @       @        @     @     @

Sample Git v1.1.0 -- Sample implementation of git clone

USAGE

  $ clone clone [(-l, --local)] <repository>

COMMANDS

  clone [(-l, --local)] <repository>  Creates a copy of an existing repository
```

If there is a `CliApp`, you can run a command using its method `run` and passing parameters in a `List[String]`.

## References

- [10 Minute Command-Line Apps With ZIO CLI](https://www.youtube.com/watch?v=UeR8YUN4Tws) by Aiswarya Prakasan
- [Hacking on ZIO-CLI](https://www.youtube.com/watch?v=HxPCXfnbg3U) by Adam Fraser and Kit Langton
- [Behold! The Happy Path To Captivate Your Users With Stunning CLI Apps!](https://www.youtube.com/watch?v=0c3zbUq4lQo) by Jorge Vasquez

---

## Options

The `Options` data type models command-line options. Contrary to arguments, options are named, position-independent parameters passed to a command-line program that modify its behavior. Note that the user must specify the name of the option just before its content. As an example, the Git CLI has a command named `git checkout` that allows changing between different development branches. It has different options that modify the functionality of the command like option `quiet`. This option allows suppressing feedback messages. It can be specified in the following manners:
```
git checkout --quiet
git checkout -q
```
Both `--quiet` and `-q` refer to the same option. `-q` is the alias of `quiet`, a shorter form of the option. Note that the alias is preceded only by `-`.

In **ZIO CLI**, Options are represented by instances of class `Options[_]`. `Options[A]` is a description of the process of constructing an instance of `A` from a valid input of the CLI. It is not yet a specified option for the CLI. In other words, an instance of `Options[A]` defines a collection of valid commands and a way to construct a value `A` from them.

## Construction of basic Options
**ZIO CLI** offers a variety of methods to create basic `Options` that take a string from the user as input and transform it into a value of the corresponding type. The benefit of using these methods is that it is automatically checked if they describe a value of the given type and, in that case, are transformed into it. In case of an invalid input, a `ValidationError` will be thrown.

The following methods construct an `Options` with a `name` that requires an input of the corresponding type from the user. Observe that for `Options` it is mandatory to specify the `name`.
### Boolean Options
Produces an `Options[Boolean]`. It accepts as input `true` or `false`, while other pieces of text are not valid.
```scala

val name = "name"

Options.boolean(name)
```
### Enumeration Options
If we desire to allow the user to choose between different options, we can use the `enumeration` method. It must specify a name and tuples `(String, A)` to create an instance of `Options[A]`. A valid input from the user is a key of a tuple and it will produce the corresponding value. We can use this method if there are different ways to execute a command. As an example, the command `git commit` allows choosing between options `--all`, `--interactive` and `--patch` that control how the user specifies the changes to be committed. One way to implement this would use `enumeration`:
```scala
sealed trait Mode
case class All() extends Mode
case class Interactive() extends Mode
case class Patch() extends Mode

Options.enumeration[Mode]("mode")(("all", All()), ("interactive", Interactive()), ("patch", Patch()))
```
The output produced by the HelpDoc is
```
OPTIONS

  --mode all | interactive | patch
    One of the following cases: all, interactive, patch.
```

### Path Options
They are used to produce an `Options[Path]`.
- Produces `Options[java.nio.file.Path]`.
```scala
Options.file(name) // Used for a file
```
- Produces `Options[java.nio.file.Path]`.
```scala
Options.directory(name) // Used for a directory
```
By default, the `CliApp` accepts paths pointing to files or directories that might or not exist. If you need to work with one of them exclusively you can use the type `Exists`:
```scala
// Path can point to both existing or non-existing files or directories
Options.file(name, exists = Exists.Either)
Options.directory(name, exists = Exists.Either)

// Path must point to existing files or directories
Options.file(name, exists = Exists.Yes)
Options.directory(name, exists = Exists.Yes)

// Path must point to non-existing files or directories
Options.file(name, exists = Exists.Yes)
Options.directory(name, exists = Exists.Yes)
```
### Text Options
Produces an `Options[String]`.
```scala
Options.text(name)
```
### Numeric Options
Numeric `Options` produce `BigInt` and `BigDecimal`, so they can represent numbers of arbitrary size and precision. Note that the CLI checks that the input really represents a number.
- Produces `Options[BigDecimal]`.
```scala
Options.decimal(name)
```

- Produces `Options[BigInt]`.
```scala
Options.integer(name)
```

### Date/Time Options
The following methods produce `Options` whose type parameter is a Date/Time type of the `java.time` library.
- Produces `Options[java.time.Duration]`. The input must be a time-based amount of time in the ISO-8601 format, such as 'P1DT2H3M'.
```scala
Options.duration(name)
```
- Produces `Options[java.time.Instant]`. The input must be an instant in time in UTC format, such as 2007-12-03T10:15:30.00Z.
```scala
Options.instant(name)
```
- Produces `Options[java.time.LocalDate]`. The input must be a date in ISO_LOCAL_DATE format, such as 2007-12-03.
```scala
Options.localDate(name)
```
- Produces `Options[java.time.LocalDateTime]`. The input must be a date-time without a time-zone in the ISO-8601 format, such as 2007-12-03T10:15:30.
```scala
Options.localDateTime(name)
```
- Produces `Options[java.time.LocalTime]`. The input must be a time without a time-zone in the ISO-8601 format, such as 10:15:30.
```scala
Options.localTime(name)
```
- Produces `Options[java.time.MonthDay]`. The input must be a month-day in the ISO-8601 format such as 12-03.
```scala
Options.monthDay(name)
```
- Produces `Options[java.time.OffsetDateTime]`. The input must be a date-time with an offset from UTC/Greenwich in the ISO-8601 format, such as 2007-12-03T10:15:30+01:00.
```scala
Options.offsetDateTime(name)
```
- Produces `Options[java.time.OffsetTime]`. The input must be a time with an offset from UTC/Greenwich in the ISO-8601 format, such as 10:15:30+01:00.
```scala
Options.offsetTime(name)
```
- Produces `Options[java.time.Period]`. The input must be a date-based amount of time in the ISO-8601 format, such as 'P1Y2M3D'.
```scala
Options.period(name)
```
- Produces `Options[java.time.Year]`. The input must be a year in the ISO-8601 format, such as 2007.
```scala
Options.year(name)
```
- Produces `Options[java.time.YearMonth]`. The input must be a year-month in the ISO-8601 format, such as 2007-12.
```scala
Options.yearMonth(name)
```
- Produces `Options[java.time.ZonedDateTime]`. The input must be a date-time with a time-zone in the ISO-8601 format, such as 2007-12-03T10:15:30+01:00 Europe/Paris.
```scala
Options.zonedDateTime(name)
```
- Produces `Options[java.time.ZoneId]`. The input must be a time-zone ID, such as Europe/Paris.
```scala
Options.zoneId(name)
```
- Produces `Options[java.time.ZoneOffset]`. The input must be a time-zone offset from Greenwich/UTC, such as +02:00.
```scala
Options.zoneOffset(name)
```

### Combining and transforming options
We can use the following methods to create more complex Options:
```scala
trait Options[A] {
  def ++[A1 >: A, B](that: Options[B]): Options[(A1, B)]
  def |[A1 >: A](that: Options[A1]): Options[A1]
  def orElse[A1 >: A](that: Options[A1]): Options[A1] // same as |
  def orElseEither[B](that: Options[B]): Options[Either[A, B]] 
  def map[B](f: A => B): Options[B] 
  def optional: Options[Option[A]] 

  // Creates options with different aliases
  def alias(name: String, names: String*): Options[A]

  // Creates options that have a default value
  def withDefault[A1 >: A](value: A1): Options[A1]
}
```

### Adding Options
Operator `++` can be used to zip two options. It can be used to chain two options in a tuple. For example, `git checkout` command has options `--quiet` and `--force`. The options for this command can be created in the following manner:
```scala
val checkoutOptions = Options.boolean("quiet") ++ Options.boolean("force")
```

The output of the CLI help will be:
```
COMMANDS

  checkout [--quiet] [--force] 
```

### Choosing between Options
If we have more than one option and we need only one, we can create a new `Options` that allows to choose between two options using methods `|`, `orElse` and `orElseEither`. The difference between these three methods lies in the type parameter of the new `Options`. The user will be able to enter only the `Options` that he wants, triggering the desired functionality.
- Method `orElse`

`|` is just an alias for `orElse`. In the Git CLI, it is possible to choose between different options. This can be realized using `orElse`.
```scala

val all = Options.boolean("all")
val interactive = Options.boolean("all")
val patch = Options.boolean("all")

val orElseOption: Options[Boolean] = all | interactive | patch
```

- Method `orElseEither`

This method wraps the types in an `Either` class. This will be preferred if the types are very different.

```scala
val orElseEitherOption: Options[Either[Either[Boolean, Boolean], Boolean]] = all orElseEither interactive orElseEither patch
```
This is another way to implement the `[--all | --interactive | --patch]` option of command `git commit`.

### Transforming Options
- Method `map`

It allows to transform the type parameter of `Options[A]`. It takes a function `f: A => B` as parameter that is applied when processing the input of a user in a CLI app. This makes easier to implement the business logic of a CLI app. This allows to implement `[--all | --interactive | --patch]` option of command `git commit` using `orElse`. This is not possible without method `map`, because the result of using `orElse` is an option that returns true if some of the options are employed.
```scala

sealed trait Mode
case class All() extends Mode
case class Interactive() extends Mode
case class Patch() extends Mode

val all = Options.boolean("all").map(_ => All())
val interactive = Options.boolean("all").map(_ => Interactive())
val patch = Options.boolean("all").map(_ => Patch())

val alternative: Options[Mode] = all | interactive | patch
```

- Method `optional`

It wraps the option in an `Option` class. The resulting `Options` is now optional, so the user can choose to specify it or not. If it is specified, it will be wrapped in `Some()`. Without input from the user, it will produce `None`. For example, if we are designing a command-line application to manage databases, we might desire to retrieve data from a particular database. The user could specify an additional backup database. If there were an error connecting with the main database, the CLI app would try to retrieve data from the backup database.
```scala
val database = Options.file("database")
val backup = Options.file("backup").optional
val retrieve = Command("retrieve", database ++ backup)
```
The HelpDoc of `optionsDatabase` will be 
```
COMMANDS

  retrieve --database file [--backup file]  
```
The `[]` denotes that the parameter is optional.

- Method `withDefault`

It creates an option that will have a default parameter in case that the user does not input the option. It can be used, for example, to specify a default directory in which a CLI application may perform some task if the user does not specify another one.
```scala
val directory = Options.directory("directory").withDefault("C:\\Users\\YourUser\\Documents")
```
The output produced by the HelpDoc is
```
OPTIONS

  --directory directory
    A directory.

    This setting is optional. Default: 'C:\Users\YourUser\Documents'.
```

### Making an alias
Using method `alias` it is possible to give a shorter form of the name of the `Options`. This is used to make easier the input of options for the user. We will make an alias for option `--directory`. 
```scala
val directoryWithAlias = directory.alias("d")
```
The output produced by the HelpDoc in this case is
```
OPTIONS

  (-d, --directory directory)
    A directory.

    This setting is optional. Default: 'C:\Users\YourUser\Documents'.
```

### Adding help
Method `??` allows to add information about an options. The string is added after the current `HelpDoc` of the `Options`. We are going to recreate the `--quiet` option of `git checkout` to observe the effect of using `??`.
```scala
val quiet = Options.boolean("quiet")
  
/* HelpDoc of quiet:
 *
 * --quiet
 *   A true or false value.
 * 
 *   This setting is optional. Default: 'false'.
 */

```
Now we add a description of the option:

```scala
val quietWithHelp = quiet ?? "Suppress feedback messages."

/* HelpDoc of quietWithHelp:
 *
 * --quiet
 *   A true or false value.
 * 
 *   Suppress feedback messages.
 * 
 *   This setting is optional. Default: 'false'.
 */
```

### Example
As an example, we are going to construct an instance of `Options[Git]` that describes the options of the command `git commit`. We are going to implement only an option `--msg` that includes a commit message and alternatives between `-a`, `--interactive` and `--patch`. First, we create the basic options. Observe how we add an alias to `all`and `msg`.
```scala

val all = Options.boolean("all").alias("a")
val interactive = Options.boolean("interactive")
val patch = Options.boolean("patch")

val msg = Options.text("msg").alias("m")
```
Then we combine options:
- using `orElse` to combine `all`, `interactive` and `patch`,
- adding it to `msg`.

```scala

val alternative = all | interactive | patch

val options = alternative ++ msg
```
The output of the HelpDoc of `options` is:
```
USAGE

  $ commit [(-a, -all)]|[--interactive]|[--patch] (-m, --msg text)

OPTIONS

  (-a, --all)
    A true or false value.

    This setting is optional. Default: 'false'.

  --interactive
    A true or false value.

    This setting is optional. Default: 'false'.

  --patch
    A true or false value.

    This setting is optional. Default: 'false'.

  (-m, --msg text)
    A user-defined piece of text.
```

---

## ZIO CLI SBT Plugin

## Overview

**ZIO CLI** offers an SBT plugin to provide some tasks to make generation and deployment of 
applications simple.

## Current State

The current functionality of the plugin offers:

### Native Images 
This SBT plugin depends on and utilizes the 
[sbt-native-image](https://github.com/scalameta/sbt-native-image) plugin. Tasks and settings
related to building a native image map 1:1 to the upstream plugin.

## Outstanding Action Items
- [ ] Investigate if any cross versions are needed for different versions of SBT
- [ ] Add remaining corresponding upstream configuration options to pass into the `sbt-native-image` plugin.
- [ ] Implement `zioCliGenerateBashCompletion`, when ready.
- [ ] Implement `zioCliGenerateZshCompletion`, when ready.
- [ ] Implement `zioCliInstallCli`, when ready.

## Plugin Settings

### zioCliMainClass: Option\[String\]
The mainClass of the CLI App in the `Compile` scope. Defaults to `None`, and will need to be set.

### zioCliNativeImageOptions: Seq\[String\]
A collection of arguments to pass the native-image builder to customize native image generation. Defaults to
```scala
List(
      "--allow-incomplete-classpath",
      "--report-unsupported-elements-at-runtime",
      "--initialize-at-build-time",
      "--no-fallback"
    )
```

### zioCliNativeImageReady: \(\) => Unit
A side-effecting callback that is called the native image is ready. Defaults to
```scala
() => {
        println("ZIO CLI App Native Image Ready!")
      } 
```

## Plugin Tasks

### zioCliBuildNative: Unit
Sets mainClass in Compile, and attempts to run `nativeImage` with any applied `nativeImageOptions`. 
Fires `zioCliNativeImageReady` on success.

### zioCliGenerateBashCompletion: Unit
Not currently implemented; prints `"TODO: Not Implemented!"`

### zioCliGenerateZshCompletion: Unit
Not currently implemented; prints `"TODO: Not Implemented!"`

### zioCliInstallCli: Unit
Not currently implemented; prints `"TODO: Not Implemented!"`

## End User Project Setup

There are several steps that are needed to use this in an end user project:

### Locally Publishing the Plugin
To compile and locally publish a copy of the SBT plugin, simply call  `publishLocal` on the `sbtZioCli` project. Example:
```
sbt> project sbtZioCli
sbt> publishLocal
```
This will locally publish an artifact: `"zio.cli.sbt" % "sbt-zio-cli" % "0.0.0-SNAPSHOT"`

### End User Project
In your project, configure SBT to use the plugin by adding to `project/plugins.sbt`:
```
addSbtPlugin("zio.cli.sbt" % "sbt-zio-cli" % "0.0.0-SNAPSHOT")
```

Then, enable the plugin for your project in your `build.sbt` file. Example:
```
lazy val root = project
  .in(file("."))
  .enablePlugins(ZIOCLIPlugin)
  .settings(
    CLIPlugin.zioCliMainClass := Some("zio.cli.YourApp")
  )
```

At this point, you should be able to build a native image of your application with the task:
```
sbt> zioCliBuildNative
```

---

## Automatic Derivation of Config

By bringing in `zio-config-magnolia` we  avoid all the boilerplate required to define the config. With a single import, `Config` is automatically derived.

### Example

#### Config
```scala
sealed trait X

object X {
  case object A extends X
  case object B extends X
  case object C extends X
  case class  DetailsWrapped(detail: Detail) extends X
  
  case class Detail(firstName: String, lastName: String, region: Region)
  case class Region(suburb: String, city: String)
}

case class MyConfig(x: X)
```

#### AutoDerivation

```scala
// Setting up imports

```

```scala
// Defining different possibility of HOCON source

val aHoconSource =
  ConfigProvider
    .fromHoconString("x = A")
```

```scala
val bHoconSource =
  ConfigProvider
    .fromHoconString("x = B")
```

```scala
val cHoconSource =
  ConfigProvider
    .fromHoconString("x = C")
```

```scala
val dHoconSource =
  ConfigProvider
    .fromHoconString(
      s"""
         | x {
         |   DetailsWrapped {
         |    detail  {
         |      firstName : ff
         |      lastName  : ll
         |      region {
         |        city   : syd
         |        suburb : strath
         |     }
         |   }
         |  }
         |}
         |""".stripMargin
    )
```

```scala
// Let's try automatic derivation

aHoconSource.load(deriveConfig[MyConfig])
// res0: Right(MyConfig(A))

bHoconSource.load(deriveConfig[MyConfig])
// res0: Right(MyConfig(B))

cHoconSource.load(deriveConfig[MyConfig])
// res0: Right(MyConfig(C))

dHoconSource.load(deriveConfig[MyConfig])
// res0: Right(MyConfig(DetailsWrapped(Detail("ff", "ll", Region("strath", "syd")))))
```

**NOTE**

The fieldNames and class-names remain the same as that of case-classes and sealed-traits.

If you want custom names for your fields, use `name` annotation.

```

@name("detailsWrapped")
case class  DetailsWrapped(detail: Detail) extends X
```

```scala

deriveConfig[MyConfig].mapKey(toKebabCase)
```

With the above change`firstName` and `lastName` in the above HOCON example can be `first-name` and `last-name` 
respectively.

There are various ways in which you can customise the derivation of sealed traits. 
This is a bit involving, and more documentations will be provided soon.

### Documentation while automatic derivation

With `describe` annotation you can still document your config while automatically generating the config

```scala

@describe("This config is about aws")
case class Aws(region: String, dburl: DbUrl)
case class DbUrl(value: String)
```

This will be equivalent to the manual configuration of:

```scala
string("region").zip(string("dburl").to[DbUrl]).to[Aws] ?? "This config is about aws"
```

You could provide `describe` annotation at field level

Example:

```scala
case class Aws(@describe("AWS region") region: String, dburl: DbUrl)
```

This will be equivalent to the manual configuration of:

```scala
(string("region") ?? "AWS region" zip string("dburl").to[DbUrl]).to[Aws] ?? "This config is about aws"
```

### Custom Config

Every field in a case class should have an instance of `DeriveConfig` (and not `Config`) in order for the automatic derivation to work.
This doesn't mean the entire design of zio-config is typeclass based. For the same reason, the typeclass
`DeriveConfig` exists only in zio-config-magnolia.

As an example, given below is a case class where automatic derivation won't work, and result in a compile time error:
Assume that, `AwsRegion` is a type that comes from AWS SDK.

```

case class Execution(time: AwsRegion, id: Int)
```

In this case, `deriveConfig[Execution]` will give us the following descriptive compile error.

```
magnolia: could not find Descriptor.Typeclass for type <outside.library.package>.AwsRegion
  in parameter 'time' of product type <your.packagename>.Execution
```

This is because zio-config-magnolia failed to derive an instance of Descriptor for AwsRegion.

In order to provide implicit instances, following choices are there

```

implicit val awsRegionConfig: DeriveConfig[Aws.Region] =
  DeriveConfig[String].map(string => AwsRegion.from(string))

```

Now `deriveConfig[Execution]` compiles.

Custom descriptors are also needed in case you use value classes to describe your configuration. You can use them
together with automatic derivation and those implicit custom descriptors will be taken automatically into account

```scala

final case class AwsRegion(value: String) extends AnyVal {
  override def toString: String = value
}

object AwsRegion {
  implicit val descriptor: DeriveConfig[AwsRegion] = 
    DeriveConfig[String].map(AwsRegion(_))
}
```

### Where to place these implicits ?

If the types are owned by us, then the best place to keep implicit instance is the companion object of that type.

```scala
final case class MyAwsRegion(value: AwsRegion)

object MyAwsRegion {
  implicit val awsRegionDescriptor: DeriveConfig[MyAwsRegion] =
    DeriveConfig[String]
      .map(
        string => MyAwsRegion(AwsRegion.from(string))
      ) ?? "value of type AWS.Region"
}
```

However, sometimes, the types are owned by an external library.

In these situations, better off place the implicit closer to where we call the automatic derivation.
Please find the example in `magnolia` package in examples module.

### Change Keys (CamelCase, kebab-case etc)

Please find the examples in ChangeKeys.scala in magnolia module to find how to manipulate keys in an automatic derivation such as being able to specify keys as camelCase, kebabCase or snakeCase in the source config.

## Scala3 Autoderivation
Works just like scala-2.12 and scala-2.13.
If possible, we will make this behaviour consistent in scala-2.12 and scala-2.13 in future versions of zio-config.

#### Example:

The name of the sealed trait itself is skipped completely by default.
However, if you put a `name` annotation on top of the sealed-trait itself,
then it becomes part of the config. 

The name of the case-class should be available in config-source, 
and by default it should the same name as that of the case-class.

```scala
sealed trait A
case class B(x: String) extends A
case class C(y: String) extends A

case class Config(a: A) 
```

With the above config, `deriveConfig[A]` can read following source.

```scala
{
  "a" : {
   "B" : {
      "x" : "abc"
     }
   }
  }
}

// or a.B.x="abc", if your source is just property file
```

However, if you give `name` annotation for A, the name of the sealed trait
should be part of the config too. This is rarely used.

```scala
@name("aaaaa")
sealed trait A
case class B(x: String) extends A
case class C(y: String) extends A

case class Config(a: A) 
```

With the above config, `deriveConfig[A]` can read following source.

```scala
{
  "a" : {
    "aaaaa" : 
      "B" : {
        "x" : "abc"
     }
   }
  }
}
```
Similar to scala-2.x, you can give name annotations to any case-class as well (similar to scala 2.x)

#### Example:

```scala
sealed trait A

@name("b")
case class B(x: String) extends A
case class C(y: String) extends A

case class Config(a: A) 
```

In this case, the config should be

```scala
{
  "a" : {
   "b" : {
      "x" : "abc"
     }
   }
  }
}
```

### Config derivation via `derives` keyword

Scala 3 has introduced `derives` keyword to derive typeclasses without the need of explicitely declaring the implicit / given.
This syntax can be enabled for `zio.Config` by importing `zio.config.magnolia.*` and then using the `derives` keyword on the type that needs to be derived:

```scala 3

sealed trait A
case class B(x: String) extends A
case class C(y: String) extends A

case class MyConfig(a: A) derives Config

// And then simply summon the `Config[MyConfig]` instance as you would normally:
val cfg = ZIO.config[MyConfig]
```

### No guaranteed behavior for scala-3 enum yet
With the current release, there is no guaranteed support of scala-3 enum. 
Use `sealed trait` and `case class` pattern.

### No support for recursive config in auto-derivation, but we can make it work

There is no support for auto-deriving recursive config with scala-3.
Please refer to examples in magnolia package (not in the main examples module)

### Custom Keys

With zio-config-3.x, the only way to change keys is as follows:

```scala
// mapKey is just a function in `Config` that pre-existed

val config = deriveConfig[Config].mapKey(_.toUpperCase)
```

## Inbuilt support for pure-config

Many users make use of the label `type` in HOCON files to annotate the type of the coproduct. 
Just put `@nameWithLabel()` in sealed trait name. By default the `label` name is `type`, but you can provide
any custom name `@nameWithLabel("foo")`

```scala

@nameWithLabel("type")
sealed trait X
case class A(name: String) extends X
case class B(age: Int) extends X

case class AppConfig(x: X)

val str =
  s"""
   x : {
     type = A
     name = jon
   }
  """

read(deriveConfig[AppConfig] from ConfigProvider.fromHoconString(str))
```

---

## Automatic Validations

By bringing in `zio-config-refined` module, you get validations for your config parameters almost for free. 
`zio-config` elegantly integrates with `Refined` library for you to achieve this with same ergonomics.

If you are not familiar with `refined` library, refer https://github.com/fthomas/refined.

There are various ways that zio-config can interact with refined library. 
Take a look at `zio.config.refined` package.

```scala

```

A few examples are given below.

## Basic Example

```scala

 case class Jdbc(username: NonEmptyString, password: NonEmptyString)

 val jdbc: Config[Jdbc] =
   (refineType[NonEmptyString]("username") zip
     refineType[NonEmptyString]("password")).to[Jdbc]

 ConfigProvider.fromMap(Map("username" -> "", "password" -> "")).load(jdbc)
```

## Direct Interaction with Refined Predicates

If you need to directly interact with `Predicate`s (ex: `NonEmpty`), then
`refine[A, P]` method is useful.

```scala

 type NonEmptyString = String Refined NonEmpty
 
 val refinedConfig: Config[NonEmptyString] = 
   refineType[NonEmptyString]("USERNAME")
  
 // Another way of doing it is
 val urlConfig: Config[Refined[String, Url]] =
   refine[String, Url]("URL")
   
 // refineType takes a fully formed type (String Refined NonEmpty) where as refine allows you to play with the predicate directly (NonEmpty)  
```

## Derive from existing Config

Of various methods available in `zio.config.refined` package, 
the most interesting one is being able to get a refined type out of an already derived Config.
This shows the composable nature of zio-config. 

Take a look at the below example

```scala

 case class MyConfig(url: String, port: Int)

 val configs: Config[List[MyConfig]] =
   Config.listOf("databases", deriveConfig[MyConfig])

 // A list of database configs, such that size should be greater than 2.
 val databaseList: Config[Refined[List[MyConfig], Size[Greater[W.`2`.T]]]] =
   refine[Size[Greater[W.`2`.T]]](configs)
```

## Auto-Derivation and Refined

You can also use auto derivations with refined.

```scala

object RefinedReadConfig extends App {
  case class RefinedProd(
    ldap: Refined[String, NonEmpty],
    port: Refined[Int, GreaterEqual[W.`1024`.T]],
    dbUrl: Option[Refined[String, NonEmpty]]
  )

  val configMap =
    Map(
      "LDAP"     -> "ldap",
      "PORT"     -> "1999",
      "DBURL"   -> "ddd"
    )

  val result =
    ConfigProvider.fromMap(configMap).load(deriveConfig[RefinedProd].mapKey(_.toUpperCase))

  // RefinedProd(ldap,1999,Some(ddd))
}
```

---

## Dive Into ZIO Config

__Note that this documentation is for 1.x series. For newer versions, please refer to [docs](https://github.com/zio/zio-config/tree/master/docs) section in GitHub.__

## Describe the config by hand

We must fetch the configuration from the environment to a case class (product) in scala. Let it be `MyConfig`

```scala

```

```scala
case class MyConfig(ldap: String, port: Int, dburl: String)
```
To perform any action using zio-config, we need a configuration description.
Let's define a simple one.

```scala
val myConfig: Config[MyConfig] =
  (string("LDAP") zip int("PORT") zip string("DB_URL")).to[MyConfig]

 // Config[MyConfig]
```

To get a tuple,

```scala
val myConfigTupled: Config[(String, Int, String)] =
  (string("LDAP") zip int("PORT") zip string("DB_URL"))
```

## Fully automated Config Description

If you don't like describing your configuration manually, and rely on the names of the parameter in the case class (or sealed trait),
there is a separate module called `zio-config-magnolia`.

Note:  `zio-config-shapeless` is an alternative to `zio-config-magnolia` to support scala 2.11 projects.
It will be deprecated once we find users have moved on from scala 2.11.

```scala

val myConfigAutomatic = deriveConfig[MyConfig]
```

`myConfig` and `myConfigAutomatic` are same description, and is of the same type.

Refer to API docs for more explanations on [descriptor](https://javadoc.io/static/dev.zio/zio-config-magnolia_2.13/1.0.0-RC31-1/zio/config/magnolia/index.html#descriptor[A](implicitconfig:zio.config.magnolia.package.Descriptor[A]):zio.Config[A])
More examples on automatic derivation is in examples module of [zio-config](https://github.com/zio/zio-config)

## Read config from various sources

There are more information on various sources in [here](read-from-various-sources.md).

Below given is a simple example.

```scala
val map =
  Map(
    "LDAP" -> "xyz",
    "PORT" -> "8888",
    "DB_URL" -> "postgres"
  )

val source = ConfigProvider.fromMap(map)

source.load(myConfig)

```

### Documentations using Config

```scala
generateDocs(myConfig)
//Creates documentation (automatic)

val betterConfig =
  (string("LDAP") ?? "Related to auth" zip int("PORT") ?? "Database port" zip
    string("DB_URL") ?? "url of database"
   ).to[MyConfig]

generateDocs(betterConfig).toTable.toGithubFlavouredMarkdown
// Custom documentation along with auto generated docs
```

### Accumulating all errors

For any misconfiguration, the ReadError collects all of them with proper semantics: `AndErrors` and `OrErrors`.
Instead of directly printing misconfigurations, the `ReadError.prettyPrint` shows the path, detail of collected misconfigurations.

1. All misconfigurations of `AndErrors` are put in parallel lines.
```text
╥
╠══╗ 
║  ║ FormatError
║ MissingValue
``` 
2. `OrErrors` are in the same line which indicates a sequential misconfiguration

```text
╥
╠MissingValue
║
╠FormatError
```

Here is a complete example:

```text
   ReadError:
   ╥
   ╠══╦══╗
   ║  ║  ║
   ║  ║  ╠─MissingValue
   ║  ║  ║ path: var2
   ║  ║  ║ Details: value of type string
   ║  ║  ║ 
   ║  ║  ╠─MissingValue path: envvar3
   ║  ║  ║ path: var3
   ║  ║  ║ Details: value of type string
   ║  ║  ║ 
   ║  ║  ▼
   ║  ║
   ║  ╠─FormatError
   ║  ║ cause: Provided value is wrong, expecting the type int
   ║  ║ path: var1
   ║  ▼
   ▼
```

### Example of mapping keys

Now on, the only way to change keys is as follows:

```scala
  // mapKey is just a function in `Config` that pre-existed

  val config = deriveConfig[Config].mapKey(_.toUpperCase)
```

## Inbuilt support for pure-config

Many users make use of the label `type` in HOCON files to annotate the type of the coproduct.
Now on, zio-config has inbuilt support for reading such a file/string using `descriptorForPureConfig`.

```scala

@nameWithLabel("type")
sealed trait X
case class A(name: String) extends X
case class B(age: Int) extends X

case class AppConfig(x: X)

val str =
  s"""
   x : {
     type = A
     name = jon
   }
  """

ConfigProvider.fromHoconString(str).load(deriveConfig[AppConfig])

```

## The `to` method for easy manual configurations

```scala

final case class AppConfig(port: Int, url: String)

val config = Config.int("PORT").zip(Config.string("URL")).to[AppConfig]

```

## A few handy methods

### CollectAll

```scala

  final case class Variables(variable1: Int, variable2: Option[Int])

  val listOfConfig: List[Config[Variables]] =
    List("GROUP1", "GROUP2", "GROUP3", "GROUP4")
      .map(group => (Config.int(s"${group}_VARIABLE1") zip Config.int(s"${group}_VARIABLE2").optional).to[Variables])

  val configOfList: Config[List[Variables]] =
    Config.collectAll(listOfConfig.head, listOfConfig.tail: _*)

```

### orElseEither && Constant

```scala

sealed trait Greeting

case object Hello extends Greeting
case object Bye extends Greeting

val configSource = 
  ConfigProvider.fromMap(Map("greeting" -> "Hello"))

val config: Config[Greeting] = 
  Config.constant("Hello").orElseEither(Config.constant("Bye")).map(_.merge)

```

---

## Getting Started with ZIO Config

[ZIO Config](https://zio.dev/zio-config/) is a ZIO-based library and act as an extension to core library ZIO's `Config` language.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-config/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-config_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-config_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-config_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-config_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-config-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-config-docs_2.13) [![ZIO Config](https://img.shields.io/github/stars/zio/zio-config?style=social)](https://github.com/zio/zio-config)

Let's enumerate some key features of this library:

- **Support for Various Sources** — It can read flat or nested configurations. Thanks to `IndexedFlat`.
- **Automatic Document Generation** — It can auto-generate documentation of configurations.
- **Automatic Derivation** — It has built-in support for automatic derivation of readers and writers for case classes and sealed traits.
- **Type-level Constraints and Automatic Validation** — because it supports _Refined_ types, we can write type-level predicates which constrain the set of values described for data types.
- **Descriptive Errors** — It accumulates all errors and reports all of them to the user rather than failing fast.
- **Integrations** — Integrations with a variety of libraries

If you are only interested in automatic derivation of configuration, find the details [here](https://zio.dev/zio-config/automatic-derivation-of-config)

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-config" % "<version>" 
```

# Quick Start

Let's add these four lines to our `build.sbt` file as we are using these modules in our examples:

```scala
libraryDependencies += "dev.zio" %% "zio-config"          % "<version>"
libraryDependencies += "dev.zio" %% "zio-config-magnolia" % "<version>"
libraryDependencies += "dev.zio" %% "zio-config-typesafe" % "<version>"
libraryDependencies += "dev.zio" %% "zio-config-refined"  % "<version>"
```

There are many examples in [here](https://github.com/zio/zio-config/tree/master/examples/shared/src/main/scala/zio/config/examples)

---

## Integrations

## Integration with other libraries

`zio-config` is also integrated with `enumeratum`, `cats`, `scalaz`, `aws-sdk`, `zio-aws`, `refined` etc.
Note that only a few of them is documented here. `refined` is already discussed under `automatic-validations`.

#### Enumeratum

Many applications rely on this beautiful library https://github.com/lloydmeta/enumeratum.
Zio-config can directly load it from enumeratum's `enum` without relying on auto-derivation (and rely on Enumeratum's macro indirectly witha additional features).

```scala

 sealed trait Greeting extends EnumEntry

  object Greeting extends Enum[Greeting] {

    val values = findValues

    case object Hello extends Greeting
    case object GoodBye extends Greeting
    case object Hi extends Greeting
    case object Bye extends Greeting

  }

  // Load using zio-config

  val mapProvider =
    ConfigProvider.fromMap(Map(
      "greeting" -> "Hello"
    ))

  val config =
    `enum`(Greeting).nested("greeting")

  val pgm: IO[Error, Greeting] =
    mapProvider.load(config)
    
  // Returns Hello  
    

```

#### Scalaz/Cats

Highly polymorphic code end up relying on
typeclasses, and zio-config provides instances for `Config`.

This is a simple example to showcase the capability.

```scala

 // Across the application, there can be various effect types, but there is only one addition!
 def add[F[_]: Applicative, A: Monoid](primary: F[A], secondary: F[A]): F[A] =
    primary.<*>(Applicative[F].map(secondary)(secondary => (primary: A) => primary.mappend(secondary)))
    
 // Now even `Config` can take part in this addition given the values of config parameters should be Monoid,
 // instead of using native `zip` and separately implementing addition for various types
 val configResult = add(Config.int("marks1"), Config.int("marks2")))
 
 ConfigProvider.fromMap(Map("marks1" -> "10", "marks2" -> "20")).load(configResult) // returns 30
 

```

In addition to it, it can also load cats/scalaz specific datatypes

```scala

  val config: Config[Maybe[Int]] = maybe(Config.int("age"))
  
```

Have a look at modules of zio-config to know about other integrations such as `aws`, `zio-aws` etc

---

## Read from various Sources

zio-config supports various sources.

```scala

```

```scala
case class MyConfig(ldap: String, port: Int, dburl: String)
```

```scala
val myConfig =
  (string("LDAP") zip int("PORT") zip string("DB_URL")).to[MyConfig]

 // val automatedConfig = deriveConfig[MyConfig]; using zio-config-magnolia
```

## HOCON String

To enable HOCON source, you have to bring in `zio-config-typesafe` module.
There are many examples in examples module in zio-config.

Here is an quick example

```scala

```

```scala
case class SimpleConfig(port: Int, url: String, region: Option[String])

val automaticDescription = deriveConfig[SimpleConfig]

val hoconSource =
  ConfigProvider.fromHoconString(
      """
      {
        port : 123
        url  : bla
        region: useast
      }

      """
    )

val anotherHoconSource =
  ConfigProvider.fromHoconString(
      """
        port=123
        url=bla
        region=useast
      """
  )

hoconSource.load(deriveConfig[SimpleConfig])

// yielding SimpleConfig(123,bla,Some(useast))
```

## HOCON File

```scala
ConfigProvider.fromHoconFile(new java.io.File("fileapth"))
```

## Json

You can use `zio-config-typesafe` module to fetch json as well

```scala
val jsonString =
   """
   {
     "port" : "123"
     "url"  : "bla"
     "region": "useast"
   }

   """

ConfigProvider.fromHoconString(jsonString)
```

## Yaml FIle

Similar to Hocon source, we have `ConfigProvider.fromYamlString`

```scala

ConfigProvider.fromYamlString

```

## Xml String

zio-config can read XML strings. Note that it's experimental with a dead simple native xml parser, 
Currently it cannot XML comments, and has not been tested with complex data types, which will be fixed in the near future.

```scala

final case class Configuration(aws: Aws, database: Database)

object Configuration {
  val config: Config[Configuration] =
    Aws.config.nested("aws").zip(Database.config.nested("database")).to[Configuration].nested("config")

  final case class Aws(region: String, account: String)

  object Aws {
    val config: Config[Aws] = Config.string("region").zip(Config.string("account")).to[Aws]
  }
  final case class Database(port: Int, url: String)

  object Database {
    val config: Config[Database] = Config.int("port").zip(Config.string("url")).to[Database]
  }
}

val config =
  s"""
     |<config>
     |  <aws region="us-east" account="personal"></aws>
     |  <database port="123" url="some url"></database>
     |</config>
     |
     |""".stripMargin

val parsed = ConfigProvider.fromYamlString(config).load(Configuration.config)

```

### Indexed Map, Array datatype, and a some implementation notes

`zio-config` comes up with the idea of `IndexedFlat` allowing you to define indexed configs (see examples below).
However, the constructors of `IndexedFlat` is not exposed to the user for the time being, since it can conflate with some ideas in `zio.core` `Flat`,
and resulted in failures whenever `IndexedFlat` was converted to a `Flat` internally. Example: https://github.com/zio/zio-config/issues/1095

Therefore, some of these ideas around `Indexing` is  pushed back to `ZIO` and incorporated within the `Flat` structure.

See https://github.com/zio/zio/pull/7823 and https://github.com/zio/zio/pull/7891

These changes are to keep the backward compatibility of ZIO library itself.

#### What does it mean to users?
It implies, for sequence (or list) datatypes, you can use either `<nil>` or `""` to represent empty list in a flat structure.
See the below example where it tries to mix indexing into flat structure.
We recommend using `<nil>` over `""` whenever you are trying  to represent a real indexed format

Example:

```scala

final case class Department(name: String, block: Int)

final case class Employee(departments: List[Department], name: String)
final case class Config(employees: List[Employee])

val map =
  Map(
    "employees[0].name" -> "jon",
    "employees[0].departments[0].name" -> "science",
    "employees[0].departments[0].block" -> "10",
    "employees[0].departments[1].name" -> "maths",
    "employees[0].departments[2].block" -> "11",
    "employees[1].name" -> "foo",
    "employees[1].departments" -> "<nil>",
  )
  

ConfigProvider.fromMap(map).load(derivedConfig[Config])

```

Although we support indexing within Flat, formats such as Json/HOCON/XML is far better to work with indexing,
and zio-config supports these formats making use of the above idea.

#### Another simple example of an indexed format

```scala

final case class Employee(age: Int, name: String)

 val map = 
   Map(
     "department.employees[0].age" -> "10",
     "department.employees[0].name" -> "foo",
     "department.employees[1].age" -> "11",
     "department.employees[1].name" -> "bar",
     "department.employees[2].age" -> "12",
     "department.employees[2].name" -> "baz",
   )

val provider = ConfigProvider.fromMap(map)
val config = Config.listOf("employees", deriveConfig[Employee]).nested("department")
val result = provider.load(config)

```

---

## ZIO Config Resources

- [Easy Config For Your App](https://www.youtube.com/watch?v=4SrSKluyyKo) by Afsal Thaj (December 2020) — Managing application configuration can be quite challenging: we often have to support multiple data sources with overrides, including HOCON, system properties, environment variables, and more. We have to document our configuration so it is clear to IT and DevOps how to configure our applications. We have to do strong validation with error accumulation to ensure bad data is rejected and good error messages are generated for end-users. In this presentation, Afsal Thaj, the author of ZIO Config, shows attendees how to solve all of these problems in a composable and testable way. By separating the description of configuration from what must be done with the configuration, ZIO Config provides all standard features—including multiple data sources, data source overrides, documentation, and validation with error accumulation—for free. Come learn how to make your applications configurable in an easy way that will delight IT and DevOps and make it easy to change your applications over time.

- Introduction to ZIO Config by Afsal Thaj
    - Part 1: [Start writing better Scala with zio-config](https://www.youtube.com/watch?v=l5CVQmSp7fY)
    - Part 2: [Maximise the use of Scala types (Option & Either in zio-config)](https://www.youtube.com/watch?v=SusCbrSK5eA&t=0s)
    - Part 3: [Intro to ADT, and scalable configuration management!](https://www.youtube.com/watch?v=LGo_g1GK6_k&t=0s)
    - Part 4: [Auto generate sample configurations of your application in Scala](https://www.youtube.com/watch?v=--mcs4HztJY&t=0s)

PS: Note that the videos use older version of zio-config. However, if you are new to Scala, you can still learn a lot from these videos.

---

## Introduction to ZIO Direct Style

ZIO Direct Style is a library that allows using directly-style i.e. imperative programming with ZIO effects which is based on the Monadless paradigm.

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-direct/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-direct_3.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-direct_3/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-direct_3.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-direct_3/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-direct-docs_3/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-direct-docs_3) [![ZIO Direct Style](https://img.shields.io/github/stars/zio/zio-direct?style=social)](https://github.com/zio/zio-direct)

## Installation

To use zio-direct, add the following to your `build.sbt` file.

```scala
libraryDependencies += "dev.zio" %% "zio-direct" % "1.0.0-RC7"
```

You can also use zio-direct with ZStream and ZPure by importing the following modules.
> Currently this is only supported in Scala 3.

```scala
// ZStream
libraryDependencies += "dev.zio" %% "zio-direct-streams" % "1.0.0-RC7"
// ZPure
libraryDependencies += "dev.zio" %% "zio-direct-pure" % "1.0.0-RC7"
```
See the [Other Supported Monads](other-supported-monads) section for more details.

## IDE Support

The preferred IDE to use with ZIO-Direct is Visual Studio Code + Metals. This is because Metals correctly reads the returns from `defer` calls directly from the Scala compiler which is not the case of IntelliJ. To remedy this issue, a Library Extension is provided for ZIO-Direct. See the [IntelliJ Support](intellij-support) section for more details.

## Introduction

Talk at Functional Scala 2022:
* Video - https://www.youtube.com/watch?v=DgqfLfzq5eo
* Slides - https://www.slideshare.net/deusaquilus/ziodirect-functional-scala-2022

ZIO-Direct allows direct style programming with ZIO. This library provides a *syntactic sugar* that is more powerful than for-comprehensions as well as more natural to use. Simply add the `.run` suffix to any ZIO effect in order to retrieve its value.

ZIO-Direct works by using macros to rewrite sequential code into flatMap-chains based on the [Monadless](https://github.com/monadless/monadless) paradigm. The values resulting in `.run` calls from the ZIO effects are not actually awaited. Instead, they are rolled-up into a chain of flatMaps.

For example, in imperative programming operations typically are done in a simple set of steps.
```scala
object FileOps:
  def read(file: File): String
  def write(file: File, content: String): Unit

val textA = read(fileA)
val textB = read(fileB)
write(fileC, textA + textB)
```

Using functional programming, the equivalent of this functionality is a set of nested flatMap-chains.
```scala
object FileOps
  def read(file: File): ZIO[Any, Throwable, String]
  def write(file: File, content: String): ZIO[Any, Throwable, Unit]

read(fileA).flatMap { textA =>
  read(fileB).flatMap { textB =>
    write(fileC, textA + textB)
  }
}
```

In order to avoid this complexity scala provides a for-comprehension syntactic sugar.
```scala
for {
  textA <- read(fileA)
  textB <- read(fileB)
  _ <- write(fileC, textA + textB)
} yield ()
```

Unfortunately this syntactic sugar is limited in many cases, for example, inserting a conditional value inside is impossible.
```scala
for {
  textA <- read(fileA)
  // Not a possible syntax
  if (fileA.contains("some string")) {
    textB <- read(fileB)
    _ <- write(fileC, textA + textB)
  }
} yield ()
```

ZIO-Direct offers an equivalent syntactic sugar that is more ergonomic and allows many constructs that for-comprehensions do not.
```scala
defer {
  val textA = read(fileA).run
  if (fileA.contains("some string")) {
    val textB = read(fileB).run
    write(fileC, textA + textB).run
  }
}
```

# ZIO-Tailored
ZIO-Direct is specifically tailored to ZIO capabilities as it supports Environment and Error composition in ZIO effects similar to the for-comprehension.

```scala
val out: ZIO[CustomerConfig & DistributorConfig, CustomerGetException | DistrubutorGetException, (Customer, Distributor)] =
  defer {
    // Get a customer-configuration object from the environment and extract its .url field
    val custUrl: String = ZIO.service[CustomerConfig].run.url
    // Get a distributor-configuration from the environment and extract its .url field
    val distUrl: String = ZIO.service[DistributorConfig].run.url
    (
      // Use the two configurations to make an HTTP-call
      parseCustomer(httpGetCustomer(custUrl).run),
      parseDistrubutor(httpGetDistributor(distUrl).run)
    )
  }
```

## Branching and Looping Support
Unlike the for-comprehension, ZIO-Direct supports branching and looping in the use of flatMaps composition.
Let's have a look at a another non-trivial example.

```scala
class Database:
  def nextRow(): ZIO[Any, Throwable, Row]
  def hasNextRow(): ZIO[Any, Throwable, Boolean]
  def lockNextRow(): ZIO[Any, Throwable, Boolean]
object Database:
  def open: ZIO[Any, Throwable, Database]

defer {
  // Open a database connection
  val db = Database.open().run
  // See if there is is a next-row
  while (db.hasNextRow().run) {
    // try to lock, if aquired continue
    if (db.lockNextRow().run)
      val nextRow = db.nextRow().run
      doSomethingWith(nextRow)
    else
      waitT()
  }
}
```
> NOTE: The above database-api is imaginary.

The above code needs to be translated into something like this:

```scala
Database.open.flatMap { db =>
  def whileFun(): ZIO[Any, Throwable, Unit] =
    db.hasNextRow().flatMap { hasNextRow =>
      if (hasNextRow)(
        db.lockNextRow().flatMap { lockNextRow =>
          if (!lockNextRow)
            db.nextRow().map(nextRow => doSomethingWith(nextRow))
          else
            ZIO.succeed(waitT())
        }
      ).flatMap(_ => whileFun())
      else
        ZIO.unit
    }
  whileFun()
}
```

Note that normally this is the exact code that would have to be written to capture such functionality For-comprehensions do not provide a way to do looping and branching so in such cases
a combination of flatMaps and recursion is necessary to avoid calling effects unnecessarily.

## Great for Refs and FiberRefs!

ZIO-direct makes it much easier to use ZIO mutable Ref and FiberRef variables. Since retrieveing and updating Ref and FiberRef variables requries a flatMap/for-comprehension call, it is typically very difficult to use them with branching/looping constructs. ZIO-direct makes these cases much easier.

```scala
class DatabaseApi {
  val connRef = FiberRef.make[Option[Connection]](None)
  def openConnection(): Connection = lowLevelDatabaseApi.openConnection()
  def transaction(action: Action) =
    defer {
      val conn = connRef.get.run
      if (conn == None) {
        connRef.set(Some(openConnection())).run
      }
      val conn1 = connRef.get.run
      conn.execute(action).run
    }
}
```

Instead of having to write the following code:
```scala
class DatabaseApi {
  val connRef = FiberRef.make[Option[Connection]](None)
  def openConnection(): Connection = lowLevelDatabaseApi.openConnection()
  def transaction(action: Action) =
    connRef.get.flatMap { conn =>
      (if (conn == None) {
        connRef.set(Some(openConnection()))
      } else {
        ZIO.unit
      }).flatMap(_ => connRef.get.flatMap { conn1 =>
        conn1.execute(action)
      })
    }

  // Note that for-comprehensions do not help very much in this use-case
  def transaction0(action: Action) =
    for {
      conn <- connRef.get
      _ <- if (conn == None) {
        connRef.set(Some(openConnection()))
      } else {
        ZIO.unit
      }
      conn1 <- connRef.get
      _ <- conn1.execute(action)
    }
}
```

---

## ZIO-Direct IntelliJ Support

# IntelliJ Support

> It is highly recommended that you use Metals instead of IntelliJ for ZIO-Direct development since Metals correctly infers types for `defer` blocks (and other whitebox macros). This is especially true in Scala 3 where IntelliJ does not properly support union-types upon which the `defer` mechanism relies for ZIO error type composition.

It is well known that IntelliJ does not support type inference with whitebox macros because it does not read types from the Scala compiler.
Since `defer` is a whitebox macro, IntelliJ will not be able to infer the type of the `defer` block.

![IntelliJ Not Works](img/intellij-not-works.png)

To circumvent this limitation, IntelliJ provides a mechanism to load custom library-specific
plugins called Library-Extensions that can provide type information to the IDE (info [here](https://github.com/JetBrains/intellij-scala/wiki/Library-Extensions)).
Library-Extension plugins are automatically loaded by IntelliJ when a library like zio-direct "asks" for one
by providing a configuration file in the library's jar.

ZIO-Direct provides a Library-Extension plugin that is loaded by IntelliJ to provide type information.

## Installation

1. When adding a library dependency on zio-direct in the build.sbt file:
    ```scala
    // Build.sbt
    libraryDependencies += "dev.zio" %% "zio-direct" % "..."
    ```
2. Click the `reload` button in the SBT panel of the IntelliJ project.

    ![IntelliJ SBT Reload](img/intellij-sbt-reload.png)

3. Once the SBT configuration is reloaded, the following message will appear in the bottom right corner of the IntelliJ project window. Click "Yes".

    ![IntelliJ Import Plugin](img/intellij-import-plugin.png)

4. IntelliJ will automatically download the zio-direct library and the Library-Extension plugin. The typing of `defer` blocks should then work as expected.

    ![IntelliJ Works](img/intellij-works.png)

## Troubleshooting

If the "Extensions Available" dialog does not appear or a manual reload of the zio-direct-intellij plugin is required. This typically involves manually clearing the imported libraries and then forcing a re-import from SBT. This is necessary so that IntelliJ runs the correct triggers to load the zio-direct-intellij plugin.

1. Go to the `Project Structure` -> `Libraries` dialog. Select all the libraries (Cmd+A) and click the `-` button.

    ![IntelliJ Clear Libraries](img/intellij-clear-libraries.png)

2. Once this is complete and all the libraries are removed. Go back to the SBT panel and click the `reload` button.

    ![IntelliJ SBT Reload](img/intellij-sbt-reload.png)

3. Once the SBT project structure is reloaded, the "Extensions Available" dialog should appear.

    ![IntelliJ Import Plugin](img/intellij-import-plugin.png)

## Caveats

#### Type Incongruence
Since the zio-direct-intellij plugin is still experimental, the types that it infers may be incorrect. Use the `defer.info` function to check the Scala-compiler inferred type of the `defer` block.

![IntelliJ Info](img/intellij-info-check.png)

#### Scala 3 Union-Type Limitations

Additionally, since IntelliJ does not properly support union-types, it is impossible to correctly infer the error type of a `defer` that composes multiple ZIO effects with errors in Scala 3. Instead, the zio-direct-intellij plugin will use a least-upper bound instead of a union-type.

![IntelliJ Lub VS Console](img/intellij-lub-vs-console.png)

While in practice, this is not a problem since the least-upper bound is always a supertype of the union-type, it is still a limitation of the zio-direct-intellij plugin. If any unexpected issues occur, you can force zio-direct to infer least-upper-bound types by using `defer(Use.withAbstractError) { ... }` however as this setting is not fully supported (e.g. it does not correctly infer error-types that contain type-parameters) it must be used with caution.

---

## Other Supported Monads

As of RC5, zio-direct now supports ZStream and ZPure as well as `scala.concurrent.Future` and `scala.List`. The latter two are largely fully functional but largely for demonstration purposes.
> Note that all of these are currently only supported in Scala 3.

## ZStream

To use zio-direct with ZStream, add the following to your `build.sbt` file.
```scala
libraryDependencies += "dev.zio" %% "zio-direct-streams" % "1.0.0-RC7"
```

You can then use zio-direct with ZStream:
```scala

val out =
  defer {
    val a = ZStream(1, 2, 3).each
    val b = ZStream("foo", "bar").each
    (a, b)
  }

out.runCollect
// ZIO.succeed(Chunk((1,"foo"),(1,"bar"),(2,"foo"),(2,"bar"),(3,"foo"),(3,"bar")))
```

Note that if you are also using zio-direct with ZIO, you should rename the `defer` function to avoid conflicts:

```scala

// The `run` function of ZStream is called `each`
val outStream: ZStream[Any, Nothing, (Int, String)] =
  deferStream {
    val a = ZStream(1, 2, 3).each
    val b = ZStream("foo", "bar").each
    (a, b)
  }

val outZio: ZIO[Any, Nothing, Chunk[(Int, String)]] =
  defer {
    val a: Chunk[(Int, String)] = outStream.runCollect.run
    val b = ZIO.succeed((123, "baz")).run
    a :+ b
  }

// Yields:
// ZIO.succeed(Chunk((1,"foo"),(1,"bar"),(2,"foo"),(2,"bar"),(3,"foo"),(3,"bar"),(123, "baz")))
```

## ZPure

> Note that Metals auto-complete/type-info popups may be sluggish when using ZPure, especially when try/catch constructs are being used.
> In some cases, you may need to wait for a "Loading..." popup message for 20-30 seconds although the actual (bloop) compile time
> will just be a few seconds.

To use zio-direct with ZPure, add the following to your `build.sbt` file.
```scala
libraryDependencies += "dev.zio" %% "zio-direct-pure" % "1.0.0-RC7"
```

In order to use zio-direct with ZPure, you first need to define a `deferWith[W, S]` context which will define the Logging (`W`) and State (`S`) types for your ZPure computation.

> Due to limitations of Scala 3, you may need to create the state object type in a separate file (or you may get cyclical-dependency compile-time errors).
```scala
val dc = deferWith[String, MyState]

// The `run` function of ZStream is called `eval`
val out =
  defer {
    val s1 = ZPure.get[MyState].eval.value
    ZPure.set(MyState("bar")).eval
    val s2 = ZPure.get[MyState].eval.value
    (s1, s2)
  }

out.provideState(MyState("foo")).run
// ("foo", "bar")
```

In order to avoid having to specify the state-type over and over again, several helpers are provided (they are imported from `dc._`).
```scala
val dc = deferWith[String, MyState]

// The `run` function of ZStream is called `eval`
val out =
  defer {
    val s1 = getState().value
    setState(MyState("bar"))
    val s2 = getState().value
    (s1, s2)
  }

out.provideState(MyState("foo")).run
// ("foo", "bar")
```

## List and Future

Support for Scala's `List` and `Future` objects is provided from zio-direct.

To use zio-direct with `List` do the following:
```scala

val out =
  defer {
    val a = List(1, 2, 3)
    val b = List("foo", "bar")
    (a, b)
  }

// Yields:
// List((1,"foo"),(1,"bar"),(2,"foo"),(2,"bar"),(3,"foo"),(3,"bar"))
```

To use zio-direct with `Future` do the following:
```scala

val out =
  defer {
    Future("a").run match {
      case "a" => Future(1).run
      case "b" => Future(2).run
    }
  }

// Yields: Future(1)
```

Note that it is not necessary to implement ExecutionContext.Implicits.global. You can
implicitly pass in any ExecutionContext you want. It just needs to be in-scope when you
call the `defer` function (i.e. `zio.direct.future.defer`).
```scala

def out(implicit ctx: ExecutionContext) =
  defer {
    Future("a").run match {
      case "a" => Future(1).run
      case "b" => Future(2).run
    }
  }

out(scala.concurrent.ExecutionContext.global)
// Yields: Future(1)
```

---

## Scala Constructs Supported by ZIO-Direct

# Supported Constructs

ZIO-direct supports using the following constructs inside of a `defer` block. Approximate translations of the what the Scala code looks like are available below. In order to see the exact translations for any code in a defer block, use `defer.info`.

### blocks

```scala
defer {
  val a = ZIO.succeed("Hello").run
  val b = ZIO.succeed("World").run
  a + " " + b
}
```
Translation:
```scala
ZIO.succeed("Hello").flatMap { a =>
  ZIO.succeed("World").map { b =>
    a + " " + b
  }
}
```

Blocks can also have nested blocks.
```scala
defer {
  val a = ZIO.succeed("Hello").run
  val b = {
    val x = ZIO.succeed("to").run
    val y = ZIO.succeed("World").run
    x + " " + y
  }
  a + " " + b
}
```
Translation:
```scala
ZIO.succeed("Hello").flatMap { a =>
  {
    ZIO.succeed("to").flatMap { x =>
      ZIO.succeed("World").map { y =>
        x + " " + y
      }
    }
  }.map { b =>
    a + " " + b
  }
}
```

### if/else
If statements with one or multiple ZIO.run values in the condition(s) and action(s).

```scala
defer {
  if (ZIO.succeed(123).run < 456 && ZIO.succeed("foo") == "foo")
    ZIO.succeed("a").run
  else
    ZIO.succeed("b").run
}
```

Translation:
> Note that each condition is separated into it's own nested flatMap chain step
(from left-to-right) so if earlier conditions yield `false` ZIO computations of
later ones will not be executed.
```scala
ZIO.succeed(123).flatMap { a =>
  if (a < 456)
    ZIO.succeed("foo").flatMap { b =>
      if (b == "foo")
        ZIO.succeed("a")
      else
        ZIO.succeed("b")
    }
  else
    ZIO.succeed("b")
}
```

### match

Match statements with ZIO.run in the left-hand-side (before "match") and/or the right-hand-side (after the "=>").
ZIO.run calls inside of match guards (i.e. if-statements after `case Clause`) are not supported yet.

```scala
defer {
  ZIO.succeed("Hello").run match {
    case hello @ "Hello" =>
      val world = ZIO.succeed(" World").run
      hello + " " + world
    case _ =>
      "Nothing"
  }
}
```

Translation:
```scala
ZIO.succeed("Hello").flatMap { x =>
  x match {
    case hello @ "Hello" =>
      ZIO.succeed(" World").flatMap { world =>
        hello + " " + world
      }
    case _ =>
      ZIO.succeed("Nothing")
  }
}
```

### try

Try statements with ZIO.run in the left-hand-side (before "try") and/or the right-hand-side (after the "=>").

```scala
defer {
  try {
    val a = ZIO.succeed(123).run
    val b = ZIO.attempt(somethingUnsafe).run
    a + b
  } catch {
    case e: Exception =>
      ZIO.succeed(789).run
  }
}
```

Translation:
```scala
ZIO.succeed(123).flatMap { a =>
  ZIO.attempt(somethingUnsafe).map { b =>
    a + b
  }.catchAll { e =>
    ZIO.succeed(789)
  }
}
```
Note that because try-statements are translated into ZIO.catchAll, errors that go into fail fail-channel
will not be caught by the catch block. For example:

```scala
def throwsException() = throw new Exception("foo")

defer {
  try {
    // Will not be caught!!
    ZIO.succeed(throwsException()).run
  } catch {
    case e: Exception => 123
  }
}

defer {
  try {
    // WILL be caught!!
    ZIO.attemt(throwsException()).run
  } catch {
    case e: Exception => 123
  }
}
```

In cases where methods that throw exceptions not not wrapped into ZIO computations, they will also
not be caught because the assumption is that they are pure-computations hence can be wrapped into `ZIO.succeed` blocks.

```scala
def throwsException() = throw new Exception("foo")

defer {
  try {
    // Will not be caught!!
    throwsException()
  } catch {
    case e: Exception => 123
  }
}

// Translation:
ZIO.succeed(throwsException()).catchAll { e =>
  case e: Exception => 123
}
```

In order to rectify this situation, a region-based operator `unsafe { ... }` can be used to wrap
all blocks of code that could potentially throw exceptions. ZIO-Direct will the know to wrap
them into `ZIO.attempt` clauses instead of `ZIO.succeed`.
```scala
def throwsException() = throw new Exception("foo")

defer {
  try {
    unsafe {
      // This WILL be caught!!
      throwsException()
    }
  } catch {
    case e: Exception => 123
  }
}

// Translation:
ZIO.attempt(throwsException()).catchAll { e =>
  case e: Exception => 123
}
```

Note that that ZIO computations with `.run` calls and other kinds of constructs supported by zio-direct
can be used inside of `unsafe` blocks, and these computations will be used as-is (i.e. if they contain
ZIO.succeed calls the will not be changed into something else).

```scala
defer {
  try {
    unsafe {
      val a = ZIO.succeed(123).run
      throwsException()
      val b = ZIO.succeed(456).run
      a + b
    }
  } catch {
    case e: Exception => 123
  }
}

// Translation:
ZIO.succeed(123).flatMap { a =>
  ZIO.attempt(throwsException()).flatMap { _ =>
    ZIO.succeed(456).map { b =>
      a + b
    }
  }
}.catchAll { e =>
  case e: Exception => 123
}
```

### while

While-clauses will be translated into recursive functions that conditionally recurse into a flatMap call based on the while-condition.

> Generally due to the presence of functions like [`ZIO.iterate`](https://zio.dev/reference/control-flow/#iterate) and [`ZIO.repeat`](https://zio.dev/reference/test/aspects/repeat-and-retry/#repeat) the critical use-case for ZIO-direct's while-loop should be limited.

```scala
// Note that because mutable variable usage is generally not allowed in zio-direct the below code can only be run in "Lenient Mode."
var i = 0
defer {
  while (i < 10) {
    ZIO.attempt(println("Hello")).run
    i += 1
  }
}

// Translation:
val i = 0
def loop(): ZIO[Any, Throwable, Unit] = {
  if (i < 10) {
    ZIO.attempt(println("Hello")).flatMap { _ =>
      i += 1
      loop()
    }
  } else {
    ZIO.unit
  }
}
loop()
```

Since mutable variables are generally not allowed in `defer { ... }` blocks, it is recommended to use mutable references from ZIO's
[`Ref`](https://zio.dev/reference/concurrency/ref/) class instead.

```scala
defer {
  val ref = Ref.make(0).run
  while (ref.get.run < 10) {
    ZIO.attempt(println("Hello")).run
    ref.update(_ + 1).run
  }
}

// Translation:
Ref.make(0).flatMap { ref =>
  def loop(): ZIO[Any, Throwable, Unit] = {
    ref.get.flatMap { x =>
      if (x < 10) {
        ZIO.attempt(println("Hello")).flatMap { _ =>
          ref.update(_ + 1).flatMap { _ =>
            loop()
          }
        }
      } else {
        ZIO.unit
      }
    }
  }
  loop()
}
```

### for-loop/foreach

Scala for-loops and collection.foreach are the same thing (the former dis desugars into the latter).
ZIO-direct will translate them into ZIO.foreach calls.

> Similar to while-loops, this construct is
largely overshawoed by ZIO's own [`foreach`](https://zio.dev/reference/control-flow/#iterate) and [`iterate`](https://zio.dev/reference/control-flow/#foreach) combinators.

```scala
defer {
  for (i <- 1 to 10) {
    ZIO.attempt(println(i)).run
  }
}

// Translation:
ZIO.foreach(1 to 10) { i =>
  ZIO.attempt(println(i))
}.map(_ => ()) // since the final result must have a type of Unit
```

---

## Architecture


In the below diagram you can see the different layers of library and its dependencies.

- The user can choose to call the High Level Type Safe API which is based on types from Scala case classes (in green) or the Low Level Type Safe API (in blue) which is based on DynamoDB specific abstractions like `AttributeValue`. 
- **High Level Type Safe API** has a dependency on [ZIO Schema](https://zio.dev/zio-schema/) for compile time type information that is used to automatically generate codecs and in turn calls the ...
- **Low Level API** for query creation and execution, and which has in turn has a dependency on [ZIO AWS](https://zio.dev/zio-aws/), which in turn has a dependency on the AWS Java SDK.

![architecture diagram](architecture.png)

Although there are two APIs, the High Level API is the recommended way to interact with the library as it provides a type safe way to interact with DynamoDB together with a huge amount of boilerplate reduction.

---

## High Level API

The high level API relies on two mechanisms to provide type safety and reduce boilerplate code:

1. Automatic ZIO Schema derivation for case classes
2. Semi automatic generation of `ProjectionExpression`'s for case classes and sealed traits 

## ZIO Schema derivation

The High Level API methods rely on a ZIO Schema for a particular type being in implicit scope. This is achieved using the `DeriveSchema.gen` macro. Internally codecs are automatically generated for the case classes based on the meta data provided by the `Schema`'s.

```scala
final case class Person(id: String, name: String)
object Person {
  implicit val schema: Schema.CaseClass2[String, String, Person] = DeriveSchema.gen[Person]
}
```

## Projection Expression generation

The automated generation of `ProjectionExpression`'s is achieved using the `ProjectionExpression.accessors` method. For classes this method generates a `ProjectionExpression` for each field. For sealed traits it generates a `ProjectionExpression` for each child.

```scala
final case class Person(id: String, name: String)
object Person {
  implicit val schema // ...

  val (id, name) = ProjectionExpression.accessors[Person]
}
```

## Using a `ProjectionExpression` as a springboard to creating further expressions

In the above example `Person.id` and `Person.name` are `ProjectionExpression`s automatically generated by the `ProjectionExpression.accessors` method.
They are used as a springboard for creating further type safe APIs eg 
- `Person.id === "1"` creates a `ConditionExpression`
- `Person.name.set("Smith")` creates an `UpdateExpression`
- `Person.id.partitionKey === "1"` creates a primary key expression

## `DynamoDBQuery` CRUD methods

There are also type safe query creation methods in the `DynamoDBQuery` companion object such as `get`, `put`, `update`, `deleteFrom`, `queryAll` and all these take expressions as arguments. So taking our example further we can see how all these APIs can be used together to create a type safe CRUD queries: 

```scala
final case class Person(id: String, name: String)
object Person {
  implicit val schema: Schema.CaseClass2[String, String, Person] = DeriveSchema.gen[Person]
  val (id, name) = ProjectionExpression.accessors[Person]
}

val table = "person-table"
val person = Person("1", "John")
for {
  _ <- DynamoDBQuery.put(table, person).where(!Person.id.exists).execute
  found <- DynamoDBQuery.get(table)(Person.id.partitionKey === "1").execute.absolve
  _ <- DynamoDBQuery.update(table)(Person.id.partitionKey === "1")(
    Person.name.set("Smith")
  ).execute
  _ <- ZIO.debug(found == person) // true
} yield ()
```

---

## Low Level API

The low level API provides low level query creation and execution while still offering a large reduction in boilerplate. It's is based one to one on DynamoDB abstractions and the surface area consists of:
- `AttrMap` which is a convenience container with automatic conversion between Scala values and `AttributeValue`'s. It also has type aliases of `PrimaryKey` and `Item` (see reference section for more details)
- `$` function for creating untyped `ProjectionExpression`s eg `$("person.address.houseNumber")`
- query methods in the `DynamoDBQuery` companion object that all contain the word `Item` such as `getItem`, `putItem`, `updateItem`, `deleteItem`, `queryAllItem`. 
- Expressions - most of these query methods take expressions as arguments. Once we have a `ProjectionExpression` via the dollar function we can use it as a springboard to create further expressions such as `ConditionExpression`, `UpdateExpression` and `PrimaryKeyExpression`:
  - `ConditionExpression`'s - `$("name") === "John"`
  - `UpdateExpression`'s - `$("name").set("Smith")`

Examples of low level queries are shown below:
```scala
for {
  _ <- DynamoDBQuery.putItem("person-table", Item("id" -> "1", "name" -> "John", "age" -> 42)).execute
  maybeFound <- DynamoDBQuery.getItem("person-table")(PrimaryKey("id" -> "1")).execute
  _ <- DynamoDBQuery.updateItem("person-table")(PrimaryKey("id" -> "1"))($("name").set("Smith") + $("age").set(21)).execute
  _ <- DynamoDBQuery.deleteItem("person-table")(PrimaryKey("id" -> "1")).execute
} yield ()
```

However there are some caveats to using the Low Level API:
- It is not type safe - subtle runtime errors can occur - for example if there are typos in the field names, or if incompatible types are used in expressions.
- Serialization and deserialization of case classes is the responsibility of the user - this is usually a major burden.

---

## Cats Effect Interop

The **`zio-dynamodb-ce`** cats effect interop module provides a way to use ZIO DynamoDB with Cats Effect 3 with minimal 
effort.

## Usage

Add the following line to your `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb-ce" % "1.0.0-RC24"
)
```

The entry points are the `DynamoDBExecutorF.of` and `DynamoDBExecutorF.ofCustomised` constructors which provide a `Resource` managed
`DynamoDBExecutorF` instance. Once we have this instance in implicit scope we can use the extension method `executeToF` 
to run the queries. Queries that would normally return a `ZIO` effect now return a `F` effect, and queries that would
normally return a `ZStream` now return an FS2 `Stream`.

## Example

```scala
package zio.dynamodb.examples.dynamodblocal.interop

/**
 * example cats effect interop application
 *
 * to run in the sbt console:
 * {{{
 * zio-dynamodb-examples/runMain zio.dynamodb.examples.dynamodblocal.interop.CeInteropExample
 * }}}
 */
object CeInteropExample extends IOApp.Simple {

  final case class Person(id: String, name: String)
  object Person {
    implicit val schema: Schema.CaseClass2[String, String, Person] = DeriveSchema.gen[Person]
    val (id, name)                                                 = ProjectionExpression.accessors[Person]
  }

  def program[F[_]](implicit F: Async[F]) = {
    val console = Console.make[F]

    for {
      _ <- DynamoDBExceutorF
             .ofCustomised[F] { builder => // note only AWS SDK model is exposed here, not zio.aws
               builder
                 .endpointOverride(URI.create("http://localhost:8000"))
                 .region(Region.US_EAST_1)
                 .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("dummy", "dummy")))
             }
             .use { implicit dynamoDBExecutorF => // To use extension method "executeToF" we need implicit here
               for {
                 _         <- createTable("Person", KeySchema("id"), BillingMode.PayPerRequest)(
                                AttributeDefinition.attrDefnString("id")
                              ).executeToF
                 _         <- put(tableName = "Person", Person(id = "avi", name = "Avinder")).executeToF
                 result    <- get(tableName = "Person")(Person.id.partitionKey === "avi").executeToF
                 _         <- console.println(s"found=$result")
                 fs2Stream <- DynamoDBQuery
                                .scanAll[Person](tableName = "Person")
                                .parallel(50) // server side parallel scan
                                .filter(Person.name.beginsWith("Avi") && Person.name.contains("de"))
                                .executeToF
                 _         <- fs2Stream.evalTap(person => console.println(s"person=$person")).compile.drain
                 _         <- deleteTable("Person").executeToF
               } yield ()
             }
    } yield ()
  }

  val run = program[IO]
}
```

---

## High Level API Cheat Sheet

Note this guide assumes the reader has some basic knowledge of [AWS DynamoDB API](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html).

Assuming the below model
```scala
final case class Person(id: String, name: String, year: Int)
object Person {
  implicit val schema: Schema.CaseClass3[String, String, Int, Person] = DeriveSchema.gen[Person]
  val (id, name, year) = ProjectionExpression.accessors[Person]
}
```

For more detailed working examples please see the High Level API integration tests [crud](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiCrudSpec.scala), [mapping](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiMappingSpec.scala), [scan and query](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeScanAndQuerySpec.scala), [streaming](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeStreamingUtilsSpec.scala)

| AWS                           | ZIO DynamoDB |
|-------------------------------| --- |
| [GetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html)                       | `person <- get("personTable")(Person.id.partitionKey === "1").execute` |
| [UpdateItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html)                    | `_ <- update("personTable")(Person.id.partitionKey === "1")(Person.name.set("Foo")).execute` |
| [PutItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html)                       | `_ <- put('personTable', Person('42', 'John', 2020)).execute` |
| [DeleteItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html)                    | `_ <- deleteFrom("personTable")(Person.id.partitionKey === "1").execute` |
|                               | |
| [Projection Expressions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ProjectionExpressions.html)        | `Person.id`, `Person.name`, `Person.year`  |
| [Condition Expressions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html)         | `<DynamoDBQuery>.where(Person.id === "1")` |
| **Filter Expressions** apply to Scan and Query | `<DynamoDBQuery>.filter(Person.year > 2020)`   |
| [Update Expressions](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html#DDB-UpdateItem-request-UpdateExpression)            | `update("personTable")(Person.id.partitionKey === "1")(Person.name.set("John") + Person.year.add(1))` |
| [Primary Keys](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html)                  | `Person.id.partitionKey === "1"` or `Person.id.partitionKey === "1" && Person.year.sortKey === 2020` if table has both partition _and_ sort keys |
| [Key Condition Expressions](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-KeyConditionExpression)     | `<query>.whereKey(Person.id.partitionKey === "1" && Person.year.sortKey > 2020)` |
| [_Expression Attribute Names_](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ExpressionAttributeNames.html)  | _Managed automatically!_ |
| [_Expression Attribute Values_](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ExpressionAttributeValues.html) | _Managed automatically!_ |
|                               | |
| [Scan](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html)                          |	`stream <- scanAll[Person]("personTable").execute`
| [Scan with parallel processing](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) |	`stream <- scanAll[Person]("personTable").parallel(42).execute`
| [Scan with paging](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html)              |	`(people, lastEvaluatedKey) <- scanSome[Person]("personTable", limit = 5).startKey(oldLastEvaluatedKey).execute`
| [Query](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html)                         |	`stream <- queryAll[Person]("personTable").whereKey(Person.id.partitionKey === id).execute`
| [Query with paging](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html)             |	`(people, lastEvaluatedKey) <- querySome[Person]("personTable", limit = 5).whereKey(Person.id.partitionKey === "1").execute`
| | |
| [BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html) | `people <- DynamoDBQuery.batch(listOfIds)(id => DynamoDBQuery.get[Person]("personTable")(Person.id.partitionKey === id)).execute`|
| [BatchWriteItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) | `_ <- DynamoDBQuery.batch(people)(p => put("personTable", p)).execute` |
| | |
| [TransactGetItems](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactGetItems.html) | `tuple <- (get("personTable")(Person.id.partitionKey === "1") zip get("employeeTable")(Employee.id.partitionKey === "2")).transaction.execute` Note transactions can span different tables. |
| [TransactWriteItems](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html) | `_ <- (put("personTable", Person(1, "John", 2020)) zip put("employeeTable", Person(2, "Smith", 2024))).transaction.execute` Note transactions can span different tables. |

---

## Codec Customization

## Sealed trait members that are case classes

```scala
sealed trait TrafficLight
final case class Green(rgb: Int) extends TrafficLight 
final case class Red(rgb: Int) extends TrafficLight
final case class Box(trafficLightColour: TrafficLight)
```

The default encoding for `Box(Green(42))` is:

`Map(trafficLightColour -> Map(String(Green) -> Map(String(rgb) -> Number(42))))`

Here an intermediate map is used to identify the member of `TrafficLight` using the member class name ie `Map(String(Green) -> Map(...))`

## Sealed trait members that are case objects

```scala
sealed trait TrafficLight
case object GREEN extends TrafficLight 
case object RED extends TrafficLight
final case class Box(trafficLightColour: TrafficLight)
```

The default encoding for `Box(GREEN)` is:

`Map(trafficLightColour -> Map(String(GREEN) -> Null))`

Here an intermediate map is used to identify the member of `TrafficLight` ie `Map(String(GREEN) -> Null)`
Note that the `Null` is used as in this case we do not care about the value.

# Customising encodings via annotations
Encodings can be customised through the use of the following annotations `@discriminatorName` and `@fieldName`.
These annotations are useful when working with a legacy DynamoDB database.

The `@discriminatorName` encodings does not introduce another map for the purposes of identification but rather adds another 
discriminator field to the attribute Map.

Concrete examples of using the `@discriminatorName` and `@field` annotations can be seen below.

## Sealed trait members that are case classes

```scala
@discriminatorName("light_type")
sealed trait TrafficLight
final case class Green(rgb: Int) extends TrafficLight
@caseName("red_traffic_light")
final case class Red(rgb: Int) extends TrafficLight
final case class Amber(@fieldName("red_green_blue") rgb: Int) extends TrafficLight
final case class Box(trafficLightColour: TrafficLight)
```

encoding for an instance of `Box(Green(42))` would be:

`Map(trafficLightColour -> Map(String(rgb) -> Number(42), String(light_type) -> String(Green)))`

We can specify the field name used to identify the case class through the `@discriminatorName` annotation. The discriminator
encoding removes the intermediate map and inserts a new field with a name specified by discriminator annotation and a
value that identifies the member which defaults to the class name.

This can be further customised using the `@caseName` annotation - encoding for an instance of `Box(Red(42))` would be:

`Map(trafficLightColour -> Map(String(rgb) -> Number(42), String(light_type) -> String(red_traffic_light)))`

The encoding for case class field names can also be customised via `@fieldName` - encoding for an instance of `Box(Amber(42))` would be:

`Map(trafficLightColour -> Map(String(red_green_blue) -> Number(42), String(light_type) -> String(Amber)))`

## Sealed trait members that are all case objects

```scala
sealed trait TrafficLight
case object GREEN extends TrafficLight 
@caseName("red_traffic_light")
case object RED extends TrafficLight
final case class Box(trafficLightColour: TrafficLight)
```

We can get a more compact and intuitive encoding of trait members that are case objects by using the `@simpleEnum`
annotation which encodes to just a value that is the member name. This annotation doesn't need to be added explicitly 
since this is done by the ZIO Schema macro automatically.
Encoding for an instance of `Box(GREEN)` would be:

`Map(trafficLightColour -> String(GREEN))`

This can be further customised by using the `@caseName` annotation again - encoding for `Box(RED)` would be

`Map(trafficLightColour -> String(red_traffic_light))`

## No disriminator codecs using @noDiscriminator annotation

```scala
@noDiscriminator
sealed trait TrafficLight
@caseName("blue")
final case object Blue extends TrafficLight
final case object Purple extends TrafficLight 
final case class Green(rgb: Int, i: Int) extends TrafficLight
@caseName("red_traffic_light") // this annotation is ignored in the context of @noDiscriminator
final case class Red(rgb: Int, j: Int) extends TrafficLight
final case class Amber(@fieldName("red_green_blue") rgb: Int) extends TrafficLight
final case class Box(trafficLightColour: TrafficLight)
```

This primarily useful when working with legacy DynamoDB databases where a discriminator field is not present (some DynamoDB mapping libraries allow users to create codecs with no discriminators or tags to disambiguate each sum type case). 

WARNING! - this leads to the inefficiency of having to try each case and checking for success, and also forces the dangerous assumption that all the sum type cases will be different when encoded. When decoding if there are ambiguities amongst the codecs for the sum type instances this is handled gracefully by returning a Left of a DynamoDBError.DecodingError

Mapping for `Box(Blue)` would be `Map(trafficLightColour -> String(blue))`

Mapping for `Box(Amber(42))` would be `Map(trafficLightColour -> Map(String(red_green_blue) -> Number(42))`

## Recommendations
For greenfield development it is recommended to use:
- the default encoding which uses an intermediate map (see the above default encoding) - (note this mapping does not work with top level sum types as it requires an intermediate map and partition keys must be scalar values) *or*
- `@discriminatorName` encoding ([see above](#customising-encodings-via-annotations)) if you would like a more compact encoding - note this *must* be used for top level sum types
- for deeply nested hierarchies avoid a top level sum type altogether and have a concrete top level class that uses only product and sum types - see [High Level API Data Modelling and Limitations](data-modelling#deep-oo-style-hierarchies-are-not-supported---modelling-data-using-product-and-sum-types)

---

## High Level API Data Modelling and Limitations

The High Level API relies heavily on ZIO Schema and inherits some of it's limitations, namely:

- A maximum of **22** fields per case class 
- Deep Object-Oriented hierarchies with abstract classes/fields are not supported - only hierarchies one level deep are supported

At first glance these seem limitations seem quite restrictive, however the next sections describe how these can be overcome.

## A maximum of **22** fields per case class
The high level API uses the Reified Optics feature of ZIO Schema to generate optics for case classes and sealed traits. This feature has a limitation of 22 fields per case class. 

This limitation is something to be aware of when designing your models. In practice however this can be overcome by using nested case classes and in the case of deeply nested hierarchies, by using product and sum types (see section below). 

## Deep OO Style hierarchies are not supported - modelling data using Product and Sum Types
Deep Object-Oriented like hierarchies with abstract classes/fields are not supported - only hierarchies one level deep are supported - again this is a limitation of ZIO Schema - however these limitations can be overcome by using product and sum types rather than inheritance.

```scala
object OopStyle {
  // ensure that all concrete classes have id field
  sealed trait Invoice {
    def id: Int
  }
  // (1) Intermediate abstraction with abstract fields - not supported by the ZIO Schema Reified Optics
  sealed trait Billed                                                 extends Invoice {
    def amount: Double
  }
  final case class BilledMonthly(id: Int, amount: Double, month: Int) extends Billed
  final case class BilledYearly(id: Int, amount: Double, year: Int)   extends Billed
  final case class PreBilled(id: Int, count: Int)                     extends Invoice
}

// FP style modelling uses pure data (product and sum types) rather than inheritance hence avoiding classes like (1)
object FPStyle {
  sealed trait BilledBody
  final case class BilledMonthly(month: Int) extends BilledBody
  final case class BilledYearly(year: Int)   extends BilledBody

  sealed trait InvoiceBody
  // (3) extract product refactoring
  final case class Billed(amount: Double, billedBody: BilledBody) extends InvoiceBody
  final case class PreBilled(count: Int)                          extends InvoiceBody

  // (2) extract product refactoring
  final case class Invoice(id: Int, body: InvoiceBody)
}
```

By using the FP approach to modelling we reduce the size of the concrete classes by factoring out common fields into a 
product type. 

For brevity the above examples do not show the full integration with ZIO Schema - [for a full example see this IT test](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiAlternateModeling.scala).

---

## Future Interop

The **`zio-dynamodb-future`** interop module provides a way to use ZIO DynamoDB with vanilla scala Futures with minimal 
effort.

## Usage

Add the following line to your `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb-future" % "1.0.0-RC24"
)
```

The entry point is `DynamoDBExecutorF.make` which allows the user to customise the DynamoDB client and is placed in implicit scope. 
Rather that using the normal `execute` which would return a `ZIO` effect, we import a syntax class `import zio.dynamodb.interop.future.syntax._`
which allows us to use the extension method `executeToF` to run the queries and via interop return a `Future`. At the end we use the `DynamoDBExecutorF.close()` method to release the underlying resources.

## Example

```scala
package zio.dynamodb.examples.dynamodblocal.interop

object FutureInteropExample extends App {
  implicit val ddbExec: DynamoDBExecutorF = DynamoDBExecutorF.make(
    buildNettyClient = identity,
    buildDynamoDbClient = _.endpointOverride(java.net.URI.create("http://localhost:8000"))
      .region(software.amazon.awssdk.regions.Region.US_EAST_1)
      .credentialsProvider(
        software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create(
          software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create("dummy", "dummy")
        )
      )
  )

  val program          = for {
    _      <- DynamoDBQuery
                .createTable("Person", KeySchema("id"), BillingMode.PayPerRequest)(
                  AttributeDefinition.attrDefnString("id")
                )
                .executeToF
    _      <- put(tableName = "Person", Person(id = "avi", name = "Avinder")).executeToF
    result <- get(tableName = "Person")(Person.id.partitionKey === "avi").executeToF
    _       = println(s"found=$result")
    _      <- DynamoDBQuery.deleteTable("Person").executeToF

  } yield ()
  val programWithClose = program.andThen { case _ => ddbExec.close() }

  Await.result(programWithClose, 30.seconds)
}
```

---

## Getting Started

## Add the dependency to your build.sbt file

```scala
libraryDependencies += "dev.zio" %% "zio-dynamodb" % 1.0.0-RC24
```

### Read & write data to/from DynamoDB

```scala

object Main extends ZIOAppDefault {

  final case class Person(id: Int, firstName: String)
  object Person {
    implicit lazy val schema: Schema.CaseClass2[Int, String, Person] = DeriveSchema.gen[Person]

    val (id, firstName) = ProjectionExpression.accessors[Person]
  }
  val examplePerson = Person(1, "avi")

  private val program = for {
    _      <- put("personTable", examplePerson).execute
    person <- get("personTable")(Person.id.partitionKey === 1).execute
    _      <- zio.Console.printLine(s"hello $person")
  } yield ()

  override def run =
    program.provide(
      netty.NettyHttpClient.default,
      config.AwsConfig.default, // uses real AWS dynamodb
      dynamodb.DynamoDb.live,
      DynamoDBExecutor.live
    )
}
```

---

## Testing ZIO DynamoDB Applications

### Full Stack Integration Testing

The recommendation is to use full-stack integration tests of your ZIO DynamoDB repository code against a "local" DynamoDB 
instance of which there are two choices:

- [DynamoDB Local JAR](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) - a stand-alone 
JAR file that can be run on your local machine, also is available as a docker image.
- [Docker LocalStack](https://docs.docker.com/guides/localstack/) - a docker image that provides a local AWS cloud stack including DynamoDB

You should use whichever is most convenient for your development environment. The ZIO DynamoDB library itself uses DynamoDB Local JAR 
as a docker image as the full functionality of LocalStack is not required. For examples of how to use DynamoDB Local 
please see the many ZIO DynamoDB integration tests.   

### Unit Testing Database Mapping using the ZIO DynamoDB JSON Module

There is also an opportunity to unit test Database mappings using the optional `zio-dynamodb-json` module. 
This module provides a way to render a case class to a JSON string that represents the native DynamoDB types, including
any discriminators for sum types - please see the [ZIO DynamoDB JSON](../reference/zio-dynamodb-json) section for more details.

---

## DynamoDB Transactions

[Transactions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html) are as simple as calling the `.transaction` method on a `DynamoDBQuery`. As long as every component of the query is a valid transaction item and the `DynamoDBQuery` does not have a mix of get and write transaction items. A list of valid items for both types of queries is listed below.

## Examples

Assuming the below model:
```scala
final case class Student(email: String, subject: String)
object Person {
  implicit lazy val schema: Schema.CaseClass2[String, String, Student] = DeriveSchema.gen[Student]

  val (email, subject) = ProjectionExpression.accessors[Student]
}
final case class Bill(email: String, amount: Int)
object Bill {
  implicit lazy val schema: Schema.CaseClass2[String, Int, Bill] = DeriveSchema.gen[Bill]

  val (email, amount) = ProjectionExpression.accessors[Bill]
}
final case class WaitList(email: String)
object Bill {
  implicit lazy val schema: Schema.CaseClass1[String, WaitList] = DeriveSchema.gen[WaitList]

  val email = ProjectionExpression.accessors[WaitList]
}
```

### [Write Transactions](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html)
```scala
val student = Student("avi@gmail.com", "maths")
val bill = Bill("avi@gmail.com", 1)

val putStudent = put("student", student)
val billedStudent = put("billing", bill)
val deleteFromWaitlist = deleteFrom("waitlist")(WaitList.email.partitionKey === student.email)

val studentEnrollmentTransaction = (putStudent zip billedStudent zip deleteFromWaitlist).transaction

for {
  _ <- studentEnrollmentTransaction.execute
} yield ()
```

### [ReadTransactions](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactGetItems.html)

```scala
val getStudent = get("student")(Student.id.partitionKey === "1")
val getBill = get("billing")(Bill.id.partitionKey === "1")

val getStudentAndBillTransaction = (getStudent zip getBill).transaction

for {
  studentAndBill <- getStudentAndBillTransaction.execute
} yield studentAndBill
```

## Transaction Failures

DynamoDBQueries using the `.transaction` method will fail at runtime if there are invalid transaction actions such as creating a table, scanning for items, or querying. Note a limited number of actions that can be performed for either a read or a write transaction. There is a `.safeTransaction` method that is also available that will return `Either[DynamoDBError.TransactionError, DynamoDBQuery[A]]`.

There are more examples in our integration tests [here](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiCrudSpec.scala) and [here](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/LiveSpec.scala).

### Valid Transact Write Items

* PutItem
* DeleteItem
* BatchWriteItem
* UpdateItem
* ConditionCheck

### Valid Transact Get Item

* GetItem
* BatchGetItem

---

## Introduction to ZIO DynamoDB

Simple, type-safe, and efficient access to DynamoDB

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-dynamodb/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-dynamodb_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-dynamodb_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-dynamodb_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-dynamodb_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-dynamodb-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-dynamodb-docs_2.13) [![ZIO DynamoDB](https://img.shields.io/github/stars/zio/zio-dynamodb?style=social)](https://github.com/zio/zio-dynamodb)

## Introduction

ZIO DynamoDB is a library that is used for type-safe, efficient, and boilerplate free access to AWS's DynamoDB service. It provides a type-safe API for many query types and the number of type-safe APIs is expanding. ZIO DynamoDB will automatically batch queries and execute un-batchable queries in parallel.

Under the hood we use the excellent [ZIO AWS](https://zio.dev/zio-aws) library for type-safe DynamoDB access, and the awesome [ZIO Schema](https://zio.dev/zio-schema) library for schema derived codecs (see here for documentation on how to [customise these through annotations](guides/codec-customization.md)).

For an overview of the High Level API please see the [ZIO DynamoDB cheat sheet](guides/cheat-sheet.md).

## Installation

To use ZIO DynamoDB, we need to add the following lines to our `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb" % "1.0.0-RC24"
)
```

### Cats Effect Interop

To use the Cats Effect 3 interop module, we need to also add the following line to our `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb-ce" % "1.0.0-RC24"
)
```

For CE interop examples please see [examples sbt module](https://github.com/zio/zio-dynamodb/blob/series/2.x/examples/src/main/scala/zio/dynamodb/examples/dynamodblocal/interop/CeInteropExample.scala).

### Future Interop

For projects that do not use effect libraries (e.g. Play applications) we can use the Future interop module.
We need to add the following line to our `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb-future" % "1.0.0-RC24"
)
```

For Future interop examples please see [examples sbt module](https://github.com/zio/zio-dynamodb/blob/series/2.x/examples/src/main/scala/zio/dynamodb/examples/dynamodblocal/interop/FutureInteropExample.scala).

### Read/write DynamoDB JSON
AWS tools like the CLI and Console read/write a special JSON representation of dynamoDB items. The new experimental optional `zio-dynamodb-json` module provides a way to read/write this form of JSON when working with both the High Level and Low Level API. To use this module, we need to also add the following line to our `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb-json" % "1.0.0-RC24"
)
```

## Example

For examples please see [examples sbt module](https://github.com/zio/zio-dynamodb/tree/series/2.x/examples/src/main/scala/zio/dynamodb/examples). Below is `Main.scala` from that module:

```scala

object Main extends ZIOAppDefault {

  final case class Person(id: Int, firstName: String)
  object Person {
    implicit lazy val schema: Schema.CaseClass2[Int, String, Person] = DeriveSchema.gen[Person]

    val (id, firstName) = ProjectionExpression.accessors[Person]
  }
  val examplePerson = Person(1, "avi")

  private val program = for {
    _      <- put("personTable", examplePerson).execute
    person <- get("personTable")(Person.id.partitionKey === 1).execute
    _      <- zio.Console.printLine(s"hello $person")
  } yield ()

  override def run =
    program.provide(
      netty.NettyHttpClient.default,
      config.AwsConfig.default, // uses real AWS dynamodb
      dynamodb.DynamoDb.live,
      DynamoDBExecutor.live
    )
}
```

For examples on how to use the DynamoDBLocal in memory database please see the [integration tests](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiCrudSpec.scala)
and [DynamoDBLocalMain](https://github.com/zio/zio-dynamodb/blob/series/2.x/examples/src/main/scala/zio/dynamodb/examples/dynamodblocal/DynamoDBLocalMain.scala) .
Note before you run these you must first run the DynamoDBLocal docker container using the provided docker-compose file:

```
docker compose -f docker/docker-compose.yml up -d
```

Don't forget to shut down the container after you have finished

```
docker compose -f docker/docker-compose.yml down
```

## Resources
- [Introducing ZIO DynamoDB by Avinder Bahra & Adam Johnson](https://www.youtube.com/watch?v=f68-69eA8Vc&t=33s) - DynamoDB powers many cloud-scale applications, with its robust horizontal scalability and uptime. Yet, interacting with the Java SDK is error-prone and tedious. In this presentation, Avinder Bahra presents ZIO DynamoDB, a new library by Avi and Adam Johnson designed to make interacting with DynamoDB easy, type-safe, testable, and productive.
- [Introducing The ZIO DynamoDB Type-Safe API by Avinder Bahra](https://www.youtube.com/watch?v=Qte4WUfHQ3g&t=10s) - Adam Johnson and Avinder released ZIO DynamoDB, a new Scala library that significantly reduces boilerplate when compared to working directly with AWS client libraries. However, there was still work to be done to improve type safety. In this talk, Avinder introduces a new type-safe API that can prevent many errors at compile time while remaining user-friendly.

---

## Auto batching and parallelisation

When `DynamoDBQuery`'s are composed manually via the `zip` combinator function they become eligible for auto-batching and parallelisation in the `execute` method. 

When they are composed automatically using the `batch` function they are eligible for auto-batching but **no parallelisation** occurs.

```scala
val batchedWrite1 = DynamoDBQuery.put("person", Person("1", "John", 21))
        .zip(DynamoDBQuery.put("person", Person("2", "Jane", 22)))

val batchedWrite2 = DynamoDBQuery.batch(people)(person => put("person", person))

for {
  _ <- batchedWrite1.execute // PutItem operations will be batched
  _ <- batchedWrite2.execute // PutItem operations will be batched
} yield ()
```

## Rules for determining auto-batching vs parallelisation behaviour

The rules for determining whether a query is auto-batched are determined by what query types are eligible for batching in the AWS API. The AWS [BatchWriteItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) operation can only deal with `PutItem` and `DeleteItem` operations. Furthermore, for both of these operations - condition expressions are not allowed. The AWS [BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html) operation is used for batching `GetItems`'s .

So the rules are as follows: 

- if there are multiple queries `zip`'ed together they are grouped by their type  `GetItem` or `Writes` (`PutItem`/`DeleteItem`) and batched using the `AWS` `BatchGetItem`/`BatchWriteItem` APIs - but _only if_ they pass the below rules:

  - The query is a `PutItem` or `DeleteItem` operation (`put` and `deleteFrom` in the High Level API)
    - The query does not have a condition expression
    - The query has `ReturnValues.None` specified (which is the default) - any other return value will invalidate batched execution.
  - The query is a `GetItem` operation (`get` in the High Level API)
    - The query's `projections` list contains the primary key - this is required to match the response data to the request. Note all fields are included by default so this is only a concern if you explicitly specify the projection expression.  
- for manually `zip`'ed queries, if a query does not qualify for auto-batching it will be parallelised automatically

# Batching using the `batch` function

The `batch(someCollection)(el => someQuery)` function is the preferred way of composing batched queries. Queries are automatically zipped together, however 
on `execute` the parallelisation step above is omitted and instead a `DynamoDBError.BatchError.UnbatchableQueryError` is returned with a detailed error message for each rule violation.         

The use of `batch` the recommended approach for batching queries rather than using `zip`, however you will still have to manually manage the size of the batch manually (see next section).  

## Maximum batch sizes for `BatchWriteItem` and `BatchGetItem`

When using the `zip` or `batch` operations one thing to bear in mind is the maximum number of queries that the `BatchWriteItem` and `BatchGetItem` operations can handle:

- `BatchWriteItem` can handle up to **25** `PutItem` or `DeleteItem` operations
- `BatchGetItem` can handle up to **100** `GetItem` operations

If these are exceeded then you will get a runtime AWS error. For further information please refer to the AWS documentation linked above.

If you want to avoid managing the batch size manually please see the `batchWriteFromStream` and `batchReadFromStream` functions in the section below.

## Automatic retry of unprocessed batch items/keys

Note that both the AWS `BatchWriteItem` and `BatchGetItem` operations return a list of unprocessed items/keys. If this list is non-empty then the operation are retried automatically by the ZIO DynamoDB library. 

If retries do not succeed in eliminating the unprocessed items/keys then the whole batch is failed with a `BatchError.WriteError`/`BatchError.GetError` - both of which will contain a list of the unprocessed items/keys. 

The default retry policy is:

```scala
Schedule.recurs(3) && Schedule.exponential(50.milliseconds)
```

This can be overridden by using the `withRetryPolicy` combinator:

```scala
batchedWrite2.withRetryPolicy(myCustomRetryPolicy).execute
```

## Integrating Batching with ZIO Streams

For examples of how to integrate batching with ZIO Stream please see the utility functions 
[`batchWriteFromStream`](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/main/scala/zio/dynamodb/package.scala#L22-L55) 
and [`batchReadFromStream`](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/main/scala/zio/dynamodb/package.scala#L97-L138) in the `zio.dynamodb` package.
These functions take care of details mentioned above such as managing the maximum batch sizes and can also be used as examples for writing your own custom batched streaming operations.
```scalaa

---

## DynamoDBQuery

When we use the Low or High Level API constructors to create a query we end up with the type `DynamoDBQuery` which is a sealed trait.

One of the primary methods on this trait is `execute` which will run the query and return the result as a ZIO effect. 

```scala
def execute: ZIO[DynamoDBExecutor, DynamoDBError, Out] = ???
```

The `execute` method requires a `DynamoDBExecutor` service in order to execute the query using the lower level ZIO AWS DynamoDB library.

The `execute` method does the following:

- if the query type is a composite (`Zip`) then it will automatically batch or parallelise the queries - see 
[Auto batching and parallelisation](auto-batching-and-parallelisation) for the exact rules
- executes the query using the `DynamoDBExecutor` service, which:
  - converts it to an underlying ZIO AWS DynamoDB query
  - converts the ZIO AWS DynamoDB response back to an `Item` (type alias for an `AttrMap`)

When using the High Level API transformations are done between the Scala model and the `Item` type using the automatically
generated codecs that make use of the `ZIO Schema` in implicit scope.

The next sections cover the surface area exposed by `DynamoDBQuery`.

## `DynamoDBQuery` Combinators and Operations

DynamoDBQuery Combinators | Alias | Description
---|-------|---
map |       | map the result of a query with a function
zip | `<*>`   | combine 2 queries together and returns a tuple - makes the resulting query eligible for automatic batching or parallelisation [see Autobatching and Parallelisation](auto-batching-and-parallelisation) for more details
zipWith |       |does a `zip` and then immediately maps the result with a function
zipLeft | `<*`    | a zip that ignores the result of the right query
zipRight| `*>`    | a zip that ignores the result of the left query

DynamoDBQuery Functions | Description
---|---
`def batch[In, A, B](values: Iterable[A])(body: A => DynamoDBQuery[In, B]): DynamoDBQuery[In, List[B]]`  | `DynamoDB.batch` automates the zipping of queries of the same type using a collection as input. [see Autobatching and Parallelisation](auto-batching-and-parallelisation) for more details.  Note that unprocessed items/keys are retried automatically and if they still fail a `BatchError.WriteError`/`BatchError.GetError` is returned both of which will contain a list of the unprocessed items/keys - see `withRetryPolicy` in the below section for overriding the default retry policy.

DynamoDBQuery Operations | Description
---|---
capacity | sets the ReturnConsumedCapacity. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html#DDB-GetItem-request-ReturnConsumedCapacity). Note capacity data in the response is ignored by the High Level Api
consistency | sets the `ConsistencyMode` for read operations. Valid values are `Strong`and `Weak`(default) [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html#DDB-GetItem-request-ConsistentRead)  
filter | sets the `FilterExpression` - applies to `ScanSome`, `ScanAll`, `QuerySome`, `QueryAll`. Note the filter is applies **after** the read by DDB so no read units are saved, however latency costs are reduced.
gsi | creates a Global Secondary Index - applies to a `CreateTable` query. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html#DDB-CreateTable-request-GlobalSecondaryIndexes)
indexName | sets the local secondary index or global secondary index name - applies to `ScanSome`, `ScanAll`, `QuerySome`, `QueryAll`. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html#DDB-Scan-request-IndexName) 
lsi | creates a local Secondary Index - applies to a `CreateTable` query. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html#DDB-CreateTable-request-LocalSecondaryIndexes) 
metrics | set `ReturnItemCollectionMetrics`, valid values are `None` (default) and `Size` - applies to PutItem, UpdateItem, Delete, Transaction. Note that metric data in the response is ignored by the High Level API. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html#DDB-PutItem-request-ReturnItemCollectionMetrics)
parallel(N) | Applies only to `Scan` - sements and runs the query in parallel in DDB and merges the items in the response. N is level of parallelism. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan). 
returns | sets the `ReturnValues` - applies to `UpdateItem`, `DeleteItem`, `PutItem` (see [Crud Operations](reference/hi-level-api/crud-operations/index.md) reference section for each operation for more details). [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html#DDB-UpdateItem-request-ReturnValues)
selectAllAttributes, selectAllProjectedAttributes, selectSpecificAttributes, selectCount | Determines the attributes returned by Scan and Query [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-Select)    
sortOrder | sets the sort order for `Query`'s [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-ScanIndexForward) 
startKey |Applies to `Query` and `Scan` and specifies the start key for the query. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-ExclusiveStartKey)
transaction | executes the query in a transaction - see [Transactions Guide](../guides/transactions) for more details.
where | sets the `ConditionExpression` - applies to `PutItem`, `DeleteItem`, `UpdateItem` and `Scan` [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html)
whereKey | set the `KeyConditionExpr` applies to `QuerySome` and `QueryAll`. [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-KeyConditionExpression) 
withClientRequestToken | set the client request token` - applies to write transactions [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html#DDB-TransactWriteItems-request-ClientRequestToken)
withRetryPolicy | override the default retry policy for a batched query - [see Autobatching and Parallelisation](auto-batching-and-parallelisation) for more details

---

## Error Handling(Reference)

DynamoDBError is a sealed trait that represents all the possible errors that can occur when interacting with DynamoDB
and is returned by the DynamoDBQuery `execute` method.

```scala
def execute: ZIO[DynamoDBExecutor, DynamoDBError, Out] = ???
```

The error hierarchy is as follows:

- **`DynamoDBError`** top level sealed trait
  - **`DynamoDBError.ItemError`** sealed trait for item level errors
    - **`DynamoDBError.ItemError.ValueNotFound`** - returned by `get` in the High Level API. If you expect items to be missing you can use the `maybeFound` extension method to return `None` in this scenario. see [Working With `get` Return Values](hi-level-api/crud-operations/get#working-with-get-return-values) for more details.  
    - **`DynamoDBError.ItemError.DecodingError`** - returned by automatic codecs in the High Level API when data does not match the expected type described by the schema
  - **`DynamoDBError.AWSError`** - case class that contains the underlying AWS `Exception`. Typically, you need to pattern match on this when you expect a specific AWS exception eg AWS `ConditionalCheckFailedException` when you do a strict insert operation by having a condition expression that asserts the primary key does not exist.
  - **`DynamoDBError.BatchError`** sealed trait for batch related errors. You need to consider this error if queries result in batching eg if you are using `DynamoDBQuery.batch` or manually `Zip`'ing together `DynamoDBQuery`'. Note at the point that this error is raised automatic retries have already occurred. For a long running process typical handler actions would be to record the errors and to carry on processing. See [Auto batching and parallelisation](auto-batching-and-parallelisation) section for more details.
    - **`DynamoDBError.BatchError.GetError`** - case class returned by automatic batching - `unprocessedKeys` contains a Map of table name to primary key   
    - **`DynamoDBError.BatchError.WriteError`** - returned by automatic batching - `unprocessedItems` contains a map of table name to item/primary key 
    - **`DynamoDBError.BatchError.UnbatchableQueryError`** - returned on execute of a query created by the `batch` function - see [Auto batching and parallelisation](auto-batching-and-parallelisation) section for more details. The error message will contain a detailed explanation.
    
  - **`DynamoDBError.TransactionError`** sealed trait for transaction related errors. You need to handle this error if you are using the transaction API ie `<dynamoDBQuery>.transaction` or `<dynamoDBQuery>.safeTransaction`
    - **`DynamoDBError.TransactionError.EmptyTransaction`**
    - **`DynamoDBError.TransactionError.MixedTransactionTypes`**
    - **`DynamoDBError.TransactionError.InvalidTransactionActions`**
  - **`DynamoDBError.UnknownNonFatalError`** Encapsulates a non-fatal error (as defined by scala.utils.control.NonFatal) that occurred during processing

---

## Field Traversal

We will be using the below model for the examples which contains a complex collection field `hobbies` and a nested case 
class field `address`. Note that ZIO DynamoDB also supports case classes as collection elements (not shown in this example).

```scala
final case class Address(number: String, street: String)
object Address {
  implicit lazy val schema: Schema.CaseClass2[String, String, Address] = DeriveSchema.gen[Address]

  // (1) number and street are ProjectionExpressions
  val (number, street) = ProjectionExpression.accessors[Address]
}
final case class Person(email: String, hobbies: Map[String, List[String]], address: Address)
object Person {
  implicit lazy val schema: Schema.CaseClass3[String, Map[String, List[String]], Address, Person] = DeriveSchema.gen[Person]

  // (2) email, hobbies and address are ProjectionExpressions
  val (email, hobbies, address) = ProjectionExpression.accessors[Person]
}
```

## `ProjectionExpression` field traversal combinators

DynamoDB allows us to dig into nested structures when updating or querying data using path expressions. Using the High Level API we do this in a _type safe_ way by
using the `ProjectionExpression`'s generated by the `accessors` method (see comments `1` and `2` above) as springboard 
for a bunch of traversal combinators. Note the `accessors` method in turn uses the reified optics feature of ZIO Schema.

Traversal Combinator | Description
---------------------|------------
`>>>`                | Returns a `ProjectionExpression` that traverses into a **product** (case class) or a **sum** type (sealed trait concrete instance)
`valueAt(key)`       | Returns a `ProjectionExpression` that traverses into a map field using a string key
`elementAt(N)`       | Returns a `ProjectionExpression` that traverses into a list field using a zero based index

```scala
  val person                                                             =
    Person("john@gmail.com", Map("sports" -> List("cricket", "football")), Address("1", "Main St"))

  // ProjectionExpressions extracted deliberately to illustrate the types
  val addressToStreetPE: ProjectionExpression[Person, String]            =
    (Person.address >>> Address.street) 
  val valueAtHobbiesPE: ProjectionExpression[Person, List[String]]       =
    Person.hobbies.valueAt("sports")
  val valueAtAndElementAtHobbiesPE: ProjectionExpression[Person, String] =
    Person.hobbies.valueAt("sports").elementAt(0)

  for {
    _ <- DynamoDBQuery.put("people", person).execute

    _ <- DynamoDBQuery
           .update("people")(Person.email.partitionKey === "john@gmail.com")(
             addressToStreetPE.set("2nd St")
           )
           .execute

    _ <- DynamoDBQuery
           .update("people")(Person.email.partitionKey === "john@gmail.com")(
             valueAtHobbiesPE.set(List("tennis", "rugby"))
           )
           .execute

    _ <- DynamoDBQuery
           .update("people")(Person.email.partitionKey === "john@gmail.com")(
             valueAtAndElementAtHobbiesPE.set("cricket")
           )
           .execute

  } yield ()
```

## Product traversal

From the the example above: 
```scala
  val addressToStreetPE: ProjectionExpression[Person, String]            =
    (Person.address >>> Address.street) 
```
`addressToStreetPE` is an example of traversing into a product (case class).

## Sum type traversal

To illustrate sum type traversal, consider the below model with sum type (sealed trait) `BilledBody`:

```scala
  @discriminatorName("billedType")
  sealed trait BilledBody

  object BilledBody {
    final case class BilledMonthly(month: Int) extends BilledBody
    object BilledMonthly {
      implicit lazy val schema: Schema.CaseClass1[Int, BilledMonthly] = DeriveSchema.gen[BilledMonthly]
      val month: ProjectionExpression[BilledMonthly, Int]             = ProjectionExpression.accessors[BilledMonthly]
    }
    final case class BilledYearly(year: Int) extends BilledBody
    object BilledYearly  {
      implicit lazy val schema: Schema.CaseClass1[Int, BilledYearly] = DeriveSchema.gen[BilledYearly]
      val year: ProjectionExpression[BilledYearly, Int]              = ProjectionExpression.accessors[BilledYearly]
    }
    implicit val schema: Schema.Enum2[BilledMonthly, BilledYearly, BilledBody] = DeriveSchema.gen[BilledBody]
    val (monthly, yearly) = ProjectionExpression.accessors[BilledBody]
  }

  final case class Invoice(id: Int, body: BilledBody)
  object Invoice {
    implicit lazy val schema: Schema.CaseClass2[Int, BilledBody, Invoice] = DeriveSchema.gen[Invoice]
    val (id, body)                                                       = ProjectionExpression.accessors[Invoice]
  }
```

We can access the `BilledYearly` case class field `year` using the `>>>` combinator as shown below:

```scala
  val invoiceToBilledYearlyToYearPE: ProjectionExpression[Invoice, Int] =
    Invoice.body >>> BilledBody.yearly >>> BilledYearly.year

  val yearlyInvoice = Invoice(1, BilledYearly(2021))

  for {
    _ <- DynamoDBQuery.put("invoices", yearlyInvoice).execute
    _ <- DynamoDBQuery
           .update("invoices")(Invoice.int.partitionKey === 1)(
             invoiceToBilledYearlyToYearPE.set(2022)
           )
           .execute

  } yield ()
```

---

## Creating Models

The High Level API provides automatic serialization and deserialization of Scala case classes to and from DynamoDB types.
This is done by requiring that an implicit ZIO Schema instance is in scope for the case class. This schema instance is 
generated semi-automatically by using the ZIO Schema `DeriveSchema.gen[A]` - placing this in the companion object of 
the case class ensures that this implicit is automatically in scope.

```scala
final case class Person(email: String, hobbies: Map[String, List[String]], registrationDate: Instant)
object Person {
  implicit val schema: Schema.CaseClass3[String, Map[String, List[String]], Instant, Person] =
    DeriveSchema.gen[Person]
}
```

This semi-automatically derived schema is used to automatically generate codecs for the case class (in the `Codecs` object)
to perform the serialization and deserialization to and from DynamoDB types.

All standard Scala types are supported by the codecs, as well as nested case classes and collections. Note that where 
possible Scala types are mapped to corresponding DynamoDB types, for example Scala `Map`'s and `Set`'s are mapped to 
native DynamoDB types.

Scala Scalar Types | Native DynamoDB Type | Notes
------------|----------------------|--------------
`Unit`              | `NULL`                    
`String`            | `S`                    
Numeric Types       | `N`                    
`Collection[byte]`  | `B`                    | Any Scala collection type of byte is serialized to a DynamoDB binary type 
`Boolean`           | `BOOL`                 
`java.time.*`       | `S`                    | There is no native date/time support. Instant is serialized to a string in ISO-8601 format
`java.util.UUID`    | `S`                    | There is no native `UUID` support
`java.util.Currency`| `S`                   | There is no native `Currency` support

Note in the below table that types `A`, `K` and `V` can be collections or case classes as well as scalar types.

Scala Collection Types | Native DynamoDB Type | Notes
------------|----------------------|--------------
`Option[A]`           |                      | `Some`/`None` are represented by the presence and the absence of the field in the DynamoDB item respectively.
`List[A]`             | `L`                    |
`Set[String]`         | `SS`                   |  Note that native Set types (`SS`, `NS`, `BS`) do not allow empty values - they are represented by the absence of the field in the DynamoDB item. However ZIO DynamoDB will automatically convert these missing values to empty Scala Sets.   
`Set` of numeric type | `NS`                   |
`Set` of binary type  | `BS`                   |
`Set[A]` of other type| `L`                    | If type is not a string or a numeric then a list is used
`Map[String, A]`      | `M`                    | if key type is a string then a native Map is used
`Map[K, V]`           | `L`                    | otherwise a list of tuple of key value pair is used

**Scala Either Type encoding**

The default encoding for the Scala `Either` type is an AttributeValue Map with a single key of `Left` or `Right` and the value being the encoded value eg for `Either[String, Int]`, a Left value would be:
```json
  "someField" : {
    "Left" : {
      "S" : "Error"
    }
  }, 
```
If you wanted a flatter "union" type encoding this can be achieved by specifying a transformation from a `Schema.Fallback` in implicit scope eg adding the following to the companion object of the case class:
```scala
implicit def fallbackEither[A, B](implicit schemaA: Schema[A], schemaB: Schema[B]): Schema[Either[A, B]] =
      Schema.Fallback(schemaA, schemaB).transform(_.toEither, Fallback.fromEither)
```      
Please see the integration suite [TypeSafeApiCrudSpec.scala](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiCrudSpec.scala) for an example.

**Visualising DynamoDB Encodings**

Note during model development you can use the optional `zio-dynamodb-json` module to view the DynamoDB types generated by the codecs.
eg for the above example:

```scala

val person = Person("email", Map("sports" -> List("cricket", "football")), Instant.now)
println(person.toJsonStringPretty[Person])
```
...would print the following which is representation of the native DynamoDB types and data in a standard JSON format used in AWS console views:
```json
{
  "registrationDate" : {
    "S" : "2024-12-05T05:47:46.300286Z"
  },
  "hobbies" : {
    "sports" : {
      "L" : [
        {
          "S" : "cricket"
        },
        {
          "S" : "football"
        }
      ]
    }
  },
  "email" : {
    "S" : "email"
  }
}
```

For more information about using the `zio-dynamodb-json` module please see the [ZIO DynamoDB Json](reference/zio-dynamodb-json.md) reference page.

For more information about customising data mappings please see the [Codec Customization](guides/codec-customization.md) guide.

---

## deleteFrom

```scala
  def deleteFrom[From: Schema](
    tableName: String
  )(
    primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From]
  ): DynamoDBQuery[Any, Option[From]] = ???
```  

The `deleteFrom` operation is used to remove an item from a table. The `primaryKeyExpr` param can be created using the `ProjectionExpression`'s in the companion object for model class: 

```scala
for {
  _ <- DynamoDBQuery.deleteFrom("person")(Person.id.primaryKey === "1").execute
} yield ()
```

### `deleteFrom` query operations

```scala
<DELETE_QUERY>
  .returns(<ReturnValues>) // ReturnValues.AllOld | ReturnValues.None <default>
  .where(<ConditionExpression>)
```

---

## get

```scala
  def get[From: Schema](tableName: String)(
    primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From]
  ): DynamoDBQuery[From, Either[ItemError, From]] = ???
```

The `get` operation is used to retrieve an item from a table. The `KeyConditionExpr.PrimaryKeyExpr` can be created using the `ProjectionExpression`'s in the companion object for model class. It returns an `Either[ItemError, From]` where `ItemError` is a sealed trait that that has `ValueNotFound` and `DecodingError` instances. 

```scala
for {
  errorOrPerson <- DynamoDBQuery.get("person")(Person.id.primaryKey === "1").execute
} yield errorOrPerson
```

### Working with `get` return values

Sometimes working with a `Either[ItemError, From]` can be a little unwieldy so there are two approaches we can take.

The first approach is to use the ZIO `absolve` method to push all ItemErrors into the ZIO error channel

```scala

for {
  person <- DynamoDBQuery.get("person")(Person.id.primaryKey === "1").execute.absolve
} yield person
```

However sometimes we wish to treat `NotFound` as a success case and for this the `maybeFound` extension method can be imported to push the `DecodingError` into the ZIO error channel and handle `NotFound` as a successful operation by using an `Option` type. 

```scala

for {
  maybePerson <- DynamoDBQuery.get("person")(Person.id.primaryKey === "1").execute.maybeFound
} yield maybePerson
```

### `get` query operations

```scala
<GET_QUERY>
  .consistency(<ConsistencyMode>)
  .where(<ConditionExpression>)
```

---

## CRUD Operations

The High Level API mirrors the CRUD operations of DDB but with a high level of type safety. 

We have to start of with a Scala model of the table and add an implicit schema reference in the companion object together with some convenience `ProjectionExpression`'s via the `accessors` function. 

```scala
final case class Person(id: String, name: String, age: Int)
object Person {
  implicit val schema: Schema.CaseClass3[String, String, Int, Person] = DeriveSchema.gen[Person]
  val (id, name, age) = ProjectionExpression.accessors[Person]
}
```

A summary is shown below and detailed sections can be found for each operation.

```scala
object DynamoDBQuery {

  // CRUD operations  

  def put[A: Schema](tableName: String, a: A): DynamoDBQuery[A, Option[A]] = ???

  def get[From: Schema](tableName: String)(
    primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From]
  ): DynamoDBQuery[From, Either[ItemError, From]] = ???

  def update[From: Schema](tableName: String)(primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From])(
      action: Action[From]
    ): DynamoDBQuery[From, Option[From]]  = ???

  def deleteFrom[From: Schema](
    tableName: String
  )(
    primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From]
  ): DynamoDBQuery[Any, Option[From]] = ???

}
```

---

## put

```scala
def put[A: Schema](tableName: String, a: A): DynamoDBQuery[A, Option[A]] = ???
```

The `put` operation is used to insert or replace an item in a table.

```scala
for {
  _ <- DynamoDBQuery.put("person", Person("1", "John", 21))
        .where(Person.id.notExists) // a ConditionExpression
        .execute
} yield ()
```

### `put` query operations

```scala
<PUT_QUERY>
  .returns(<ReturnValues>) // ReturnValues.AllOld | ReturnValues.None <default>
  .where(<ConditionExpression>) // eg Person.id.notExists
```

### Using `put` with Top level traits using `discriminatorName` annotation
When using a top level sealed trait with `@discriminatorName` annotation, it must be provided explicitly to the `put`
to ensure that discriminator field is encoded.

```scala
@discriminatorName("invoiceType")
sealed trait Invoice
final case class PreBilledInvoice(/* ... */) extends Invoice
final case class BilledInvoice(/* ... */) extends Invoice

for {
  _ <- DynamoDBQuery.put[Invoice]("invoice", BilledInvoice(/* ... */)).execute // OK - discriminator encoded
  _ <- DynamoDBQuery.put[BilledInvoice]("invoice", BilledInvoice(/* ... */)).execute // WRONG - discriminator not encoded
} yield ()
```

---

## update

```scala
def update[From: Schema](tableName: String)(primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From])(
    action: Action[From]
): DynamoDBQuery[From, Option[From]]  = ???
```

The `update` operation is used to modify an existing item in a table. Both `KeyConditionExpr.PrimaryKeyExpr` and the `Action` params can be created using the `ProjectionExpression`'s in the companion object for model class: 
    
```scala
for {
  _ <- DynamoDBQuery.update("person")(Person.id.primaryKey === "1")(
    Person.name.set(42) + Person.age.set(42)
  ).execute
} yield maybePerson
```

## `update` expressions

action | description
---|---
`+` | combines update actions eg `Person.name.set(42) + Person.age.set(42)`  
`set` | Set an attribute `Person.name.set("John")`
`setIfNotExists` | Set attribute if it does not exists `Person.name.setIfNotExists("John")`
`appendList` | Add supplied list to the end of this list attribute
`prepend` | Prepend an element to a list attribute
`prependList` | Prepend a list to list attribute
`deleteFromSet` | delete all elements that match the supplied set
`add(a: To)` | adds this value as a number attribute if it does not exists, else adds the numeric value to the existing attribute
`addSet` | Adds this set as an attribute if it does not exists, else if it exists it adds the elements of the set
`remove(index: Int)` | remove an element at the specified index
`remove` | Removes this path expression from an item

See [AWS API Reference](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html) to learn more about update expressions.

### `update` query operations

```scala
<UPDATE_QUERY>
  .returns(<ReturnValues>) // ReturnValues.AllNew | ReturnValues.AllOld | ReturnValues.None <default>
  .where(<ConditionExpression>)
```

---

## High Level API(Hi-level-api)

High Level API constructors for CRUD operations are found on the `DynamoDBQuery` companion object. They rely on a 
ZIO Schema for a particular type being in implicit scope. This is achieved using the `DeriveSchema.gen` macro. 
Internally codecs are automatically generated for the case classes based on the meta-data provided by the `Schema`'s.

```scala
object DynamoDBQuery {

  // CRUD operation constructors  

  def put[A: Schema](tableName: String, a: A): DynamoDBQuery[A, Option[A]] = ???

  def get[From: Schema](tableName: String)(
    primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From]
  ): DynamoDBQuery[From, Either[ItemError, From]] = ???

  def update[From: Schema](tableName: String)(primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From])(
      action: Action[From]
    ): DynamoDBQuery[From, Option[From]]  = ???

  def deleteFrom[From: Schema](
    tableName: String
  )(
    primaryKeyExpr: KeyConditionExpr.PrimaryKeyExpr[From]
  ): DynamoDBQuery[Any, Option[From]] = ???

  // Scan/Query operations

  def scanAll[A: Schema](
    tableName: String
  ): DynamoDBQuery[A, Stream[Throwable, A]] = ???

  def scanSome[A: Schema](
    tableName: String,
    limit: Int
  ): DynamoDBQuery[A, (Chunk[A], LastEvaluatedKey)] = ???  

  def queryAll[A: Schema](
    tableName: String
  ): DynamoDBQuery[A, Stream[Throwable, A]] = ???

  def querySome[A: Schema](
    tableName: String,
    limit: Int
  ): DynamoDBQuery[A, (Chunk[A], LastEvaluatedKey)] = ???  
}
```

Once we have constructed  a type safe query we can execute it.

Methods that need a primary key expression take a `KeyConditionExpr.PrimaryKeyExpr[From]` as an argument, however 
rather than create one directly we can create one using a `ProjectionExpression` as a springboard

---

## Primary Keys

In the native AWS DynamoDB API primary keys are represented in two different ways depending on context:

 
| AWS|Example|Context
| ---|---|---
| Primary Keys | `{"id": "1", "year": 2023}` | `GetItem`, `PutItem`, `DeleteItem` [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html#DDB-GetItem-request-Key)
| Key Condition Expressions | `#id=:val1 and #year > :val2` | `Query` [AWS API](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.KeyConditionExpressions.html)

Here we see that for CRUD operations the primary key is represented as a JSON object where each value is checked for _equality_, while for Query operations the primary key is represented as a key condition _expression_ .

The High Level API unifies the two different ways to represent a primary key into a single _type safe_ API.

## Unified Type Safe High Level API for Primary Key Expressions
Assuming the below model
```scala
final case class Person(id: String, year: Int, address: String)
object Person {
  implicit val schema: Schema.CaseClass3[String, Int, String, Person] = DeriveSchema.gen[Person]
  val (id, year, address) = ProjectExpression.accessors[Person]
}
final case class Employee(id: String, group: String, address: String)
object Person {
  implicit val schema: Schema.CaseClass3[String, Int, String, Person] = DeriveSchema.gen[Person]
  val (id, year, address) = ProjectExpression.accessors[Person]
}
```

The High Level API unifies the two different ways into a single Type Safe API that is accessed by using the `ProjectExpression` returned by the `ProjectExpression.accessors` function as a springboard via the `partitionKey` and `sortKey` methods.

| AWS| Example                                                                          |Context
| ---|----------------------------------------------------------------------------------|---
| Primary Keys | `Person.id.partitionKey === "1" && Person.year.sortKey === "2020`"               | `GetItem`, `PutItem`, `DeleteItem` 
| Key Condition Expressions | [Query]`.whereKey(Person.id.partitionKey === "1" && Person.year.sortKey > 2020)` | [Query]`.whereKey`

Valid operations on primary keys are:

context | operation | applies to
---|---|---
`Person.id.partitionKey``Person.year.sortKey` |  `===` | `GetItem`, `PutItem`, `DeleteItem`, [Query]`.whereKey` 
`Person.id.partitionKey === "1"` |  `&&` | `GetItem`, `PutItem`, `DeleteItem`, [Query]`.whereKey`.  Provides conjunction from a partition key to a sort key
`Person.year.sortKey` |  `>` | [Query]`.whereKey` only
 |  `>=` | 
 |  `<` | 
 |  `<=` | 
 |  `<>` | 
 |  `.between(2021, 2023)` |
`Employee.group.sortKey` | `.beginsWith("Group1")` | Applies to String sort keys only

---

## Scan and Query Operations

The High Level API mirrors the Scan and Query operations of DDB but with a high level of type safety. 

We have to start of with a Scala model of the table and add an implicit schema reference in the companion object together with some convenience `ProjectionExpression`'s via the `accessors` function. 

```scala
// year is the sort key
final case class Equipment(id: String, year: Int, name: String, price: Double)
object Equipment {
  implicit val schema: Schema.CaseClass4[String, Int, String, Double, Equipment] = DeriveSchema.gen[Equipment]
  val (id, year, name, price)                                                    = ProjectionExpression.accessors[Equipment]
}
```

A summary is shown below and detailed sections can be found for each operation.

```scala
object DynamoDBQuery { 
  // Scan/Query operations

  def scanAll[A: Schema](
    tableName: String
  ): DynamoDBQuery[A, Stream[Throwable, A]] = ???

  def scanSome[A: Schema](
    tableName: String,
    limit: Int
  ): DynamoDBQuery[A, (Chunk[A], LastEvaluatedKey)] = ???  

  def queryAll[A: Schema](
    tableName: String
  ): DynamoDBQuery[A, Stream[Throwable, A]] = ???

  def querySome[A: Schema](
    tableName: String,
    limit: Int
  ): DynamoDBQuery[A, (Chunk[A], LastEvaluatedKey)] = ???

}
```

---

## queryAll

```scala
  def queryAll[A: Schema](
    tableName: String
  ): DynamoDBQuery[A, Stream[Throwable, A]] = ???
```

The `queryAll` operation must be used in conjunction with the `whereKey` combinator which allows you to specify the partition key and optionally sort key expression the query.

```scala
for {
  _          <- put(tableName, Equipment("1", 2020, "Widget1", 1.0)).execute
  _          <- put(tableName, Equipment("1", 2021, "Widget1", 2.0)).execute
  stream     <- queryAll[Equipment](tableName)
                  .whereKey(Equipment.id.partitionKey === "1" && Equipment.year.sortKey > 2020)
                  .execute
} yield ()
```

## Combinators

```scala
<SCAN_ALL_QUERY>
  .consistency(<ConsistencyMode>)
  .whereKey(<KeyConditionExpr>)  // eg Equipment.id.partitionKey === "1" && Equipment.year.sortKey > 2020
  .filter(<ConditionExpression>) // eg Equipment.price > 1.0 - filtering is done server side AFTER the scan
  .sortOrder(ascending)          // ascending is a boolean flag 
  .indexName(<IndexName>)        // use a secondary index    
```

---

## querySome

```scala
  def scanSome[A: Schema](
    tableName: String,
    limit: Int
  ): DynamoDBQuery[A, (Chunk[A], LastEvaluatedKey)] = ???  
```

The `scanSome` operation is used page `limit` number of items in a table, and returns them in a tuple of `Chunk[A]` and `LastEvaluatedKey`. The `LastEvaluatedKey` can be used to continue scanning the table from where the last page left off using the `startKey` combinator. 

This paging behavior is useful when you have a large number of items in a table and you want to process them in smaller chunks to avoid memory issues - eg a REST API for a paging front end.

```scala
for {
  _                          <- put(tableName, Equipment("1", 2020, "Widget1", 1.0)).execute
  _                          <- put(tableName, Equipment("1", 2021, "Widget1", 2.0)).execute
  _                          <- put(tableName, Equipment("1", 2022, "Widget1", 2.1)).execute
  t                          <- querySome[Equipment](tableName, limit = 2).execute
  (page1, lastEvaluatedKey1) =  t
  t2                         <- querySome[Equipment](tableName, limit = 1).startKey(lastEvaluatedKey1).execute
  (page2, lastEvaluatedKey2) =  t2
} yield ()
```

## Combinators

```scala
<SCAN_SOME_QUERY>
  .consistency(<ConsistencyMode>)
  .whereKey(<KeyConditionExpr>)  // eg Equipment.id.partitionKey === "1" && Equipment.year.sortKey > 2020
  .startKey(<LastEvaluatedKey>)
  .filter(<ConditionExpression>) // eg Equipment.price > 1.0 - filtering is done server side AFTER the scan
  .sortOrder(ascending)          // ascending is a boolean flag
  .indexName(<IndexName>)        // use a secondary index    
```

---

## scanAll

```scala
  def scanAll[A: Schema](
    tableName: String
  ): DynamoDBQuery[A, Stream[Throwable, A]] = ???
```

The `scanAll` operation is used to scan all items in a table, and uses a ZIO stream to return the results.  

Note that scanning all items in a table can be an expensive operation in terms of elapsed time - to speed things up the 
AWS API offers a parallel/segmented scanning mode which can be invoked in the High Level API using the `parallel` combinator - 
internally it takes care of the details of calling the AWS segmented scan API using a fiber per segment and the results 
are merged back into the results stream in an undetermined order.

```scala
for {
  _          <- put(tableName, Equipment("1", 2020, "Widget1", 1.0)).execute
  _          <- put(tableName, Equipment("1", 2021, "Widget1", 2.0)).execute
  stream     <- scanAll[Equipment](tableName)
                  .whereKey(Equipment.id.partitionKey === "1")
                  .execute
  _ <- stream.tap(equip => ZIO.debug(s"equipment: $equip")).runDrain 
} yield ()
```

## Combinators

```scala
<SCAN_ALL_QUERY>
  .consistency(<ConsistencyMode>)
  .filter(<ConditionExpression>) // eg Equipment.price > 1.0 - filtering is done server side AFTER the scan  
  .parallel(<N>)                 // executes a native DDB parallel scan on the server and merges the results back to the stream
  .indexName(<IndexName>)        // use a secondary index    
```

---

## scanSome

```scala
  def scanSome[A: Schema](
    tableName: String,
    limit: Int
  ): DynamoDBQuery[A, (Chunk[A], LastEvaluatedKey)] = ???  
```

The `scanSome` operation is used page `limit` number of items in a table, and returns them in a tuple of `Chunk[A]` and `LastEvaluatedKey`. The `LastEvaluatedKey` can be used to continue scanning the table from where the last page left off using the `startKey` combinator. 

This paging behavior is useful when you have a large number of items in a table and you want to process them in smaller chunks to avoid memory issues - eg a REST API for a paging front end.

```scala
for {
  _                          <- put(tableName, Equipment("1", 2020, "Widget1", 1.0)).execute
  _                          <- put(tableName, Equipment("1", 2021, "Widget1", 2.0)).execute
  _                          <- put(tableName, Equipment("1", 2022, "Widget1", 2.1)).execute
  t                          <- scanSome[Equipment](tableName, limit = 2).execute
  (page1, lastEvaluatedKey1) = t 
  t2                         <- scanSome[Equipment](tableName, limit = 1).startKey(lastEvaluatedKey1).execute
} yield ()
```

## Combinators

```scala
<SCAN_SOME_QUERY>
  .consistency(<ConsistencyMode>)
  .startKey(<LastEvaluatedKey>)
  .filter(<ConditionExpression>) // eg Equipment.price > 1.0 - filtering is done server side AFTER the scan 
  .indexName(<IndexName>)        // use a secondary index    
```

---

## AttributeValue

The sealed trait `AttributeValue` has a one to one correspondence with the concept of an [attribute value in the AWS DDB API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html). It has implementations for all the types that are supported by DynamoDB

Internally there are the type classes `ToAttributeValue` and `FromAttribute` to convert between Scala types and `AttributeValue` and vice versa
whenever these conversions are required. Some example constructors are show below:

```scala
val s = AttributeValue.String("hello")
val bool = AttributeValue.Boolean(true)
```

However, even when working with the Low Level API you would not use these constructors directly, instead you would use the `AttrMap` class (or more likely one of it's type aliases) which is a convenience container for working with `AttributeValue` instances (see next section).

# AttrMap
An `AttrMap` is a convenience container for working an [DDB Item](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html) and _cuts down on boilerplate code when working with the Low Level API_. Conceptually it is a map of field name to AttributeValue `Map[String, AttributeValue]`.
However rather than requiring you to create an `AttributeValue` instance manually for each field, you can work with literal Scala types and type classes will handle the conversion to `AttributeValue` for you. There are also the the `PrimaryKey` and `Item` type aliases for `AttrMap` for readability as these structures follow the same pattern.

Some examples are shown below:

```scala
val attrMap = AttrMap("id" -> "1", "age" -> 30) 
val item = Item("id" -> "1", "age" -> 30) // uses Item type alias
val pk = PrimaryKey("id" -> "1", "count" -> 30) // uses PrimaryKey type alias 
// AttrMaps can also be nested 
val item = Item("id" -> "1", "age" -> 30, "address" -> Item("city" -> "London", "postcode" -> "SW1A 1AA")) 
```

The example below demonstrate the reduction in boilerplate when compared to working with `AttributeValue` directly:

```scala
val attrMap1 = AttrMap("id" -> "1", "age" -> 30) 
val attrMap2 = Map("id" -> AttributeValue.String("1"), "age" -> AttributeValue.Number(30))
```

There are also some convenience methods on the `AttrMap` class for accessing fields:

```scala

  // Field accessors

  def get[A](field: String)(implicit ev: FromAttributeValue[A]): Either[ItemError, A] = ???

  def getOptional[A](field: String)(implicit ev: FromAttributeValue[A]): Either[Nothing, Option[A]] = ???

  def getItem[A](field: String)(f: AttrMap => Either[ItemError, A]): Either[ItemError, A] = ???

  // methods for accessing fields that are Item's themselves

  def getOptionalItem[A](
    field: String
  )(f: AttrMap => Either[ItemError, A]): Either[ItemError, Option[A]] = ???

  def getIterableItem[A](
    field: String
  )(f: AttrMap => Either[ItemError, A]): Either[ItemError, Iterable[A]] = ???

  def getOptionalIterableItem[A](
    field: String
  )(f: AttrMap => Either[ItemError, A]): Either[ItemError, Option[Iterable[A]]] = ???
```

An example of using AttrMap access methods is shown below:

```scala
val attrMap                                = AttrMap("f1" -> AttrMap("f2" -> "a", "f3" -> "b"))
val either: Either[ItemError, Option[Foo]] = for {
  maybeFoo <- attrMap.getOptionalItem("f1") { m =>
           for {
             s <- m.get[String]("f2")
             o <- m.getOptional[String]("f3")
           } yield Foo(s, o)
         }
  } yield maybeFoo
```

---

## $ and parse functions

The `$` and `parse` functions are the primary way to create a `ProjectionExpression` in the Low Level API.

They take a string representation of the attribute path that uses the 
[standard AWS DynamoDB path expression syntax](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ProjectionExpressions.html)
and return a `ProjectionExpression`.

eg

```scala
final case class Person(email: String, hobbies: Map[String, List[String]])

val errorOrProjectionExpression: Either[String, ProjectionExpression] = ProjectionExpression.parse("hobbies.sport[0]")
val projectionExpression: ProjectionExpression = ProjectionExpression.$("hobbies.sport[0]")
```

The `parse` function takes a string representation of the path expression (that uses standard AWS DynamoDB path expression syntax).
It returns an `Either` which is a `Left[String]` when the string projection expression is invalid, or a `Right[ProjectionExpression]`
when the string projection expression valid and successfully parsed.

The `$` function is the unsafe version of the `parse` function - if the string projection expression is invalid it throws an `IllegalStateException`. 

The below table shows the supported path expression syntax:

Expression Syntax | Description
---|---
`"foo"` | top level element  
`"foos[0]"` | simple top level list element - indexes are zero based  
`"foo.bar"` | addressing a map element inside a top level element  
`"foo_bar"` | underscores are valid
`"foo-bar"` | hyphens are valid
&#96`foo.bar`&#96 | special characters such as "dot" need to be escaped with backticks. Here we have an attribute name that contains a special character. 

After the expression is parsed into a `ProjectionExpression`, at execution time the AWS API [Expression Attributes Names](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ExpressionAttributeNames.html) and 
[Expression Attributes Values](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ExpressionAttributeValues.html)
maps are used to escape all path elements safely when required. 

## Low level representation of `ProjectionExpression`

For those who are interested the below code snippet shows the slightly simplified version of the `ProjectionExpression` implementation:

```scala
sealed trait ProjectionExpression
object ProjectionExpression {
  final case object Root extends ProjectionExpression
  final case class ListElement(parent: ProjectionExpression, index: Int) extends ProjectionExpression
  final case class MapElement(parent: ProjectionExpression, key: String) extends ProjectionExpression
}

val projectionExpression: ProjectionExpression = ProjectionExpression.$("hobbies.sport[0]")
println(projectionExpression) // ListElement(MapElement(MapElement(Root, "hobbies"), "sport"), 0)
```

Note that top level elements are represented as `MapElement(Root, <ELEMENT_NAME>)`.

---

## Low Level API(Low-level-api)

Low Level API operations are found on the `DynamoDBQuery` companion object. All the function names contain the word `Item` to indicate that they are operations on AWS DDB [items](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html) in a table.

```scala
object DynamoDBQuery {

  // CRUD operations  

  def putItem(tableName: String, item: Item): DynamoDBQuery[Any, Option[Item]] = ???

  def getItem(
    tableName: String,
    key: PrimaryKey,
    projections: ProjectionExpression[_, _]*
  ): DynamoDBQuery[Any, Option[Item]] = ???

  def updateItem[A](tableName: String, key: PrimaryKey)(action: Action[A]): DynamoDBQuery[A, Option[Item]] = ???

  def deleteItem(tableName: String, key: PrimaryKey): Write[Any, Option[Item]] = ???

  // Scan/Query operations

  def scanSomeItem(tableName: String, limit: Int, projections: ProjectionExpression[_, _]*): ScanSome = ???

  def scanAllItem(tableName: String, projections: ProjectionExpression[_, _]*): ScanAll = ???

  def queryAllItem(tableName: String, projections: ProjectionExpression[_, _]*): QueryAll = ???

  def querySomeItem(tableName: String, limit: Int, projections: ProjectionExpression[_, _]*): QuerySome = ???

}
```

---

## ProjectionExpression

The `ProjectionExpression` API is common to both the Type Safe High Level API (with the exception the primary keys API) 
and the Low Level API - however the way one is constructed is different.

**High Level API construction**

```scala
final case class Person(id: String, group: String name: String)
object Person {
  implicit val schema: Schema.CaseClass3[String, String, String, Person] = DeriveSchema.gen[Person]
  
  // id and name are ProjectionExpressions
  val (id, group, name) = ProjectionExpression.accessors[Person]
}
```

**Low Level API construction**

```scala
// id and name are ProjectionExpressions
val id = ProjectionExpression.$("id")
val name = ProjectionExpression.$("name")
```
For more details see [$ and Parse functions](low-level-api/dollar-function) section

**Type Safety**

The other notable difference is that High Level API is totally type safe and many subtle errors are caught at compile time:

```scala
Person.id === 1 // will not compile
Person.id === "1" && Person.name === 2 // will not compile
Person.id === "1" && Employee.name === "2" // will not compile

ProjectionExpression.$("name") === 1 // will compile!
```

### `ProjectionExpression` API Summary

Once we have a `ProjectionExpression` we can use it as a springboard to create further expressions such as 
`ConditionExpression`, `UpdateExpression` and `PrimaryKeyExpression`:

 Category                                                    | Method               | Description
-------------------------------------------------------------|----------------------|---
 Primary Keys (High Level API only)                          | `partitionKey`       | `Person.id.partitionKey === "1"`
             | `sortKey`            | `Person.id.partitionKey === "1" && Person.group.sortKey === "2"`
`ConditionExpression` / `FilterExpression` | `===`                | Equality
             | `<>`                 | Inequality
             | `>`                  | Greater than
             | `>=`                 | Greater than or equal
             | `<`                  | Less than
             | `<=`                 | Less than or equal
             | `exists`             | returns true if the attribute exists
             | `notExists`          | returns true if the attribute does not exists
             | `size`               | returns size of attribute. Applies to all types except Number and Boolean
             | `isXXXX`             | There is a function per type eg `isBinary` - returns true if the attribute is of the specified type
             | `between(from, to)`  | returns true if the attribute is between the supplied values, inclusive
             | `beginsWith`         | Only applies to string attributes
             | `inSet`              | returns true if the attribute is in the supplied set
             | `in(a, b, c ...)`    | returns true if the attribute matches one of the supplied values
            | `contains`           | returns true if the attribute contains the supplied value - applies to a String, Set, List
            | `containsSet`        | returns true if the attribute contains the supplied set - applies to a String, Set, List. Creates a composite condition expression consisting of multiple `contains` expressions joined by an `and`
`UpdateExpression`| `+`                  | combines update actions eg `Person.name.set(42) + Person.age.set(42)`  
            | `set`                | Set an attribute `Person.name.set("John")`                                                                         
            | `setIfNotExists`     | Set attribute if it does not exists `Person.name.setIfNotExists("John")`                                           
            | `appendList`         | Add supplied list to the end of this list attribute                                                                
            | `prepend`            | Prepend an element to a list attribute                                                                             
            | `prependList`        | Prepend a list to list attribute                                                                                   
            | `deleteFromSet`      | delete all elements that match the supplied set                                                                    
            | `add(a: To)`         | adds this value as a number attribute if it does not exists, else adds the numeric value to the existing attribute 
            | `addSet`             | Adds this set as an attribute if it does not exists, else if it exists it adds the elements of the set             
            | `remove(index: Int)` | remove an element at the specified index                                                                           
            | `remove`             | Removes this path expression from an item

---

## ZIO DynamoDB JSON Module

**`zio-dynamodb-json`** is a new experimental optional module designed for debugging and troubleshooting purposes.
It renders a representation of the native DynamoDB types and data in a standard JSON format used in AWS console views

It uses the same codecs as the regular library and provides an accurate representation of how the data that would be stored in DynamoDB. 
It works at the level of the AttributeValue type and so works with both the low level and high level APIs.
Note ATM it does not support the Binary and Binary Set types.

Internally the module uses ZIO JSON and in particular it uses the AST which is not used by ZIO Json itself for efficiency reasons,
hence the recommendations is not to use the module in production code.

Some example use cases include:
- visualizing the Attribute Value representation of a case class during model development
- can be used in unit tests to verify that DB mapping is as expected
- production troubleshooting - grabbing DDB JSON from the AWS console in production and decoding it to a case class for debugging

## Usage

In your `build.sbt` file add the following line:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-dynamodb-json" % "1.0.0-RC24"
)
```

Add the following import to your code:

```scala

```

## Methods

Method | Description
--- | ---
**`toJsonString`** // extension method | Converts a case class to a JSON string
**`toJsonStringPretty`** // extension method | Converts a case class to a pretty printed JSON string
**`def parse[A: Schema](jsonString: String): Either[DynamoDBError.ItemError, A]`** | takes a JSON string and returns an Either of an error or an `A` 
**`def parseItem(json: String): Either[DynamoDBError.ItemError, AttrMap]`** | takes a JSON string and returns an Either of an error or an `Item` 

## Visualizing the DB representation of a case class during model development

This can be accomplished by using either the `toJsonString` or `toJsonStringPretty` extension methods on a case class instance - eg:

```scala
final case class Person(
  email: String,
  hobbies: Map[String, List[String]],
  friends: Set[String],
  registrationDate: Instant
)
object Person {
  implicit val schema: Schema.CaseClass4[String, Map[String, List[String]], Set[String], Instant, Person] =
    DeriveSchema.gen[Person]
}
val person = Person("email", Map("sports" -> List("cricket", "football")), Set("John", "Tarlochan"), Instant.now)
println(person.toJsonStringPretty[Person])
```
Console output:
```json
{
  "registrationDate" : {
    "S" : "2024-12-05T06:44:49.011916Z"
  },
  "friends" : {
    "SS" : [
      "John",
      "Tarlochan"
    ]
  },
  "hobbies" : {
    "sports" : {
      "L" : [
        {
          "S" : "cricket"
        },
        {
          "S" : "football"
        }
      ]
    }
  },
  "email" : {
    "S" : "email"
  }
}
```

For a basic introduction to data modelling [Creating Models](reference/hi-level-api/creating-models/index.md) section.

For details on how to customise data mappings please see the [Codec Customisation](guides/codec-customization.md) section.

## Grabbing JSON from the AWS console and decoding it to a case class

This can be useful in a production troubleshooting scenario where you have a JSON representation of a DynamoDB item 
from the AWS console, and you want to decode it to a case class for local testing/debugging purposes.

eg:

```scala
val json             =
    """{
        "registrationDate" : {
          "S" : "2024-12-05T06:44:49.011916Z"
        },
        "friends" : {
          "SS" : [
          "John",
          "Tarlochan"
          ]
        },
        "hobbies" : {
          "sports" : {
          "L" : [
        {
          "S" : "cricket"
        },
        {
          "S" : "football"
        }
          ]
        }
        },
        "email" : {
          "S" : "email"
        }
      }"""
  val errorOrPerson = parse[Person](json) // ZIO DynamoDB Json parse method used here
  println(
    errorOrPerson
  ) // Right(Person("email", Map("sports" -> List("cricket", "football")), Set("John", "Tarlochan"), 2024-12-05T06:44:49.011916Z))
```

---

## Environment

FtpClient & Blocking
---

ZIO Ftp use Environment type parameter to required a blocking context. Since (s)ftp java client don't provide any async client, 
we are lifting all blocking (effectful) functions into a blocking execution context by using `zio.blocking.effectBlocking()`.

Since we never expose the underlying ftp client, you can only use a ZIO wrapper `FtpAccessors[_]`

``` scala 
trait FtpAccessors[+A] {
 def stat(path: String): ZIO[Blocking, IOException, Option[FtpResource]]
 def ls(path: String): ZStream[Blocking, IOException, FtpResource]
}
```

All provided functions require a `Blocking` context.

There is already predefined Blocking context defined in ZIO `zio.blocking.Blocking.live`

How to provide Blocking Environment ?
---

Your main entrypoint should extend `zio.App` 

```scala

// Provide default environment(s) context such as (Blocking, Clock, Console,...) since it is provided by zio.App which extend zio.DefaultRuntime
object MyApp extends App {
    val program: ZIO[Blocking, IOException, Unit] = ???
    
    final def run(args: List[String]): ZIO[ZEnv, Nothing, Int] = program
  }
```

or you can simply provide your own blocking context 

```scala

val program: ZIO[Blocking, IOException, Unit] = ???
// Blocking environment is not anymore required
val run: ZIO[Any, IOException, Unit] = program.provideLayer(Blocking.live)
```

Resources
---
- [ZIO Effect Blocking](https://zio.dev/docs/overview/overview_creating_effects#blocking-synchronous-side-effects)
- [ZIO environment by John de goes](http://degoes.net/articles/zio-environment)
- [ZIO environment meets constructor based-dependency injection](https://blog.softwaremill.com/zio-environment-meets-constructor-based-dependency-injection-6a13de6e000)

---

## ZIO FTP

[ZIO FTP](https://zio.dev) is a thin wrapper over (s)Ftp client for [ZIO](https://zio.dev).

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-ftp/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-ftp_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-ftp_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-ftp_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-ftp_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-ftp-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-ftp-docs_2.13) [![ZIO FTP](https://img.shields.io/github/stars/zio/zio-ftp?style=social)](https://github.com/zio/zio-ftp)

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-ftp" % "0.5.2" 
```

## How to use it?

* Imports
```scala

```

* FTP
```scala
// FTP
val unsecureSettings = UnsecureFtpSettings("127.0.0.1", 21, PasswordCredentials("foo", "bar"))

//listing files
Ftp.ls("/").runCollect.provideLayer(unsecure(unsecureSettings))
```

* FTPS
```scala
// FTPS
val secureSettings = SecureFtpSettings("127.0.0.1", 21, PasswordCredentials("foo", "bar"))

//listing files
SFtp.ls("/").runCollect.provideLayer(secure(secureSettings))
```

* SFTP (support ssh key)

```scala
val sftpSettings = SecureFtpSettings("127.0.0.1", 22, PasswordCredentials("foo", "bar"))

//listing files
SFtp.ls("/").runCollect.provideLayer(secure(sftpSettings))
```

## Example

First we need an FTP server, so let's create one:

```bash
docker run -d \
    -p 21:21 \
    -p 21000-21010:21000-21010 \
    -e USERS="one|1234" \
    -e ADDRESS=localhost \
    delfer/alpine-ftp-server
```

Now we can run the example:

```scala

object ZIOFTPExample extends ZIOAppDefault {

  private val settings =
    UnsecureFtpSettings("127.0.0.1", 21, PasswordCredentials("one", "1234"))

  private val myApp: ZIO[Ftp, IOException, Unit] =
    for {
      _        <- Console.printLine("List of files at root directory:")
      resource <- ls("/").runCollect
      _        <- ZIO.foreach(resource)(e => Console.printLine(e.path))
      path      = "~/file.txt"
      _        <- upload(
                    path,
                    ZStream.fromChunk(
                      Chunk.fromArray("Hello, ZIO FTP!\nHello, World!".getBytes)
                    )
                  )
      file     <- readFile(path)
                    .via(ZPipeline.utf8Decode)
                    .runCollect
      _        <- Console.printLine(s"Content of $path file:")
      _        <- Console.printLine(file.mkString)
    } yield ()

  override def run = myApp.provideSomeLayer(unsecure(settings))
}
```

## Support any commands?

If you need a method which is not wrapped by the library, you can have access to underlying FTP client in a safe manner by using

```scala

trait FtpAccessors[+A] {
  def execute[T](f: A => T): ZIO[Any, IOException, T]
} 
```

---

## Configuration(Zio-json)

## Field naming

By default, the field names of a case class are used as the JSON fields, but it is easy to override this with an annotation `@jsonField`.

Moreover, you can also mark a whole case class with a member name transformation that will be applied to all members using `@jsonMemberNames` annotation. It takes an argument of type `JsonMemberFormat` which encodes the transformation that will be applied to member names.

Four most popular transformations are already provided: KebabCase, SnakeCase, PascalCase and CamelCase. If you require something more specific you can also use CustomCase which takes a function of shape `String => String` as an argument and can be used to perform any arbitrary transformation. `@jsonField` annotation takes priority over the transformation defined by `@jsonMemberNames`.

Here's an example json with most fields snake_cased and one kebab-cased:
```json
{
  "passion_fruit": true,
  "granny_smith": true,
  "dragon_fruit": true,
  "blood-orange": false
}
```

And here's the target case class:

```scala

@jsonMemberNames(SnakeCase)
case class FruitBasket(
  passionFruit: Boolean, 
  grannySmith: Boolean, 
  dragonFruit: Boolean, 
  @jsonField("blood-orange") bloodOrange: Boolean
)
```

Notice that all fields are camelCased in Scala and will be both encoded and decoded correctly to snake_case in JSON except `bloodOrange` field that is annotated with a `@jsonField` override that will force it to become `"blood-orange"` after serialization.

It is also possible to change the type hint that is used to discriminate case classes with `@jsonHint`.

For example, these annotations change the expected JSON of our `Fruit` family

```scala

sealed trait Fruit

@jsonHint("banaani") case class Banana(
  @jsonField("bendiness") curvature: Double
) extends Fruit

@jsonHint("omena") case class Apple(
  @jsonField("bad") poison: Boolean
) extends Fruit

object Fruit {
  implicit val codec: JsonCodec[Fruit] =
    DeriveJsonCodec.gen[Fruit]
}

val banana: Fruit = Banana(0.5)
// banana: Fruit = Banana(curvature = 0.5)
val apple: Fruit = Apple(false)
// apple: Fruit = Apple(poison = false)
```

from

```json
{"Banana":{"curvature":0.5}}
{"Apple":{"poison":false}}
```

to

```scala
banana.toJson
// res1: String = {"banaani":{"bendiness":0.5}}
```

```scala
apple.toJson
// res2: String = {"omena":{"bad":false}}
```

Another way of changing type hint is using `@jsonHintNames` annotation on sealed class. It allows to apply transformation
to all type hint values in hierarchy. Same transformations are provided as for `@jsonMemberNames` annotation.

Here's an example:

```scala

@jsonHintNames(SnakeCase)
sealed trait FruitKind

case class GoodFruit(good: Boolean) extends FruitKind

case class BadFruit(bad: Boolean) extends FruitKind

object FruitKind {
  implicit val codec: JsonCodec[FruitKind] =
    DeriveJsonCodec.gen[FruitKind]
}

val goodFruit: FruitKind = GoodFruit(true)
// goodFruit: FruitKind = GoodFruit(good = true)
val badFruit: FruitKind = BadFruit(true)
// badFruit: FruitKind = BadFruit(bad = true)

goodFruit.toJson
// res3: String = "{\"good_fruit\":{\"good\":true}}"
badFruit.toJson
// res4: String = "{\"bad_fruit\":{\"bad\":true}}"
```

Note that with this code, you can't directly decode the subclasses of `FruitKind`. You would need to create a dedicated decoder for each subclass.

```scala
object GoodFruit {
  implicit val codec: JsonCodec[GoodFruit] =
    DeriveJsonCodec.gen[GoodFruit]
}
```

Since `GoodFruit` is only a case class, it will not require any kind of discriminator to be decoded.
    
```scala
"""{"good":true}""".fromJson[GoodFruit]
// res5: Either[String, GoodFruit] = Right(value = GoodFruit(good = true))
```

If you want for some reason to decode only for a specific type of `FruitKind` that has a discriminator, don't derive the codec for the subtype, but transform the `FruitKind` codec.

```scala
object BadFruit {
  implicit val decoder: JsonDecoder[BadFruit] =
    FruitKind.codec.decoder.mapOrFail {
        case GoodFruit(_) => Left("Expected BadFruit, got GoodFruit")
        case BadFruit(bad) => Right(BadFruit(bad))
      }
}
```

## jsonDiscriminator

A popular alternative way to encode sealed traits:

```json
{"type":"banaani", "bendiness":0.5}

{"type":"omena", "bad":false}
```

is discouraged for performance reasons. However, if we have no choice in the matter, it may be accommodated with the `@jsonDiscriminator` annotation

```scala
@jsonDiscriminator("type") sealed trait Fruit
```

## Extra fields

We can raise an error if we encounter unexpected fields by using the `@jsonNoExtraFields` annotation on a case class.

```scala
@jsonNoExtraFields case class Watermelon(pips: Int)

object Watermelon {
  implicit val decoder: JsonDecoder[Watermelon] =
    DeriveJsonDecoder.gen[Watermelon]
}
```

```scala
"""{ "pips": 32 }""".fromJson[Watermelon]
// res7: Either[String, Watermelon] = Right(value = Watermelon(pips = 32))
```

```scala
"""{ "pips": 32, "color": "yellow" }""".fromJson[Watermelon]
// res8: Either[String, Watermelon] = Left(value = "(invalid extra field)")
```

## Aliases

    Since zio-json 0.4.3.

After a case class field has changed name, you may still want to read JSON documents that use the old name. This is supported by the `@jsonAliases` annotation.

```scala
case class Strawberry(
  @jsonAliases("seeds") seedCount: Int
)

object Strawberry {
  implicit val decoder: JsonDecoder[Strawberry] =
    DeriveJsonDecoder.gen[Strawberry]
}
```

The following two expressions result in an equal value:
```scala
"""{ "seeds": 32 }""".fromJson[Strawberry]
// res9: Either[String, Strawberry] = Right(value = Strawberry(seedCount = 32))
"""{ "seedCount": 32 }""".fromJson[Strawberry]
// res10: Either[String, Strawberry] = Right(
//   value = Strawberry(seedCount = 32)
// )
```

The `@jsonAliases` annotation supports multiple aliases. The annotation has no effect on encoding.

## Nulls, explicitNulls

By default `null` values are omitted from the JSON output. This behavior can be changed by using the `@jsonExplicitNull` annotation on a case class, field or setting `JsonCodecConfiguration.explicitNulls` to `true`.
Missing nulls on decoding are always allowed.

```scala
@jsonExplicitNull
case class Mango(ripeness: Option[Int])

object Mango {
  implicit val codec: JsonCodec[Mango] = DeriveJsonCodec.gen[Mango]
}
```
The following expression results in a JSON document with a `null` value:
```scala
Mango(None).toJson
// res11: String = "{\"ripeness\":null}"
"""{}""".fromJson[Mango]
// res12: Either[String, Mango] = Right(value = Mango(ripeness = None))
```

## Empty Collections, explicitEmptyCollections

By default `empty collections` (all supported collection types and case classes) are included from the JSON output an decoding requires empty collections to be present. This behavior can be changed by using the `@jsonExplicitEmptyCollections(encoding = false, decoding = false)` annotation on a case class, field or setting `JsonCodecConfiguration.explicitEmptyCollections` to `ExplicitEmptyCollections(encoding = false, decoding = false)`. The result is that empty collections are omitted from the JSON output and when decoding empty collections are created. It is also possible to have different values for encoding and decoding by using `@jsonExplicitEmptyCollections(encoding = true, decoding = false)` or `@jsonExplicitEmptyCollections(encoding = false, decoding = true)`.

```scala
@jsonExplicitEmptyCollections(encoding = false, decoding = false)
case class Pineapple(leaves: List[String])

object Pineapple {
  implicit val codec: JsonCodec[Pineapple] = DeriveJsonCodec.gen[Pineapple]
}
```
The following expression results in a JSON document with an empty collection:
```scala
Pineapple(Nil).toJson
// res13: String = "{}"
"""{}""".fromJson[Pineapple]
// res14: Either[String, Pineapple] = Right(value = Pineapple(leaves = List()))
```

## @jsonDerive

**Requires zio-json-macros**

`@jsonDerive` allows to reduce that needs to be written using an annotation macro to generate JsonDecoder/JsonEncoder at build-time.

For generating both Encoder and Decoder, simply use jsonDerive

For example: 

```scala

@jsonDerive case class Watermelon(pips: Int)
```
It is equivalent to:

```scala

case class Watermelon(pips: Int)

object Watermelon {
  implicit val codec: JsonCodec[Watermelon] =
    DeriveJsonCodec.gen[Watermelon]
}
```

To generate only an encoder, we can set it as config parameter:

For example:

```scala

@jsonDerive(JsonDeriveConfig.Encoder) case class Watermelon(pips: Int)
```
It is equivalent to:

```scala

case class Watermelon(pips: Int)

object Watermelon {
  implicit val encoder: JsonEncoder[Watermelon] =
    DeriveJsonEncoder.gen[Watermelon]
}
```

To generate only a decoder, we can set it as config parameter:

For example:

```scala modc:compile-only

@jsonDerive(JsonDeriveConfig.Decoder) case class Watermelon(pips: Int)
```
It is equivalent to:

```scala

case class Watermelon(pips: Int)

object Watermelon {
  implicit val decoder: JsonDecoder[Watermelon] =
    DeriveJsonDecoder.gen[Watermelon]
}
```

---

## Decoding

## Automatic Derivation

Say we want to be able to read some JSON like

```json
{ "curvature": 0.5 }
```

into a Scala `case class`

```scala
case class Banana(curvature: Double)
```

To do this, we create an *instance* of the `JsonDecoder` typeclass for `Banana` using the `zio-json` code generator. It is best practice to put it on the companion of `Banana`, like so

```scala

object Banana {
  implicit val decoder: JsonDecoder[Banana] =
    DeriveJsonDecoder.gen[Banana]
}
```

Now we can parse JSON into our object

```scala
"""{ "curvature": 0.5 }""".fromJson[Banana]
// res0: Either[String, Banana] = Right(value = Banana(curvature = 0.5))
```

### Automatic Derivation and case class default field values

If a case class field is defined with a default value and the field is not present or `null`, the default value will be used (or evaluated when it is a method).

Say we have a Scala `case class`

```scala
case class Entity(id: Long, description: String = "", related: Seq[Entity] = Seq())

implicit val decoder: JsonDecoder[Entity] =
  DeriveJsonDecoder.gen[Entity]
// decoder: JsonDecoder[Entity] = zio.json.DeriveJsonDecoder$$anon$1@744ae108
```

```scala
"""{ "id": 42, "related": null }""".fromJson[Entity]
// res1: Either[String, Entity] = Right(
//   value = Entity(id = 42L, description = "", related = List())
// )
```

_Note: If you’re using Scala 3 and your case class is defining default parameters, `-Yretain-trees` needs to be added to `scalacOptions`._

## ADTs

Say we extend our data model to include more data types

```scala
sealed trait Fruit

case class Banana(curvature: Double) extends Fruit
case class Apple (poison: Boolean)   extends Fruit
```

we can generate the decoder for the entire `sealed` family:

```scala

object Fruit {
  implicit val decoder: JsonDecoder[Fruit] =
    DeriveJsonDecoder.gen[Fruit]
}
```

```scala
"""{ "Banana":{ "curvature":0.5 }}""".fromJson[Fruit]
// res3: Either[String, Fruit] = Right(value = Banana(curvature = 0.5))
```

```scala
"""{ "Apple": { "poison": false }}""".fromJson[Fruit]
// res4: Either[String, Fruit] = Right(value = Apple(poison = false))
```

### String-based union types (Enum)
The codecs support string-based union types (enums) out of the box. This is useful when the overhead of a Enum is not desired.

```scala
val appleOrBanana: "Apple" | "Banana" = "Apple"
```
Decoding succeeds because 'Apple' is a valid value
```scala
appleOrBanana.toJson
"Apple".fromJson["Apple" | "Banana"]
```
Decoding fail because 'Pear' is not a valid value
```scala
"Peer".fromJson["Apple" | "Banana"]
```

Almost all of the standard library data types are supported as fields on the case class, and it is easy to add support if one is missing.

### Sealed families and enums for Scala 3
Sealed families where all members are only objects, or a Scala 3 enum with all cases parameterless are interpreted as enumerations and will encode 1:1 with their value-names.
```scala
enum Foo derives JsonDecoder:
  case Bar
  case Baz
  case Qux
```
or
```scala
sealed trait Foo derives JsonDecoder
object Foo:
  case object Bar extends Foo
  case object Baz extends Foo
  case object Qux extends Foo
```

## Manual instances

Sometimes it is easier to reuse an existing `JsonDecoder` rather than generate a new one. This can be accomplished using convenience methods on the `JsonDecoder` typeclass to *derive* new decoders

```scala
trait JsonDecoder[A] {
  def map[B](f: A => B): JsonDecoder[B]
  def mapOrFail[B](f: A => Either[String, B]): JsonDecoder[B]
  ...
}
```

### `.map`

We can `.map` from another `JsonDecoder` in cases where the conversion will always succeed. This is very useful if we have a `case class` that simply wraps another thing and shares the same expected JSON.

For example, say we want to model the count of fruit with a `case class` to provide us with additional type safety in our business logic (this pattern is known as a *newtype*).

```scala
case class FruitCount(value: Int)
```

but this would cause us to expect JSON of the form

```json
{"value": 1}
```

wheres we really expect the raw number. We can derive a decoder from `JsonDecoder[Int]` and `.map` the result into a `FruitCount`

```scala
object FruitCount {
  implicit val decoder: JsonDecoder[FruitCount] =
    JsonDecoder[Int].map(FruitCount(_))
}
```

and now the `JsonDecoder` for `FruitCount` just expects a raw `Int`.

```scala
"""3""".fromJson[FruitCount]
// res5: Either[String, FruitCount] = Right(value = FruitCount(value = 3))
```

Another use case is if we want to encode a `case class` as an array of values, rather than an object with named fields. Such an encoding is very efficient because the messages are smaller and require less processing, but are very strict schemas that cannot be upgraded.

```scala:mdoc:reset

case class Things(s: String, i: Int, b: Boolean)

object Things {
  implicit val decoder: JsonDecoder[Things] =
    JsonDecoder[(String, Int, Boolean)].map { case (p1, p2, p3) => Things(p1, p2, p3) }
}

"""[ "hello", 1, true ]""".fromJson[Things]
```

### `.mapOrFail`

We can use `.mapOrFail` to take the result of another `JsonDecoder` and try to convert it into our custom data type, failing with a message if there is an error.

Say we are using the [`refined`](https://github.com/fthomas/refined) library to ensure that a `Person` data type only holds a non-empty string in its `name` field

```scala

case class Person(name: String Refined NonEmpty)
```

we will get a compile time error because there is no `JsonDecoder[String Refined NonEmpty]`.

```scala
object Person {
  implicit val decoder: JsonDecoder[Person] = DeriveJsonDecoder.gen
}
// error: magnolia: could not find JsonDecoder.Typeclass for type eu.timepit.refined.api.Refined[String,eu.timepit.refined.collection.NonEmpty]
//     in parameter 'name' of product type MdocApp0.this.Person
// 
//   implicit val decoder: JsonDecoder[Person] = DeriveJsonDecoder.gen
//                                               ^^^^^^^^^^^^^^^^^^^^^
```

However, we can derive one by requesting the `JsonDecoder[String]` and calling `.mapOrFail`, supplying the constructor for our special `String Refined NonEmpty` type

```scala

implicit val decodeName: JsonDecoder[String Refined NonEmpty] =
  JsonDecoder[String].mapOrFail(refined.refineV[NonEmpty](_))
// decodeName: JsonDecoder[Refined[String, NonEmpty]] = zio.json.JsonDecoder$$anon$4@44a2a185
```

Now the code compiles.

# Parsing custom JSON

In this section we show several approaches for decoding JSON that looks like:

```json
{
  "01. symbol": "IBM",
  "02. open": "182.4300",
  "03. high": "182.8000"
}
```

Which we want to decode into the following case class:

```scala
final case class Quote(
  symbol: String,
  open: String,
  high: String
)
```

All approaches have the same result:

```scala
"""{"01. symbol":"IBM","02. open": "182.4300","03. high": "182.8000"}""".fromJson[Quote]
// >> Right(Quote(IBM,182.4300,182.8000))
// error: could not find implicit value for parameter decoder: zio.json.JsonDecoder[MdocApp0.this.Quote]
// """{"01. symbol":"IBM","02. open": "182.4300","03. high": "182.8000"}""".fromJson[Quote]
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

## Approach 1: use annotation hints

In this approach we enrich the case class with annotations to tell the derived decoder which field names to use.
Obviously, this approach only works if we can/want to change the case class.

```scala

final case class Quote(
  @jsonField("01. symbol") symbol: String,
  @jsonField("02. open") open: String,
  @jsonField("03. high") high: String
)

object Quote {
  implicit val decoder: JsonDecoder[Quote] = DeriveJsonDecoder.gen[Quote]
}
```

## Approach 2: use an intermediate case class

Instead of hints, we can also put the actual field names in an intermediate case class. In our example the field names
are not valid scala identifiers. We fix this by putting the names in backticks:

```scala

final case class Quote(symbol: String, open: String, high: String)

object Quote {
  private final case class JsonQuote(
    `01. symbol`: String,
    `02. open`: String,
    `03. high`: String
  )

  implicit val decoder: JsonDecoder[Quote] =
    DeriveJsonDecoder
      .gen[JsonQuote]
      .map { case JsonQuote(s, o, h) => Quote(s, o, h) }
}
```

## Approach 3: decode to JSON

In this approach we first decode to the generic `Json` data structure. This approach is very flexible because it can
extract data from any valid JSON.

Note that this implementation is a bit sloppy. It uses `toString` on a JSON node. The node is not necessarily a
String, it can be of any JSON type! So this might happily process JSON that doesn't match your expectations.

```scala

final case class Quote(symbol: String, open: String, high: String)

object Quote {
  implicit val decoder: JsonDecoder[Quote] = JsonDecoder[Json]
    .mapOrFail {
      case Json.Obj(fields) =>
        def findField(name: String): Either[String, String] =
          fields
            .find(_._1 == name)
            .map(_._2.toString())  // ⚠️ .toString on any JSON type
            .toRight(left = s"Field '$name' is missing")
  
        for {
          symbol <- findField("01. symbol")
          open <- findField("02. open")
          high <- findField("03. high")
        } yield Quote(symbol, open, high)
      case _ =>
        Left("Not a JSON record")
    }
}
```

## Approach 4: decode to JSON, use cursors

Here we also first decode to `Json`, but now we use cursors to find the data we need. Here we do check that the fields
are actually strings.

```scala

final case class Quote(symbol: String, open: String, high: String)

object Quote {
  private val symbolC = JsonCursor.field("01. symbol") >>> JsonCursor.isString
  private val openC = JsonCursor.field("02. open") >>> JsonCursor.isString
  private val highC = JsonCursor.field("03. high") >>> JsonCursor.isString

  implicit val decoder: JsonDecoder[Quote] = JsonDecoder[Json]
    .mapOrFail { c =>
      for {
        symbol <- c.get(symbolC)
        open <- c.get(openC)
        high <- c.get(highC)
      } yield Quote(symbol.value, open.value, high.value)
  }
}
```

# More custom decoder examples

Let's consider an `Animal` case class with a `categories` field that should be a list of strings. However, some
producers accidentally represent the categories as a comma-separated string instead of a proper list. We want to parse
both cases.

Here's a custom decode for our Animal case class:

```scala

case class Animal(name: String, categories: List[String])

object Animal {
  private val nameC = JsonCursor.field("name") >>> JsonCursor.isString
  private val categoryArrayC = JsonCursor.field("categories") >>> JsonCursor.isArray
  private val categoryStringC = JsonCursor.field("categories") >>> JsonCursor.isString

  implicit val decoder: JsonDecoder[Animal] = JsonDecoder[Json]
    .mapOrFail { c =>
      for {
        name <- c.get(nameC).map(_.value)
        categories <- arrayCategory(c).map(_.toList)
          .orElse(c.get(categoryStringC).map(_.value.split(',').map(_.trim).toList))
      } yield Animal(name, categories)
    }

  private def arrayCategory(c: Json): Either[String, Chunk[String]] =
    c.get(categoryArrayC)
      .flatMap { arr =>
        // Get the string elements, and sequence the obtained eithers to a single either
        sequence(arr.elements.map(_.get(JsonCursor.isString).map(_.value)))
      }

  private def sequence[A, B](chunk: Chunk[Either[A, B]]): Either[A, Chunk[B]] =
    chunk.partition(_.isLeft) match {
      case (Nil, rights) => Right(rights.collect { case Right(r) => r })
      case (lefts, _) => Left(lefts.collect { case Left(l) => l }.head)
    }
}
```

And now, the Json decoder for Animal can handle both formats:
```scala
"""{"name": "Dog", "categories": "Warm-blooded, Mammal"}""".fromJson[Animal]
// res13: Either[String, Animal] = Right(
//   value = Animal(name = "Dog", categories = List("Warm-blooded", "Mammal"))
// )
// >> Right(Animal(Dog,List(Warm-blooded, Mammal)))
"""{"name": "Snake", "categories": [ "Cold-blooded", "Reptile"]}""".fromJson[Animal]
// res14: Either[String, Animal] = Right(
//   value = Animal(name = "Snake", categories = List("Cold-blooded", "Reptile"))
// )
// >>  Right(Animal(Snake,List(Cold-blooded, Reptile)))
```

# JSON AST and Cursors

In most cases it is not necessary to work with the JSON AST directly,
instead it is more convenient to decode directly to domain objects.
However, sometimes it is handy to work with a lower level representation of JSON.
This may for example be the case when you need to work with deeply nested JSON structures 
that would result in deeply nested case classes,
or when you expect a lot of variation in the JSON structure, which would result in nasty decoders.

## JSON AST

To get the AST representation of a JSON string, use the `fromJson[Json]` method.

```scala

val jsonString: String            = """{"name": "John Doe"}"""
// jsonString: String = "{\"name\": \"John Doe\"}"
val jsonAst: Either[String, Json] = jsonString.fromJson[Json]
// jsonAst: Either[String, Json] = Right(
//   value = Obj(fields = IndexedSeq(("name", Str(value = "John Doe"))))
// )
```

The `Json` type is a recursive data structure that can be navigated in a fairly straightforward way.

```scala

val jsonString: String = """{"name": "John Doe"}"""
// jsonString: String = "{\"name\": \"John Doe\"}"
val jsonAst: Json      = jsonString.fromJson[Json].toOption.get
// jsonAst: Json = Obj(fields = IndexedSeq(("name", Str(value = "John Doe"))))
jsonAst match {
  case Obj(fields: Chunk[(String, Json)]) => ()
  case Arr(elements: Chunk[Json])         => ()
  case Bool(value: Boolean)               => ()
  case Str(value: String)                 => ()
  case Num(value: java.math.BigDecimal)   => ()
  case Json.Null                          => ()
}
```

To get the `name` field, you could do the following:

```scala

val json: Option[Json] = """{"name": "John Doe"}""".fromJson[Json].toOption
// json: Option[Json] = Some(
//   value = Obj(fields = IndexedSeq(("name", Str(value = "John Doe"))))
// )
val name: Option[String] = json.flatMap { json =>
  json match {
    case Json.Obj(fields) => fields.collectFirst { case ("name", Json.Str(name)) => name }
    case _                => None
  }
}
// name: Option[String] = Some(value = "John Doe")
```

## Cursors

In practice, it is normally more convenient to use cursors to navigate the JSON AST.

```scala

val json: Either[String, Json]    = """{"name": "John Doe"}""".fromJson[Json]
// json: Either[String, Json] = Right(
//   value = Obj(fields = IndexedSeq(("name", Str(value = "John Doe"))))
// )
val cursor: JsonCursor[Json, Str] = JsonCursor.field("name").isString
// cursor: JsonCursor[Json, Str] = FilterType(
//   parent = DownField(
//     parent = FilterType(parent = Identity, jsonType = Obj),
//     name = "name"
//   ),
//   jsonType = Str
// )
val name: Either[String, String]  = json.flatMap(_.get(cursor).map(_.value))
// name: Either[String, String] = Right(value = "John Doe")
```

Cursors can be composed to navigate more complex JSON structures.

```scala

val json1: Either[String, Json] = """{"posts": [{"id": 0, "title": "foo"}]}""".fromJson[Json]
// json1: Either[String, Json] = Right(
//   value = Obj(
//     fields = IndexedSeq(
//       (
//         "posts",
//         Arr(
//           elements = IndexedSeq(
//             Obj(
//               fields = IndexedSeq(
//                 ("id", Num(value = 0)),
//                 ("title", Str(value = "foo"))
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )
val json2: Either[String, Json] = """{"userPosts": [{"id": 1, "title": "bar"}]}""".fromJson[Json]
// json2: Either[String, Json] = Right(
//   value = Obj(
//     fields = IndexedSeq(
//       (
//         "userPosts",
//         Arr(
//           elements = IndexedSeq(
//             Obj(
//               fields = IndexedSeq(
//                 ("id", Num(value = 1)),
//                 ("title", Str(value = "bar"))
//               )
//             )
//           )
//         )
//       )
//     )
//   )
// )

val commonCursor = 
  JsonCursor.isArray >>> 
    JsonCursor.element(0) >>> 
    JsonCursor.isObject >>> 
    JsonCursor.field("title") >>> 
    JsonCursor.isString
// commonCursor: JsonCursor[Json, Str] = FilterType(
//   parent = DownField(
//     parent = FilterType(
//       parent = FilterType(
//         parent = DownElement(
//           parent = FilterType(
//             parent = FilterType(parent = Identity, jsonType = Arr),
//             jsonType = Arr
//           ),
//           index = 0
//         ),
//         jsonType = Obj
//       ),
//       jsonType = Obj
//     ),
//     name = "title"
//   ),
//   jsonType = Str
// )

val cursor1 = JsonCursor.field("posts")
// cursor1: JsonCursor[Json.Obj, Json] = DownField(
//   parent = FilterType(parent = Identity, jsonType = Obj),
//   name = "posts"
// )
val cursor2 = JsonCursor.field("userPosts")
// cursor2: JsonCursor[Json.Obj, Json] = DownField(
//   parent = FilterType(parent = Identity, jsonType = Obj),
//   name = "userPosts"
// )

def getTitle(json: Either[String, Json]) =
  for {
    ast   <- json
    posts <- ast.get(cursor1).orElse(ast.get(cursor2))
    title <- posts.get(commonCursor).map(_.value)
  } yield title

val title1 = getTitle(json1)
// title1: Either[String, String] = Right(value = "foo")
val title2 = getTitle(json2)
// title2: Either[String, String] = Right(value = "bar")
```

---

## Encoding

## Automatic Derivation

Assume we want to encode this case class

```scala
case class Banana(curvature: Double)
```

To produce JSON from our data we define a `JsonEncoder` like this:

```scala

object Banana {
  implicit val encoder: JsonEncoder[Banana] =
    DeriveJsonEncoder.gen[Banana]
}
```

```scala
Banana(0.5).toJson
// res0: String = "{\"curvature\":0.5}"
```

### ADTs

Say we extend our data model to include more data types

```scala
sealed trait Fruit

case class Banana(curvature: Double) extends Fruit
case class Apple (poison: Boolean)   extends Fruit
```

we can generate the encoder for the entire `sealed` family

```scala

object Fruit {
  implicit val encoder: JsonEncoder[Fruit] =
    DeriveJsonEncoder.gen[Fruit]
}
```

```scala
val apple: Fruit = Apple(poison = false)
// apple: Fruit = Apple(poison = false)
apple.toJson
// res2: String = "{\"Apple\":{\"poison\":false}}"
```

Almost all of the standard library data types are supported as fields on the case class, and it is easy to add support if one is missing.

### Sealed families and enums for Scala 3
Sealed families where all members are only objects, or a Scala 3 enum with all cases parameterless are interpreted as enumerations and will encode 1:1 with their value-names.
```scala
enum Foo derives JsonEncoder:
  case Bar
  case Baz
  case Qux
```
or
```scala
sealed trait Foo derives JsonEncoder
object Foo:
  case object Bar extends Foo
  case object Baz extends Foo
  case object Qux extends Foo
```

## Manual instances

Sometimes it is easier to reuse an existing `JsonEncoder` rather than generate a new one. This can be accomplished using convenience methods on the `JsonEncoder` typeclass to *derive* new decoders:

```scala
trait JsonEncoder[A] {
  def contramap[B](f: B => A): JsonEncoder[B]
  ...
}
```

### `.contramap`

We can use `contramap` from an already existing encoder:

```scala

case class FruitCount(value: Int)

object FruitCount {
  implicit val encoder: JsonEncoder[FruitCount] =
    JsonEncoder[Int].contramap(_.value)
}

FruitCount(3).toJson
// res3: String = "3"
```

---

## Getting Started with ZIO Json

[ZIO Json](https://github.com/zio/zio-json) is a fast and secure JSON library with tight ZIO integration.

@PROJECT_BADGES@

## Introduction

The goal of this project is to create the best all-round JSON library for Scala:

- **Performance** to handle more requests per second than the incumbents, i.e. reduced operational costs.
- **Security** to mitigate against adversarial JSON payloads that threaten the capacity of the server.
- **Fast Compilation** no shapeless, no type astronautics.
- **Future-Proof**, prepared for Scala 3 and next-generation Java.
- **Simple** small codebase, concise documentation that covers everything.
- **Helpful errors** are readable by humans and machines.
- **ZIO Integration** so nothing more is required.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-json" % "@VERSION@"
```

For cross-platform projects with Scala.js and Scala Native need to replace `%%` operator by `%%%`, 
and optionally when using `java.time.ZoneId` and `java.time.ZonedDateTime` types need to add 
the dependency on the latest version of Timezone DB:

```scala
libraryDependencies += "io.github.cquiroz" %%% "scala-java-time-tzdb" % "latest.integration"
```

## Example

Let's try a simple example of encoding and decoding JSON using ZIO JSON.

All the following code snippets assume that the following imports have been declared

<Tabs groupId="language">
  <TabItem value="scala 2" label="Scala 2">

```scala

```

  </TabItem>
  <TabItem value="scala 3" label="Scala 3" default>

```scala

```

  </TabItem>
</Tabs>

Say we want to be able to read some JSON like

```json
{"curvature":0.5}
```

into a Scala `case class`

```scala
final case class Banana(curvature: Double)
```

<Tabs groupId="language">
  <TabItem value="scala 2" label="Scala 2">

To do this, we create an *instance* of the `JsonDecoder` typeclass for `Banana` using the `zio-json` code generator. It is best practice to put it on the companion of `Banana`, like so

```scala
object Banana {
  implicit val decoder: JsonDecoder[Banana] = DeriveJsonDecoder.gen[Banana]
}
```

  </TabItem>
  <TabItem value="scala 3" label="Scala 3" default>

To do this, we derive an *instance* of the `JsonDecoder` typeclass for `Banana`.

```scala
final case class Banana(curvature: Double) derives JsonDecoder
```

Note: If your case class is defining default parameters, -Yretain-trees needs to be added to scalacOptions.

  </TabItem>
</Tabs>

Now we can parse JSON into our object

```
scala> """{"curvature":0.5}""".fromJson[Banana]
val res: Either[String, Banana] = Right(Banana(0.5))
```

Likewise, to produce JSON from our data we derive a `JsonEncoder`

<Tabs groupId="language">
  <TabItem value="scala 2" label="Scala 2">

```scala
object Banana {
  ...
  implicit val encoder: JsonEncoder[Banana] = DeriveJsonEncoder.gen[Banana]
}
```

  </TabItem>
  <TabItem value="scala 3" label="Scala 3" default>

```scala
final case class Banana(curvature: Double) derives JsonEncoder
```

  </TabItem>
</Tabs>

```
scala> Banana(0.5).toJson
val res: String = {"curvature":0.5}

scala> Banana(0.5).toJsonPretty
val res: String =
{
  "curvature" : 0.5
}
```

And bad JSON will produce an error in `jq` syntax with an additional piece of contextual information (in parentheses)

```
scala> """{"curvature": womp}""".fromJson[Banana]
val res: Either[String, Banana] = Left(.curvature(expected a Double))
```

Say we extend our data model to include more data types

<Tabs groupId="language">
  <TabItem value="scala 2" label="Scala 2">

```scala
sealed trait Fruit
final case class Banana(curvature: Double) extends Fruit
final case class Apple (poison: Boolean)   extends Fruit
```

  </TabItem>
  <TabItem value="scala 3" label="Scala 3" default>

```scala
enum Fruit {
  case Banana(curvature: Double)
  case Apple (poison: Boolean)
}
```

  </TabItem>
</Tabs>

we can generate the encoder and decoder for the entire `sealed` family

<Tabs groupId="language">
  <TabItem value="scala 2" label="Scala 2">

```scala
object Fruit {
  implicit val decoder: JsonDecoder[Fruit] = DeriveJsonDecoder.gen[Fruit]
  implicit val encoder: JsonEncoder[Fruit] = DeriveJsonEncoder.gen[Fruit]
}
```

  </TabItem>
  <TabItem value="scala 3" label="Scala 3" default>

```scala
enum Fruit derives JsonCodec {
  case Banana(curvature: Double)
  case Apple (poison: Boolean)
}
```

  </TabItem>
</Tabs>

allowing us to load the fruit based on a single field type tag in the JSON

```
scala> """{"Banana":{"curvature":0.5}}""".fromJson[Fruit]
val res: Either[String, Fruit] = Right(Banana(0.5))

scala> """{"Apple":{"poison":false}}""".fromJson[Fruit]
val res: Either[String, Fruit] = Right(Apple(false))
```

Almost all of the standard library data types are supported as fields on the case class, and it is easy to add support if one is missing.

<Tabs groupId="language">
  <TabItem value="scala 2" label="Scala 2">

```scala mdoc:compile-only

sealed trait Fruit                   extends Product with Serializable
case class Banana(curvature: Double) extends Fruit
case class Apple(poison: Boolean)    extends Fruit

object Fruit {
  implicit val decoder: JsonDecoder[Fruit] =
    DeriveJsonDecoder.gen[Fruit]

  implicit val encoder: JsonEncoder[Fruit] =
    DeriveJsonEncoder.gen[Fruit]
}

val json1         = """{ "Banana":{ "curvature":0.5 }}"""
val json2         = """{ "Apple": { "poison": false }}"""
val malformedJson = """{ "Banana":{ "curvature": true }}"""

json1.fromJson[Fruit]
json2.fromJson[Fruit]
malformedJson.fromJson[Fruit]

List(Apple(false), Banana(0.4)).toJsonPretty
```

  </TabItem>
  <TabItem value="scala 3" label="Scala 3" default>

```scala

enum Fruit derives JsonCodec {
  case Banana(curvature: Double)
  case Apple(poison: Boolean)
}

export Fruit.*

val json1         = """{ "Banana":{ "curvature":0.5 }}"""
val json2         = """{ "Apple": { "poison": false }}"""
val malformedJson = """{ "Banana":{ "curvature": true }}"""

json1.fromJson[Fruit]
json2.fromJson[Fruit]
malformedJson.fromJson[Fruit]

List(Apple(false), Banana(0.4)).toJsonPretty
```

  </TabItem>
</Tabs>

# How

Extreme **performance** is achieved by decoding JSON directly from the input source into business objects (inspired by [plokhotnyuk](https://github.com/plokhotnyuk/jsoniter-scala)). Although not a requirement, the latest advances in [Java Loom](https://wiki.openjdk.java.net/display/loom/Main) can be used to support arbitrarily large payloads with near-zero overhead.

Best in class **security** is achieved with an aggressive *early exit* strategy that avoids costly stack traces, even when parsing malformed numbers. Malicious (and badly formed) payloads are rejected before finishing reading.

**Fast compilation** and **future-proofing** is possible thanks to [Magnolia](https://propensive.com/opensource/magnolia/) which allows us to generate boilerplate in a way that will survive the exodus to Scala 3. `zio-json` is internally implemented using a [`java.io.Reader`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/io/Reader.html) / [`java.io.Writer`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/io/Writer.html)-like interface, which is making a comeback to center stage in Loom.

**Simplicity** is achieved by using well-known software patterns and avoiding bloat. The only requirement to use this library is to know about Scala's encoding of typeclasses, described in [Functional Programming for Mortals](https://leanpub.com/fpmortals/read#leanpub-auto-functionality).

**Helpful errors** are produced in the form of a [`jq`](https://stedolan.github.io/jq/) query, with a note about what went wrong, pointing to the exact part of the payload that failed to parse.

---

## Akka Http Interop

An interop module is provided for [Akka Http 10.2.4](https://doc.akka.io/docs/akka-http/10.2.4/index.html)

## Installation

```scala
libraryDependencies ++= Seq(
  "de.heikoseeberger" %% "akka-http-zio-json" % "1.36.0",
)
```

## Usage

```scala

final case class Greeting(greeting: String)

object Greeting {

  implicit val decoder: JsonDecoder[Greeting] =
    DeriveJsonDecoder.gen[Greeting]

  implicit val encoder: JsonEncoder[Greeting] =
    DeriveJsonEncoder.gen[Greeting]

  //    automatically derived from the zio-json codecs

  //    implicit val akkaMarshaller: ToEntityMarshaller[Greeting] =
  //      implicitly[ToEntityMarshaller[Greeting]]
  //
  //    implicit val akkaUnmarshaller: FromEntityUnmarshaller[Greeting] =
  //      implicitly[FromEntityUnmarshaller[Greeting]]

  implicit val system = ActorSystem()

  // under the hood, Akka Http uses marshallers and unmarshallers for Json http entities

  def marshallGreeting(greeting: Greeting): Task[HttpEntity] =
    ZIO.fromFuture(implicit ec => Marshal(greeting).to[HttpEntity])

  def unmarshallGreeting(entity: HttpEntity): Task[Greeting] =
    ZIO.fromFuture(implicit ec => Unmarshal(entity).to[Greeting])
}
```

## Example project

A fully working example [can be found here](https://github.com/hseeberger/akka-http-json/blob/master/akka-http-zio-json/src/test/scala/de/heikoseeberger/akkahttpziojson/ExampleApp.scala).

---

## HTTP4s Interop

An interop module is provided for [HTTP4s v0.21](https://http4s.org/v0.21)

## Installation

```scala
libraryDependencies ++= Seq(
  "dev.zio" % "zio-json-interop-http4s" % "0.8.0"
)
```

## Usage

```scala

final case class Greeting(greeting: String)

object Greeting {
  // This JsonCodec[A] will be picked up by the `jsonEncoderOf` method provided by zio-json-interop-http4s
  implicit val encoder: JsonCodec[Greeting] =
    DeriveJsonCodec.gen

  implicit def entityEncoder[F[_]: Applicative]: EntityEncoder[F, Greeting] =
    jsonEncoderOf[F, Greeting]
}
```

## Example project

A fully working example project [can be found here](https://github.com/zio/zio-json/tree/develop/examples/interop-http4s)

---

## Interop modules

Integrations are provided several popular libraries, which are published as separate artifacts:

* [Akka Http](akka-http.md)
* [HTTP4s](http4s.md)
* [Refined](refined.md)
* [Scalaz 7.x](scalaz-7x.md)

---

## Refined Interop

## Installation

```scala
libraryDependencies ++= Seq(
  "dev.zio" % "zio-json-interop-refined" % "0.8.0"
)
```

## Usage

```scala

case class Person(name: String Refined NonEmpty)

object Person {
  implicit val decoder: JsonDecoder[Person] = DeriveJsonDecoder.gen
}
```

```scala
"""{ "name": "" }""".fromJson[Person]
// res0: Either[String, Person] = Left(
//   value = ".name(Predicate isEmpty() did not fail.)"
// )
```

```scala
"""{ "name": "Aurora" }""".fromJson[Person]
// res1: Either[String, Person] = Right(value = Person(name = Aurora))
```

---

## Scalaz 7.x Interop

## Installation

```scala
libraryDependencies ++= Seq(
  "dev.zio" % "zio-json-interop-scalaz" % "0.8.0"
)
```

## Usage

```scala

IList(1, 2, 3).toJson
// res0: String = "[1,2,3]"
```

---

## Manual Instances

Sometimes it is easier to reuse an existing `JsonDecoder` rather than generate a new one. This can be accomplished using convenience methods on the `JsonDecoder` typeclass to *derive* new decoders:

```scala
trait JsonDecoder[A] {
  def map[B](f: A => B): JsonDecoder[B]
  def mapOrFail[B](f: A => Either[String, B]): JsonDecoder[B]
  ...
}
```

Similarly, we can reuse an existing `JsonEncoder`

```scala
trait JsonEncoder[A] {
  def contramap[B](f: B => A): JsonEncoder[B]
  ...
}
```

### `.map`

We can `.map` from another `JsonDecoder` in cases where the conversion will always succeed. This is very useful if we have a `case class` that simply wraps another thing and shares the same expected JSON.

For example, say we want to model the count of fruit with a `case class` to provide us with additional type safety in our business logic (this pattern is known as a *newtype*).

```scala
case class FruitCount(value: Int)
```

but this would cause us to expect JSON of the form

```json
{"value":1}
```

wheres we really expect the raw number. We can derive a decoder from `JsonDecoder[Int]` and `.map` the result into a `FruitCount`

```scala
object FruitCount {
  implicit val decoder: JsonDecoder[FruitCount] = JsonDecoder[Int].map(FruitCount(_))
}
```

and now the `JsonDecoder` for `FruitCount` just expects a raw `Int`.

Every time we use a `.map` to create a `JsonDecoder` we can usually create a `JsonEncoder` with `.contramap`

```scala
object FruitCount {
  ...
  implicit val encoder: JsonEncoder[FruitCount] = JsonEncoder[Int].contramap(_.value)
}
```

Another use case is if we want to encode a `case class` as an array of values, rather than an object with named fields. Such an encoding is very efficient because the messages are smaller and require less processing, but are very strict schemas that cannot be upgraded.

```scala
case class Things(s: String, i: Int, b: Boolean)
object Things {
  implicit val decoder: JsonDecoder[Things] =
    JsonDecoder[(String, Int, Boolean)].map { case (p1, p2, p3) => Things(p1, p2, p3) }
}
```

which parses the following JSON

```json
["hello",1,true]
```

### `.mapOrFail`

We can use `.mapOrFail` to take the result of another `JsonDecoder` and try to convert it into our custom data type, failing with a message if there is an error.

Say we are using the [`refined`](https://github.com/fthomas/refined) library to ensure that a `Person` data type only holds a non-empty string in its `name` field

```scala

case class Person(name: String Refined NonEmpty)

object Person {
  implicit val decoder: JsonDecoder[Person] = DeriveJsonDecoder.gen
}
```

we will get a compiletime error because there is no `JsonDecoder[String Refined NonEmpty]`.

However, we can derive one by requesting the `JsonDecoder[String]` and calling `.mapOrFail`, supplying the constructor for our special `String Refined NonEmpty` type

```scala
implicit val decodeName: JsonDecoder[String Refined NonEmpty] =
  JsonDecoder[String].mapOrFail(refined.refineV[NonEmpty](_))
```

Now the code compiles.

In fact, we do not need to provide `decodeName` for each `Refined` data type; `zio-json` comes with support out of the box, see the Integrations section below.

---

## Performance(Zio-json)

The following benchmarks are freely available to run on your hardware with `sbt "zioJsonJVM/jmh:run -prof gc"` and can be extended to include more niche libraries. We only compare `zio-json` against Circe and Play as they are the incumbent solutions used by most of the Scala ecosystem.

`zio-json`, when used in legacy mode (i.e. using a `StringReader`), is typically x2 faster than Circe and x5 faster than Play. When used with Loom, `zio-json` has finished its work before the others even begin. The following benchmarks are therefore only for legacy mode comparisons.

There are two main factors to consider when comparing the performance of JSON libraries: memory usage and operations per second. We perform measurements in one thread at a time but in a real server situation, there are multiple threads each consuming resources.

Here are JMH benchmarks (higher `ops/sec` is better, lower `MB/sec` is better) on a standard Google Maps API performance-testing dataset (stressing array and number parsing). Note that a better metric for memory usage might be `MB` per decode or encode, since it can be misleading to have the same `MB/sec` but be processing more JSON: the library that consumes the least amount of memory is likely to have highest throughput.

<!-- zioJsonJVM/jmh:run -prof gc GoogleMaps.*Success1 -->
<!-- zioJsonJVM/jmh:run -prof gc GoogleMaps.*encode* -->

```
       Decoding                    | Encoding
       ops/sec       MB/sec        | ops/sec      MB/sec
zio    15761 ± 283   1633 ± 29     | 14289 ±  84  2214 ± 12
circe   8832 ± 269   1816 ± 55     | 11980 ± 142  2030 ± 24
play    5756 ±  47   2260 ± 19     |  6669 ± 160  2677 ± 64
```

on a standard Twitter API performance-testing dataset (stressing nested case classes with lots of fields)

<!-- zioJsonJVM/jmh:run -prof gc Twitter.*Success1 -->
<!-- zioJsonJVM/jmh:run -prof gc Twitter.*encode* -->

```
       Decoding                    | Encoding
       ops/sec       MB/sec        | ops/sec      MB/sec
zio    16989 ± 113    827 ±  6     | 23085 ± 641  1791 ± 50
circe  16010 ±  72   1349 ±  6     | 15664 ± 209  1627 ± 22
play    5256 ± 165   1231 ± 39     | 15580 ± 314  2260 ± 45
```

on a standard GeoJSON performance-testing dataset (stressing nested sealed traits that use a discriminator)

<!-- zioJsonJVM/jmh:run -prof gc GeoJSON.*Success1 -->
<!-- zioJsonJVM/jmh:run -prof gc GeoJSON.*encode* -->

```
       Decoding                    | Encoding
       ops/sec       MB/sec        | ops/sec       MB/sec
zio    17104 ± 155   2768 ± 25     | 5372 ± 26      817 ±  4
circe   8388 ± 118   2879 ± 41     | 4762 ± 47      592 ±  6
play     704 ±   9   3946 ± 55     | 2587 ± 24     1091 ± 10
```

and on a standard synthetic performance-testing dataset (stressing nested recursive types)

<!-- zioJsonJVM/jmh:run -prof gc Synthetic.*Success -->
<!-- zioJsonJVM/jmh:run -prof gc Synthetic.*encode* -->

```
       Decoding                    | Encoding
       ops/sec       MB/sec        | ops/sec       MB/sec
zio    59099 ± 1307  2108 ± 46     | 32048 ±  240  2573 ± 19
circe  19609 ±  370  2873 ± 53     | 13830 ±  109  1730 ± 14
play    9001 ±  182  3348 ± 67     | 14529 ±  200  3533 ± 48
```

`zio-json` easily wins every benchmark except ops/sec for the Twitter test data where Circe matches ops/sec but loses heavily on memory usage. Play loses on every benchmark.

## Even More Performance

If `zio-json` isn't fast enough for you, then try out [jsoniter-scala](https://github.com/plokhotnyuk/jsoniter-scala); whereas `zio-json` is fully integrated into ZIO, including streams and pipeline support, jsoniter is library agnostic.

JSON is an inefficient transport format and everybody would benefit from a port of this library to msgpack or protobuf. For legacy services, a port supporting XML is also be possible.

---

## Security

A [Denial of Service](https://en.wikipedia.org/wiki/Denial-of-service_attack) (DOS) attack is a cyber-attack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services. The vast majority of public-facing servers written in Scala are vulnerable to DOS attack.

Attacks that are in the form of a valid payload are designed to be stealthy and will produce the same end-result as a legitimate payload, but will consume more resources along the way. In this section we investigate specific attacks and how `zio-json` mitigates against them.

### Resource Attack: Larger Payload

An obvious way to slow down a server is to give it more data to read. JSON is particularly susceptible to this kind of attack because it has an in-built mechanism to expand the size of the message without altering the contents: pad with whitespace.

Many web frameworks will fully consume the contents of a payload into a `String` before handing it off to the JSON library, so if we receive a JSON message consisting of 1GB of whitespace, we will consume 1GB of heap on that server.

The best way to mitigate against message size attacks is to cap the `Content-Length` to a reasonable size for the use case. A further mitigation is to use a streaming parser that does not require the entire message to be read into memory before parsing begins.

For all the remaining attacks, we will cap the malicious message size to 100KB (the original message is 25KB) and compare the attacks against this baseline. The benchmark results for the original (unedited) payload are given in parentheses, and we can immediately see a reduction in the ops/sec for all frameworks, accompanied by a reduction in memory usage.

<!-- zioJsonJVM/jmh:run -prof gc GoogleMaps.*Attack0 -->

```
       ops/sec        MB/sec
zio    10104 (14823)  1047 (1537)
circe   7456 ( 8832)  1533 (1816)
play    3589 ( 5756)  1344 (2260)
```

### Redundant Data

Most JSON libraries (but not `zio-json`) first create a representation of the JSON message in an Abstract Syntax Tree (AST) that represents all the objects, arrays and values in a generic way. Their decoders typically read what they need from the AST.

An intermediate AST enables attack vectors that insert redundant data, for example in our Google Maps dataset we can add a new field called `redundant` at top-level containing a 60K `String`. If we do this, and run the benchmarks, we see that Circe is heavily impacted, with a 75% reduction in capacity and an increase in memory usage. Play is also impacted, although not as severely. `zio-json`'s ops/sec are reduced but the memory usage is in line which means that throughput is unlikely to be affected by this kind of attack.

<!-- zioJsonJVM/jmh:run -prof gc GoogleMaps.*Attack1 -->

```
       ops/sec       MB/sec
zio    5999 (10104)   622 (1047)
circe  2224 ( 7456)  1655 (1533)
play   2350 ( 3589)  1854 (1344)
```

The reason why `zio-json` is not as badly affected is because it skips values that are unexpected. We can completely mitigate this kind of attack by using the `@jsonNoExtraFields` annotation which results in the payload being rejected at a rate of 5.5 million ops/sec.

Other kinds of redundant values attacks are also possible, such as using an array of 60K full of high precision decimal numbers that require slow parsing (also known as ["near halfway numbers"](https://www.exploringbinary.com/17-digits-gets-you-there-once-youve-found-your-way/)), attacking the CPU. However, the memory attack afforded to us by a redundant `String` is already quite effective.

### `hashCode` Collisions

Following on from the redundant data attack, we can place redundant data in the location of object fields.

JSON libraries that use an intermediate AST often store JSON objects as a stringy `HashMap` (circe uses a [`java.util.LinkedHashMap`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/util/LinkedHashMap.html)). If we insert redundant fields that have hashcode collisions with legitimate fields, we can successfully attack both memory and CPU. We need to know the hashing algorithm that is being used, which often falls down to some version of the default Java `String.hashCode` [which is very easy to exploit](https://github.com/plokhotnyuk/jsoniter-scala/blob/master/jsoniter-scala-benchmark/shared/src/main/scala/com/github/plokhotnyuk/jsoniter_scala/benchmark/HashCodeCollider.scala).

In this malicious payload, we add redundant fields that have hashcode collisions, up to 4 collisions per field; we could add more if we used a bruteforce search.

<!-- zioJsonJVM/jmh:run -prof gc GoogleMaps.*Attack2 -->

Again, `zio-json` completely mitigates this attack if the `@jsonNoExtraFields` annotation is used. Note that even if Circe and Play rejected payloads of this nature, it would be too late because the attack happens at the AST layer, not the decoders. However, for the sake of comparison, let's turn off the `zio-json` mitigation:

```
       ops/sec       MB/sec
zio    3742 (10104)   695 (1047)
circe  1992 ( 7456)  1162 (1533)
play   1312 ( 3589)  1636 (1344)
```

ops/sec is down for all decoders relative to the baseline, but since `zio-json` and Circe memory usage is also reduced the throughput on a server might not be impacted as badly as it sounds.

However, this attack hurts Play very badly; memory usage is up compared to the baseline with throughput reduced to 40% of the baseline (22% of the original).

There is a variant of this attack that can be devastating for libraries that rely on `HashMap`. In this attack, [developed by plokhotnyuk to DOS ujson](https://github.com/plokhotnyuk/jsoniter-scala/pull/325), an object is filled with many fields that have a `hashCode` of zero. This exploits two facts:

- Java `String` does not cache `hashCode` of zero, recomputing every time it is requested
- many `HashMap` implementations re-request the hashes of all objects as the number of entries increases during construction.

### Death by a Thousand Zeros

Another kind of attack is to provide data that will cause the decoder for a specific value to do more work than it needs to. Numbers are always a great example of this.

The most brutal attack of this nature is to trick a deserialization library into constructing a gigantic number as a `BigDecimal` and then to downcast to a `BigInteger`. The JVM will happily attempt to reserve GBs of heap for the conversion. Try this in your REPL:

```scala
new java.math.BigDecimal("1e214748364").toBigInteger
```

Fortunately, this kind of attack is prevented by all the mainstream libraries. But it is possible to perform a much weaker form of the attack on Circe, which (to its credit) goes to great effort to pre-validate numbers.

The Google Maps schema has fields of type `Int` and Circe supports the conversion of floating point numbers to integers if the fractional part is zero: so we can pad an integer with as many zeros as possible.

```
       ops/sec       MB/sec
circe  4529 ( 7456)  2037 (1533)
```

This attack is very effective in schemas with lots of numbers, causing ops/sec to be halved with a 33% increase in memory usage.

`zio-json` is resistant to a wide range of number based attacks because it uses a from-scratch number parser that will exit early when the number of bits of any number exceeds 256 bits.

---

## Avoiding chunk-breakers

## A warning about `mapZIO` and other chunk-breakers

ZIO streams are not just a simple sequence of elements. The elements are grouped into chunks, which makes many
operations faster. Zio-kafka helps with this by guaranteeing that all records (for a given partition), fetched together
from the broker, are in the same chunk.

Be careful when using `mapZIO`, `tap` and some other stream operators that break the chunking structure of the stream
(or more precisely, the resulting stream has chunks with a single element). The throughput may be significantly lower
than with the original chunking structure intact.

Chunk-breaking operators can be found by looking at their scaladocs. Starting with zio-streams 2.1.17, these scaladocs
contain the words `This combinator destroys the chunking structure`.

You can regain full throughput by processing all elements in a chunk together in one go. However, there is a catch: the
order of processing changes. For example, given a stream with elements `a` and `b` in the same chunk, for
`stream.mapZIO(f).mapZIO(g)` the evaluation order is `f(a), g(a), f(b), g(b)`. For the alternatives listed below the
evaluation order changes to `f(a), f(b), g(a), g(b)`. Now imagine that `g(a)` fails, with `mapZIO`, `f(b)` is not
executed, but with the alternatives it _is_ executed. It is up to you to decide if this is a problem.

### Use `mapZIOChunked`

_Available since zio-streams 2.1.17._

The simplest alternative is the stream operator `mapZIOChunked`; it has the same signature as `mapZIO` .

```scala
def f(a: A): ZIO[R, E, B]

stream               // ZStream[R, E, A]
  .mapZIOChunked(f)  // ZStream[R, E, B]
```

### Use `chunksWith`

If you have a single processing step that needs to work on a chunk, you can use `chunksWith`.

```scala
def f(a: A): ZIO[R, E, B]

stream             // ZStream[R, E, A]
  .chunksWith { chunkStream =>
    chunkStream.mapZIO(chunk => ZIO.foreach(chunk)(f))
  }                // ZStream[R, E, B]
```

### Expose chunking structure with `chunks`

Use `chunks` when you have multiple processing steps that can all work on a chunk at a time. Since `chunks` exposes the
chunking structure explicitly, the program can no longer accidentally break the chunking structure (unless
`flattenChunks` is also used).

```scala
def f(a: A): ZIO[R, E, B]
def g(b: B): ZIO[R, E, C]

stream                                         // ZStream[R, E, A]
  .chunks                                      // ZStream[R, E, Chunk[A]]
  .mapZIO { chunk => ZIO.foreach(chunk)(f) }   // ZStream[R, E, Chunk[B]]
  .mapZIO { chunk => ZIO.foreach(chunk)(g) }   // ZStream[R, E, Chunk[C]]
  .flattenChunks                               // ZStream[R, E, C]
```

### Side effects per chunk with `tapChunks`

_Available since zio-streams 2.1.17._

Unlike `tap`, the `tapChunks` stream operator preserves the chunking structure and allows side effects per chunk.

```scala
stream                                                     // ZStream[R, E, A]
  .tapChunks(c => ZIO.logInfo(s"Chunk of size ${c.size}")) // ZStream[R, E, A]
```

### Avoid `rechunk`

ZStream also provides the `rechunk` operator. This operator rebuilds the chunking structure by grouping elements into
chunks of fixed size. If there are not enough elements available for a chunk, the operator waits for these elements to
arrive. Only when the stream ends a smaller chunk is emitted.

In low-volume kafka applications, waiting for enough elements to arrive can take a long time! In high-volume kafka
applications, this operator causes data from multiple polls to be processed together. This results in increased
processing latency and memory usage.

Therefore, we recommend that you avoid `rechunk` in kafka applications.

---

## Tuning the consumer

Zio-kafka's consumer can be tuned for:

* throughput
* latency
* memory usage
* long-running processes / slow consumers

All tuning is done via `ConsumerSettings`:

```scala
val settings = ConsumerSettings(bootstrapServers)
  .withGroupId(groupId)
  .withProperties(properties)
  .... etc.
```

Read on for more details.

## Names

Two very similarly named settings are relevant. Don't mix them up:

* `pollTimeout` — how long a poll may take
* `max.poll.interval.ms` — the maximum time between polls

## Throughput and latency

The kafka client can be tuned for either high throughput or low latency, unfortunately not both.
The most important settings for tuning throughput and latency are:

* zio-kafka's `pollTimeout` — This is the maximum time to block while polling the Kafka consumer. Zio-kafka's default
  is `50ms` which is good for low latency applications. Set this higher, e.g. `500ms` for better throughput.
* kafka's [configuration `max.poll.records`](https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) — The maximum number of records a poll will return. Kafka defaults
  this to `500`. You can set this higher for more throughput, or lower for lower latency.
* zio-kafka's fetch-strategy `partitionPreFetchBufferLimit` — when the number of records in a partition queue is
  at or below this value, zio-kafka will start to pre-fetch and buffer more records from Kafka. The default value for
  this parameter is `1024`; 2 * the default `max.poll.records` of 500, rounded to the nearest power of 2.

Zio-kafka provides 2 methods that set these settings for 2 common use cases: `ConsumerSettings.tuneForHighThroughput`
and `ConsumerSettings.tuneForLowLatency`.
Note that their implementation may vary over time. You can use them as follows:

```scala
val highThroughputSettings = ConsumerSettings(bootstrapServers).tuneForHighThroughput
val lowLatencySettings = ConsumerSettings(bootstrapServers).tuneForLowLatency
```

## Small and large records

Kafka’s performance is not very sensitive to record size. However, when records become very small (< 100 bytes) it
might be beneficial to increase `max.poll.records` and `partitionPreFetchBufferLimit`. Similarly, when records are
very large (> 100Kb), `max.poll.records` can be decreased. Also, pre-fetching can be limited by decreasing
`partitionPreFetchBufferLimit` or even disabled by using `ConsumerSettngs.withoutPartitionPreFetching`.

## High number of partitions

When a lot of partitions need to be consumed, we need to take into account that heap is needed to store the records in
the partition queues. A very rough estimate for the maximum amount of heap needed is given by: `average record size` *
`number of of partitions` * max(`partitionPreFetchBufferLimit`, `max.poll.records`).

The total can be tuned by changing the `partitionPreFetchBufferLimit`, `max.poll.records` settings.

Another option is to write a custom `FetchStrategy`. For example the `ManyPartitionsQueueSizeBasedFetchStrategy` in
[draft PR 970](https://github.com/zio/zio-kafka/pull/970) (merged into zio-kafka since 2.8.1).

## Long-running processes / slow consumers

To detect stalled consumers, Kafka revokes a partition when a consumer does not poll within the max poll interval (see
[configuration `max.poll.interval.ms`](https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms)). The default max poll interval is 5 minutes. After a partition is revoked,
it will be assigned to another consumer.

In zio-kafka (versions 2.5.1+) a stream needs to pull data within the max poll interval. If this doesn't happen, the
stream is interrupted with a failure and the whole consumer shuts down.

To prevent the above failure states, either decrease `max.poll.records` (so that time between polling gets shorter),
and/or increase `max.poll.interval.ms` (so that the time between polls may be longer).

To see if your application must be configured with a lower `max.poll.records` and/or a higher `max.poll.interval.ms`
value we need to consider the maximum duration between polls. If processing is sequential, we can obtain this maximum by
multiplying `max.poll.records` with the maximum duration to process a single record. To also accommodate things like
long garbage collections and buffering, configuration `max.poll.interval.ms` should be substantially higher than the
maximum processing time.

`max.poll.records` and `max.poll.interval.ms` can be set with:

```scala
  .withMaxPollRecords(1)
  .withMaxPollInterval(15.minutes)
```

On older zio-kafka versions `withMaxPollInterval` is not available. Use the following instead:

```scala
  .withProperty("max.poll.interval.ms", 15.minutes.toMillis.toString)
```

⚠️In zio-kafka versions 2.2 up to 2.5.0 it may also be necessary to increase the `runloopTimeout` setting.
When no stream is processing data for this amount of time (while new data is available), the consumer will halt with a
failure. In zio-kafka 2.5.0 `runloopTimeout` defaults to 4 minutes, a little bit lower than `max.poll.interval.ms`.

### Long-running processes and `rebalance-safe-commits`

When `rebalance-safe-commits` is enabled, additional settings are needed for long-running processes. See
[preventing duplicates](preventing-duplicates.md) for more details.

## Using metrics to tune the consumer

Zio-Kafka exposes [metrics](metrics.md) that can be used to further tune the consumer. To interpret these metrics you need to know how zio-kafka works internally.

![](consumer-internals.svg)

The runloop is at the heart of every zio-kafka consumer.
It creates a zstream for each partition, eventually this is the zstream your applications consumes from.
When the zstream starts, and every time the records queue is empty, it sends a request for data to the runloop.
The request causes the runloop to resume the partition so that the next poll may receive records.
Any received records are put in the records queue.
When the queue reaches a certain size (as determined by the configured `FetchStrategy`), the partition is paused.
Meanwhile, the zstream reads from the queue and emits the records to your application.

An optimally configured consumer has the following properties:

- the zstreams never have to wait for new records (to get high throughput),
- most of the time, the record queues are empty (to get low latency and low heap usage).

The following strategy can help you get to this state:

1. First make sure that `pollTimeout` and `max.poll.records` make sense for the latency and throughput requirements
   of your application.
2. Configure `partitionPreFetchBufferLimit` to `0`.
3. Observe metric `ziokafka_consumer_queue_polls` which gives the number of polls during which records are idling in
   the queue.
4. Increase `partitionPreFetchBufferLimit` in steps until most measurements of the `ziokafka_consumer_queue_polls`
   histogram are in the `0` bucket .

During this process, it is useful to observe metric `ziokafka_consumer_queue_size` (number of records in the queues) to
see if the queues are indeed increasing in size.

When many (hundreds of) partitions need to be consumed, the metric `ziokafka_consumer_all_queue_size` should also be
observed as increasing `partitionPreFetchBufferLimit` can lead to high heap usage. (See 'High number of partitions'
above.)

---

## Consuming Kafka topics using ZIO Streams

You can stream data from Kafka using the `plainStream` method:

```scala

consumer
  .plainStream(Subscription.topics("topic150"), Serde.string, Serde.string) // (1)
  .tap(cr => printLine(s"key: ${cr.record.key}, value: ${cr.record.value}")) // (2)
  .map(_.offset) // (3)
  .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis)) // (4)
  .mapZIO(_.commit) // (5)
  .runDrain
```

(1) There are several types of subscriptions: a set of topics, a topic name pattern, or fully specified set of
topic-partitions (see [subscriptions](partition-assignment-and-offset-retrieval.md)).

(2) The `plainStream` method returns a stream of `CommittableRecord` objects. In this line, we access the key
and value from the `CommittableRecord` and print them to the console.

:::caution
Some stream operators (like `tap` and `mapZIO`) break the chunking structure of the stream which heavily reduces
throughput. See [avoiding chunk-breakers](avoiding-chunk-breakers.md) for more information and alternatives.
:::

(3) Here we get the offset of the record, so we now have a stream of offsets.

(4) The offsets are aggregated into an `OffsetBatch`. The offset batch keeps the highest seen offset per partition.
ZStream operator `aggregateAsyncWithin` runs upstream in a separate fiber, allowing consuming and committing to run in
parallel.

The offset batches are emitted every 100 milliseconds. The delay should be longer than the typical commit-duration in
your system. For example, lets assume the delay is set to 2ms and a commit operation takes 10ms. Then, when some records
are consumed, 2ms later `aggregateAsyncWithin` emits an offset batch and starts collecting offsets for the next batch.
In addition, the commit operation starts. Again 2ms later `aggregateAsyncWithin` tries to emit the next batch. However,
this will block because the commit is still ongoing. Since `aggregateAsyncWithin` is now blocked, it stops accepting
offsets from upstream, halting the consumer until the commit completes.

For applications that only see a few records per second or less, you can remove line (4) and commit the offset for each
record separately. In this approach the commit executes on the same fiber as the rest of the stream. This lowers
throughput. Explicitly committing every record is therefore almost never the best approach.

To get the lowest commit latency you can commit continuously; a new commit is started as soon as the previous completes.
This is done by replacing line (4) with `.aggregateAsync(Consumer.collectOffsets)`. Be aware that registering commits is
an intensive operation for the kafka brokers. With dozens of consumers continuously committing, the high load may slow
down everything.

(5) All offsets in the `OffsetBatch` are committed. As soon as the commit is done, the next offset batch is pulled
from upstream.

## Consuming with more parallelism

To process partitions (assigned to the consumer) in parallel, you may use the `partitionedStream` method, which creates
a nested stream of partitions:

```scala

consumer
  .partitionedStream(Subscription.topics("topic150"), Serde.string, Serde.string)
  .flatMapPar(Int.MaxValue) { case (topicPartition, partitionStream) => // (1)
    ZStream.fromZIO(
      printLine(s"Starting stream for topic '${topicPartition.topic}' partition ${topicPartition.partition}")
    ) *>
      partitionStream
        .tap(record => printLine(s"key: ${record.key}, value: ${record.value}")) // (2)
        .map(_.offset) // (3)
  }
  .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis)) // (4)
  .mapZIO(_.commit)  // (5)
  .runDrain
```

In this example, each partition is processed in parallel on separate fibers, while a single fiber is doing commits.

(1) When using partitionedStream with `flatMapPar(n)`, it is recommended to set n to `Int.MaxValue`. N must be equal to
or greater than the number of partitions your consumer subscribes to otherwise there will be unhandled partitions and
Kafka eventually evicts your consumer.

Each time a new partition is assigned to the consumer, a new partition stream is created and the body of the
`flatMapPar` is executed.

(2) As before, here we process the received `CommittableRecord`.

(3) Note how the offsets from all the sub-streams are merged into a single stream of `Offset`s. This ensures only a
single stream is doing commits.

(4) and (5) The offsets are aggregated into an `OffsetBatch` and committed as in the previous section.

## Controlled shutdown (experimental)

The examples above will keep processing records forever, or until the fiber is interrupted, typically at application
shutdown. When interrupted, some records may be 'in-flight', e.g. being processed by one of the stages of your consumer
stream user code. Those records will be processed partly and their offsets may not be committed. For fast shutdown in an
at-least-once processing scenario this is fine.

zio-kafka also supports a _graceful shutdown_, where the fetching of records for the subscribed topics/partitions is
stopped, the streams are ended and all downstream stages are completed, allowing in-flight records to be fully
processed.

Use the `*StreamWithControl` variants of `plainStream`, `partitionedStream` and `partitionedAssignmentStream` for this
purpose. These methods return a `StreamControl` object allowing you to stop fetching records and terminate the execution
of the stream gracefully.

The `StreamControl` can be used with `Consumer.runWithGracefulShutdown`, which can gracefully terminate the stream upon
fiber interruption. This is useful for a controlled shutdown when your application is terminated:

```scala

ZIO.scoped {
  for {
    control <- consumer.partitionedStreamWithControl(
      Subscription.topics("topic150"),
      Serde.string,
      Serde.string
    )
    _ <- consumer.runWithGracefulShutdown(control, shutdownTimeout = 30.seconds) { stream =>
        stream.flatMapPar(Int.MaxValue) { case (topicPartition, partitionStream) =>
          partitionStream
            .tap(record => printLine(s"key: ${record.key}, value: ${record.value}"))
            .map(_.offset)
        }
        .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
        .mapZIO(_.commit)
        .runDrain
      }
  } yield ()
}
```

For more control over when to end the stream, use the `StreamControl` construct like this:

```scala

ZIO.scoped {
    for {
      control <- consumer.partitionedStreamWithControl(
          Subscription.topics("topic150"),
          Serde.string,
          Serde.string
        )
      fiber <- control.stream.flatMapPar(Int.MaxValue) { case (topicPartition, partitionStream) =>
            partitionStream
              .tap(record => printLine(s"key: ${record.key}, value: ${record.value}"))
              .map(_.offset)
          }
          .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
          .mapZIO(_.commit)
          .runDrain
          .forkScoped
      // At some point in your code, when some condition is met
      _ <- control.end // Stop fetching more records
      _ <- fiber.join // Wait for graceful completion of the stream
    } yield ()
}
```

As of zio-kafka 3.0, this functionality is experimental. If no issues are reported and the API has good usability, it
will eventually be marked as stable.

---

## Consuming, Producing and Committing Offsets

Consuming from Kafka and then producing a transformed message back to Kafka, is a regular pattern in systems that use
Kafka. This page shows an example of an application that does this. Next we discuss how throughput can be improved.

This page is also useful as a deep dive into producing to Kafka with data from other streaming sources.

# Example application

This is what the example application does:

1. consumes messages from topic `topic_a`,
2. transforms the received records,
3. produces the transformed records to topic `topic_b`,
4. commits the consumer offsets.

The application uses the zio-kafka streaming APIs. We need to explicitly commit offsets; applications that use the
streaming API process records asynchronously with the consumer, and therefore, committing offsets automatically is not
possible.

```scala

val consumerSettings: ConsumerSettings =
  ConsumerSettings(List("localhost:9092")).withGroupId("my-application")
val producerSettings: ProducerSettings =
  ProducerSettings(List("localhost:9092"))

for {
  consumer <- Consumer.make(consumerSettings)
  producer <- Producer.make(producerSettings, Serde.int, Serde.string)
  _ <-
    consumer
      // Create a stream of records, consuming all records from a topic
      .plainStream(Subscription.topics("my-input-topic"), Serde.int, Serde.long)
      // Convert every record in a ProducerRecord, and the ConsumerRecord's offset
      .map { record =>
        val key: Int = record.record.key()
        val value: Long = record.record.value()
        val newValue: String = value.toString

        val producerRecord: ProducerRecord[Int, String] =
          new ProducerRecord("my-output-topic", key, newValue)
        (producerRecord, record.offset)
      }
      // For every chunk of producerRecords and offsets
      .mapChunksZIO { chunk =>
        val records = chunk.map(_._1)
        val offsetBatch = OffsetBatch(chunk.map(_._2))

        // produce the chunk of records, when acked by kafka, commit the consumed offsets,
        // returning an empty chunk (`mapChunksZIO` requires this function to return a ZIO[Chunk])
        producer.produceChunk[Any, Int, String](records) *>
          offsetBatch.commit.as(Chunk.empty)
      }
      .runDrain
} yield ()
```

### Performance analysis

To analyze throughput performance of a zio-streams program, it is important to find out the characteristics of the
stream's chunks, and to make sure that the chunking structure stays intact. (See [avoiding
chunk-breakers](avoiding-chunk-breakers.md) for more information.)

The program above uses a zio-kafka consumer as stream source. Kafka consumers poll the broker to get new records.
Zio-kafka guarantees that all records that were received in a poll for a given partition, are in the same stream chunk.
This is ideal; partitions can be processed in parallel, and we can process as many records as are available.

By default, the zio-kafka consumer does pre-fetching. The result is that as soon as the stream processed a chunk of
records, the next chunk of records is already available. This increases throughput compared to having to wait for the
next poll.

The next operation —`map`— keeps the chunking structure intact. Nothing to improve here.

The next operation —`mapChunksZIO`— also keeps the chunking structure intact. In this operator, we produce a chunk of
records. Once they are acknowledged by the Kafka broker, we commit the offsets of the consumed records.

### Optimizing consumer throughput

Let's zoom in on the consumer part of the example program.

Method `plainStream` produces a single stream, and chunks from different partitions are processed serially. At the
cost of making the program more complex, we can increase throughput with per-partition parallel processing. See
[consuming with more parallelism](consuming-kafka-topics-using-zio-streams.md#consuming-with-more-parallelism) for more
details. On this page, we do not explore this optimization further.

Zio-kafka's prefetching automatically adapts and is usually as good as it can be. Only for very small/large records, or
for large amounts of topics, you may need to configure it better. Please see [consumer tuning](consumer-tuning.md) to find
out how.

### Optimizing producer throughput

ℹ️ Know your domain. In cases where messages need to be produced in a certain order, the techniques in this section are
not suitable.

Let's focus on the producer part of the program, that is, the `mapChunksZIO` operation in the example above. The
operator pulls a chunk of records from upstream, produces that chunk to Kafka, awaits the acknowledgements, and then the
process repeats. All these tasks are executed sequentially.

We can optimize througput by pulling and producing the next chunk of records concurrently with waiting for the
acknowledgements.

To implement this with zio-streams, we need to split the `mapChunksZIO` operation from the example above in separate
stream operations:

1. produce the records
2. await the acknowledgements and commits offsets

We then use the `buffer` operator between those operations. Buffer allows upstream to run ahead in a separate fiber.

The following diagram compares the two approaches. The rectangles represent stream operations.

![](producer-stream-seq-and-par.svg)

And now in code. Replace the `mapChunksZIO` operator with the following code:

```scala
  // Expose the chunking structure of the stream
  .chunks         // ZStream[_, _, Chunk[(ProducerRecord, Offset)]]

  // For every exposed chunk
  .mapZIO { chunk =>
    val records = chunk.map(_._1)
    val offsetBatch = OffsetBatch(chunk.map(_._2))
    // Produce asynchronously
    producer.produceChunkAsync(records, Serde.int, Serde.string)
      .map(await => (await, offsetBatch))
  }               // ZStream[_, _, (Task[Chunk[RecordMetadata]], OffsetBatch)]

  // Allow the producer to run ahead, producing up to 4 extra chunks
  .buffer(4)      // ZStream[_, _, (Task[Chunk[RecordMetadata]], OffsetBatch)]

  // Await acknowledgements for a chunk, and commit
  .mapZIO { case (await, offsetBatch) =>
    // 'await' is a ZIO, run it, and then commit offsets
    await *> offsetBatch.commit
  }               // ZStream[_, _, Unit]
```

Method `producer.produceChunkAsync` returns a `Task[Task[]]`. The outer task completes when the records have been sent.
Completion gives us the inner task, which completes when the acknowledgements are received.

In the code we use `buffer(4)` which allows the program to produce up to 4 extra chunk of records while awaiting
acknowledgements.

#### Aside, when commits are not needed

When commits are not needed, we can simplify the code a bit:

```scala
val stream: ZStream[_, _, ProducerRecord] = ???

stream
  // Expose the chunking structure of the stream
  .chunks         // ZStream[_, _, Chunk[ProducerRecord]]
  // Every chunk gets produced asynchronously
  .mapZIO { records =>
    producer.produceChunkAsync(records, Serde.int, Serde.string)
  }               // ZStream[_, _, Task[Chunk[RecordMetadata]]]
  // Allow the producer to run ahead, producing up to 4 extra chunks
  .buffer(4)      // ZStream[_, _, Task[Chunk[RecordMetadata]]]
  // Await acknowledgements per chunk
  .flattenZIO     // ZStream[_, _, Chunk[RecordMetadata]]
```

### Optimizing committing consumer offsets

Commiting offsets can become the bottleneck in highly optimized zio-kafka programs. This is because high commit rates
can put a high strain on the broker. Next up, we split off committing to its own fiber as well by using
`aggregateAsyncWithin`. See [consuming with zio-streams](consuming-kafka-topics-using-zio-streams.md) to learn how this
works.

Replace the part after `.buffer()` with:

```scala
  // Await acknowledgements for a chunk 
  .mapZIO { case (await, offsetBatch) =>
    // 'await' is a ZIO, run it, and then return the offsets
    await.as(offsetBatch)
  }                   // ZStream[_, _, OffsetBatch]
  .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
                      // ZStream[_, _, OffsetBatch]
  .mapZIO(_.commit)   // ZStream[_, _, Unit]
```

---

## Creating a zio-kafka Consumer

Create a consumer with `Consumer.make`, passing some `ConsumerSettings`:

```scala

val consumerSettings: ConsumerSettings =
  ConsumerSettings(List("localhost:9092")).withGroupId("group")
val consumer: ZIO[Scope, Throwable, Consumer] =
  Consumer.make(consumerSettings)
```

The consumer settings are not only used for passing properties to the underlying Kafka consumer. You can also configure
zio-kafka features such as `rebalanceSafeCommits` (see [preventing duplicates](preventing-duplicates.md)), manual offset
retrieval (see [offset retrieval](partition-assignment-and-offset-retrieval.md)), and the pre-fetch strategy (see
[consumer tuning](consumer-tuning.md)).

Notice that the consumer requires a `Scope` in the environment. When this scope closes, the consumer closes its
connection with the Kafka cluster. A scope can be created with the `ZIO.scoped` method:

```scala
ZIO.scoped {
  // make the consumer
  // use the consumer
}
// end of scope, consumer has closed connection to Kafka cluster.
```

If the consumer is provided as a layer, you can use a scoped layer (see below).

:::caution
If no explicit `Scope` is provided, the consumer uses the application's scope, and it will only close the connection
with the Kafka cluster when the application exits.
:::

## Providing a consumer as a layer

If your application uses the [ZIO service pattern](https://zio.dev/reference/service-pattern/), the consumer can
be provided as a layer. Here's an example of how to construct a consumer layer:

```scala

val consumerLayer: Layer[Any, Throwable, Consumer] =
  ZLayer.scoped { // (1)
    val consumerSettings: ConsumerSettings =
      ConsumerSettings(List("localhost:9092")).withGroupId("group")
    Consumer.make(consumerSettings)
  }
```

(1) By using `ZLayer.scoped`, the consumer is closed when the layer is no longer needed.

## Consumer settings from a layer (zio-kafka 3.x)

If you wish to create the `ConsumerSettings` and the `Consumer` from a layer, you can use `Consumer.live`. It takes
the consumer settings as a layer and produces a consumer.

The `provide` section of the main program could look like this:

```scala
.provide(
  consumerSettingsLayer,
  Consumer.live
)
```

## Consumer settings from a layer (zio-kafka 2.x)

If you wish to create the `ConsumerSettings` and the `Consumer` from a layer, you can use `Consumer.live`. It takes
the consumer settings and diagnostics settings as a layer and produces a consumer.

The `provide` section of the main program could look like this:

```scala
.provide(
  consumerSettingsLayer,
  ZLayer.succeed(Consumer.NoDiagnostics),
  Consumer.live
)
```

---

## Getting Started with ZIO Kafka

[ZIO Kafka](https://github.com/zio/zio-kafka) is a Kafka client for ZIO. It provides a purely functional, streams-based interface to the Kafka
client and integrates effortlessly with ZIO and zio-streams. Often zio-kafka programs have a _higher_ throughput than
programs that use the Java Kafka client directly (see section [Performance](#performance) below).

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-kafka/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-kafka_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-kafka_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-kafka_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-kafka_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-kafka-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-kafka-docs_2.13) [![ZIO Kafka](https://img.shields.io/github/stars/zio/zio-kafka?style=social)](https://github.com/zio/zio-kafka) [![Scala Steward badge](https://img.shields.io/badge/Scala_Steward-helping-blue.svg?style=flat&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAAQCAMAAAARSr4IAAAAVFBMVEUAAACHjojlOy5NWlrKzcYRKjGFjIbp293YycuLa3pYY2LSqql4f3pCUFTgSjNodYRmcXUsPD/NTTbjRS+2jomhgnzNc223cGvZS0HaSD0XLjbaSjElhIr+AAAAAXRSTlMAQObYZgAAAHlJREFUCNdNyosOwyAIhWHAQS1Vt7a77/3fcxxdmv0xwmckutAR1nkm4ggbyEcg/wWmlGLDAA3oL50xi6fk5ffZ3E2E3QfZDCcCN2YtbEWZt+Drc6u6rlqv7Uk0LdKqqr5rk2UCRXOk0vmQKGfc94nOJyQjouF9H/wCc9gECEYfONoAAAAASUVORK5CYII=)](https://scala-steward.org)

## Introduction

Apache Kafka is a distributed event streaming platform that acts as a distributed publish-subscribe messaging system. It enables us to build distributed streaming data pipelines and event-driven applications.

Kafka has a mature Java client for producing and consuming events, but it has a low-level API. Zio-kafka is a ZIO native client for Apache Kafka. It has a high-level streaming API on top of the Java client. So we can produce and consume events using the declarative concurrency model of zio-streams. In addition, zio-kafka supports an even higher level API where you only write the processing part and the rest is handled by zio-kafka.

## Features

- Exposes the Java Kafka consumer, producer and admin clients with a ZIO based interface.
- Consuming:
  - 2 APIs: streaming and ZIO workflow based
  - supports custom deserialization
  - process each partition in parallel for highest throughput
  - allows batched processing for highest throughput
  - configurable per partition pre-fetching (with back-pressure)
  - the only async Kafka consumer [without duplicates](https://zio.dev/zio-kafka/preventing-duplicates) after a rebalance (as far as we know)
  - very configurable
  - automatic or manual starting offset
  - supports external commits
  - retries after authentication/authorization errors
  - exposes metrics
  - diagnostics API
- Producing:
  - 2 APIs: streaming and ZIO workflow based
  - supports custom serialization
  - allows for batches for highest throughput
  - optionally await broker acknowledgements
  - optional retries after authentication/authorization errors
  - exposes metrics
- Admin API:
  - exposes all the admin client methods with a ZIO based interface
  - safer than the wrapped java client by limiting the number of concurrent operations
- Proper errors when broker expects SSL (no [OOM crashes](https://issues.apache.org/jira/browse/KAFKA-4090))
- Test kit with embedded kafka broker
- Well documented
- Community support via [Discord](https://discord.com/channels/629491597070827530/629497941719121960)
- Commercial support via [Ziverge](https://www.ziverge.com)

## Getting started

See the [zio-kafka tutorial](tutorial.md) for a grand tour of the different ways you can use zio-kafka.

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-kafka"         % "3.6.0"
libraryDependencies += "dev.zio" %% "zio-kafka-testkit" % "3.6.0" % Test
```

Snapshots are available on Sonatype's snapshot repository https://oss.sonatype.org/content/repositories/snapshots.
[Browse here](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-kafka_3/) to find available versions.

If you use `zio-kafka` 3.x, scala 3, `zio-kafka-testkit` and `embedded-kafka`, you also need to add the following to
your `build.sbt` file:

```scala
excludeDependencies += "org.scala-lang.modules" % "scala-collection-compat_2.13"
```

## Example

Let's write a simple Kafka producer and consumer using zio-kafka with zio-streams. Before everything, we need a running instance of Kafka. We can do that by saving the following docker-compose script in the `docker-compose.yml` file and run `docker compose up`:

```docker-compose
services:
  broker:
    image: apache/kafka:3.9.0
    container_name: broker
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_BROKER_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
```

Now, we can run our ZIO Kafka Streaming application:

```scala

object ReadmeExample extends ZIOAppDefault {

  private val producerRun: ZIO[Any, Throwable, Unit] =
    ZIO.scoped {
      for {
        producer <-
          Producer.make(
            ProducerSettings(List("localhost:9092"))
          )
        _ <- ZStream
          .fromSchedule(Schedule.fixed(2.seconds))
          .mapZIO(_ => Random.nextIntBetween(0, Int.MaxValue))
          .mapZIO { random =>
            producer.produce[Any, Long, String](
              topic = "random-topic",
              key = (random % 4).toLong,
              value = random.toString,
              keySerializer = Serde.long,
              valueSerializer = Serde.string
            )
          }
          .runDrain
      } yield ()
    }

  private val consumerRun: ZIO[Any, Throwable, Unit] =
    ZIO.scoped {
      for {
        consumer <-
          Consumer.make(
            ConsumerSettings(List("localhost:9092"))
              .withGroupId("group")
          )
        _ <- consumer
          .plainStream(Subscription.topics("random"), Serde.long, Serde.string)
          .tap(r => Console.printLine(r.value))
          .map(_.offset)
          .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
          .mapZIO(_.commit)
          .runDrain
      } yield ()
    }

  override def run: ZIO[Any, Throwable, Unit] =
    ZIO.raceFirst(producerRun, List(consumerRun))
}
```

Read the [ZIO Kafka tutorial](tutorial.md) for a more gentle introduction.

## Documentation

If you are reading this page from https://zio.dev/zio-kafka/, the documentation is in the sub-pages and can be found by
scrolling the left panel of this page.

If you are reading this from GitHub, the documentation can be found in
[the docs folder](https://github.com/zio/zio-kafka/tree/master/docs).

## Resources

### Articles

- [ZIO Kafka faster than Java Kafka](https://day-to-day-stuff.blogspot.com/2024/12/zio-kafka-faster-than-java-kafka.html) by Erik van Oosten (December 2024)
- [Introduction to zio-kafka](https://www.baeldung.com/scala/zio-kafka-intro) by Stefanos Georgakis, Baeldung (January 2023)
- [How to implement streaming microservices with ZIO 2 and Kafka](https://scalac.io/blog/streaming-microservices-with-zio-and-kafka/) by Jorge Vasquez, Scalac (June 2023)
- [Zio Kafka](https://medium.com/@knoldus/zio-kafka-d865fc20174a) by Knoldus Inc (January 2022)
- [Writing a Simple Producer and Consumer Using ZIO Workflows](https://pramodshehan.medium.com/writing-a-simple-producer-and-consumer-using-zio-workflows-a57def08210c) by Pramod Shehan (December 2022)
- [ZIO Kafka with transactions - a debugging story](https://www.ziverge.com/post/zio-kafka-with-transactions---a-debugging-story/) by Daniel Vigovszky, Ziverge (June 2022)
- [Introduction to Zio-Kafka](https://blog.knoldus.com/introduction-to-zio-kafka/) by Akash Kumar (March 2022)
- [Introduction to Zio-Kafka](https://blog.nashtechglobal.com/introduction-to-zio-kafka/) by Khalid Ahmed, Nash Tech (March 2022)
- [ZIO Kafka: A Practical Streaming Tutorial](https://rockthejvm.com/articles/zio-kafka/) by Riccardo Cardin, Rock the JVM (August 2021)
- [Using ZIO Kafka with offset storage in Postgres for transactional processing](https://functional.works-hub.com/learn/using-zio-kafka-with-offset-storage-in-postgres-for-transactional-processing-be4a2) by Marek Kadek (March 2021)
- [Streaming microservices with ZIO and Kafka](https://scalac.io/streaming-microservices-with-zio-and-kafka/) by Aleksandar Skrbic (February 2021)
- [An Introduction to ZIO Kafka](https://www.ziverge.com/post/an-introduction-to-zio-kafka/) by Ziverge (April 2020)

### Video

- [Free ZIO Kafka course](https://www.learnscala.dev/) Free video training courses by Alvin Alexander, sponsored by Ziverge (2025) to learn Scala 3, Functional Programming, and ZIO 2. The zio-kafka module can be followed separately.
- [Optimizing Data Transfer Kafka to BQ: let's use Scala to make it custom](https://www.youtube.com/watch?v=McnC2UU-RIE) by Dario Amorosi, Adevinta (November 2024)
- [Making ZIO-Kafka Safer and Faster in 2023](https://www.youtube.com/watch?v=MJoRwEyyVxM) by Erik van Oosten (November 2023)
- [ZIO Kafka with Scala: A Tutorial](https://www.youtube.com/watch?v=ExFjjczwwHs) by Rock the JVM (August 2021)
- [ZIO WORLD - ZIO Kafka](https://www.youtube.com/watch?v=GECv1ONieLw) by Aleksandar Skrbic (March 2020) — Aleksandar Skrbic presented ZIO Kafka, a critical library for the modern Scala developer, which hides some of the complexities of Kafka.

### Example projects

- [Kafka BigQuery Express](https://github.com/adevinta/kafka-bigquery-express/) by Adevinta (November 2024) A production system to copy data from Kafka to BigQuery, safely and cost-effectively. (See also the video "Optimizing Data Transfer...".)
- [zio-kafka-showcase](https://github.com/ScalaConsultants/zio-kafka-showcase) by Jorge Vásquez, Example project that demonstrates how to build Kafka based microservices with Scala and ZIO
- [zio-kafka-demo1](https://github.com/pramodShehan5/zio-kafka-demo1) (December 2022), example consumer and producer using zio-kafka 2.0.5
- [zio-kafka-example-app](https://github.com/zivergetech/zio-kafka-example-app) by Ziverge (December 2020), example application using zio-kafka 0.8.0

## Adopters

Here is a partial list of companies using zio-kafka in production.

Want to see your company here? [Submit a PR](https://github.com/zio/zio-kafka/edit/master/docs/index.md)!

* [Conduktor](https://www.conduktor.io)
* [KelkooGroup](https://www.kelkoogroup.com)
* [Rocker](https://rocker.com)

## Performance

Often, _zio-kafka programs consume with a higher throughput_ than programs that use the java-kafka client directly.
Read on for the details.

By default, zio-kafka programs process partitions in parallel. The default java-kafka client does not provide parallel
processing. Of course, there is some overhead in buffering records and distributing them to the fibers that need them.
On 2024-11-23, we estimated that zio-kafka consumes faster than the java-kafka client when processing takes more than
~1.2ms per 1000 records. The precise time depends on many factors. Please
see [this article](https://day-to-day-stuff.blogspot.com/2024/12/zio-kafka-faster-than-java-kafka.html) for more
details.

If you do not care for the convenient ZStream based API that zio-kafka brings, and latency is of absolute importance,
using the java based Kafka client directly is still the better choice.

## Developers

* [Benchmarks and Flame graphs](https://github.com/zio/zio-kafka/blob/master/zio-kafka-bench/README.md)

---

## Zio-Kafka Metrics

Zio-kafka exposes all the metrics of the wrapped Java based consumer and producer, plus some more metrics about the
zio-kafka consumer itself.

## Java client metrics

The metrics from the Java metrics can be obtained via the `Consumer.metrics` and `Producer.metrics` methods. Both
return a live view on the internal metrics of the consumer/producer. We currently do not expose these metrics elsewhere,
a PR to copy them to the zio-metrics API is welcome.

## Zio-kafka metrics

The zio-kafka consumer and producer collects some additional metrics using the zio-metrics API. This allows any
zio-metrics backend to access and process the observed values.

By default, no tags are added. Tags can be configured via `ConsumerSettings.withMetricsLabels` / `ProducerSettings.withMetricsLabels`.

Like the zio-metrics we follow Prometheus conventions. This means that:

- durations are expressed in seconds,
- counters can only increase,
- metric names use snake_case and end with the unit name where possible.

The histograms each use 10 buckets. To reach a decent range while keeping sufficient accuracy at the low end, most
bucket boundaries use an exponential series based on 𝑒.

## Zio-kafka consumer metrics

To interpret the metrics you need to know how zio-kafka works internally.

![](consumer-internals.svg)

See the [tuning guide](consumer-tuning.md#using-metrics-to-tune-the-consumer) for more details on the internals.

### Poll metrics

| Type      | Name                                                  | Description                                                                        |
|-----------|-------------------------------------------------------|------------------------------------------------------------------------------------|
| counter   | `ziokafka_consumer_polls`                             | The number of polls.                                                               |
| histogram | `ziokafka_consumer_poll_latency_seconds`              | The duration of a single poll in seconds.                                          |
| histogram | `ziokafka_consumer_poll_size`                         | The number of records fetched by a single poll.                                    |
| gauge     | `ziokafka_consumer_partitions_resumed_in_latest_poll` | The number of partitions resumed in the latest poll call.                          |
| gauge     | `ziokafka_consumer_partitions_paused_in_latest_poll`  | The number of partitions paused in the latest poll call (because of backpressure). |
| counter   | `ziokafka_consumer_poll_auth_errors`                  | The number of polls that ended with an authentication or authorization error.      |

### Partition stream metrics

These metrics are updated after every poll.

| Type      | Name                                 | Description                                                            |
|-----------|--------------------------------------|------------------------------------------------------------------------|
| histogram | `ziokafka_consumer_pending_requests` | The number of partitions that ran out of records (the queue is empty). |
| histogram | `ziokafka_consumer_queue_size`       | The number of records queued for a partition.                          |
| histogram | `ziokafka_consumer_all_queue_size`   | The total number of records queued for all partitions.                 |
| histogram | `ziokafka_consumer_queue_polls`      | The number of polls during which records are idling in a queue.        |

### Commit metrics

These metrics measure the separate commit requests issued through zio-kafka's api.

| Type       | Name                                       | Description                                         |
|------------|--------------------------------------------|-----------------------------------------------------|
| histogram  | `ziokafka_consumer_pending_commits`        | The number of commits that are awaiting completion. |
| counterInt | `ziokafka_consumer_commits`                | The number of commits.                              |
| histogram  | `ziokafka_consumer_commit_latency_seconds` | The duration of a commit in seconds.                |

### Aggregated commit metrics

After every poll zio-kafka combines all outstanding commit requests into 1 aggregated commit. These metrics are for the aggregated commits.

| Type       | Name                                                  | Description                                                                |
|------------|-------------------------------------------------------|----------------------------------------------------------------------------|
| counterInt | `ziokafka_consumer_aggregated_commits`                | The number of aggregated commits.                                          |
| histogram  | `ziokafka_consumer_aggregated_commit_latency_seconds` | The duration of an aggregated commit in seconds.                           |
| histogram  | `ziokafka_consumer_aggregated_commit_size`            | An approximation of the number of records (offsets) per aggregated commit. |

### Rebalance metrics

| Type       | Name                                              | Description                                                  |
|------------|---------------------------------------------------|--------------------------------------------------------------|
| counterInt | `ziokafka_consumer_rebalances`                    | The number of rebalances.                                    |
| gauge      | `ziokafka_consumer_partitions_currently_assigned` | The number of partitions currently assigned to the consumer. |
| counterInt | `ziokafka_consumer_partitions_assigned`           | The number of partitions assigned to the consumer.           |
| counterInt | `ziokafka_consumer_partitions_revoked`            | The number of partitions revoked to the consumer.            |
| counterInt | `ziokafka_consumer_partitions_lost`               | The number of partitions lost to the consumer.               |

### Runloop metrics

These metrics are updated after every poll.

| Type      | Name                                   | Description                                        |
|-----------|----------------------------------------|----------------------------------------------------|
| gauge     | `ziokafka_consumer_subscription_state` | Whether the consumer is subscribed (1) or not (0). |
| histogram | `ziokafka_consumer_command_queue_size` | The number of commands queued in the consumer.     |
| histogram | `ziokafka_consumer_commit_queue_size`  | The number of commits queued in the consumer.      |

See [ConsumerMetrics.scala](https://github.com/zio/zio-kafka/blob/master/zio-kafka/src/main/scala/zio/kafka/consumer/internal/ConsumerMetrics.scala) for the exact details.

## Example

Here is an example dashboard that could be built with these metrics:

![metrics-dashboard.png](metrics-dashboard.png)

## Zio-kafka producer metrics

To understand the producer metrics it useful to know how the producer works.

When you call the producer it first serializes the given batch of records (a single record is treated like a batch of
one). The batch is then offered to an internal send queue. Once the queue accepts the batch, the `produce*Async` methods
return immediately. The `produce*Sync` methods return later, when all acknowledgements are in. Once the fiber is ready
for it, it pulls a batch from the queue and sends the records to the Kafka broker one by one. Meanwhile, the broker
acknowledges records.

![](producer-internals.svg)

### Producer metrics

| Type      | Name                                | Description                                                                             |
|-----------|-------------------------------------|-----------------------------------------------------------------------------------------|
| counter   | `ziokafka_producer_calls`           | The number of times a produce method is invoked.                                        |
| histogram | `ziokafka_producer_latency_seconds` | The duration of a single produce, from after serialization to acknowledged, in seconds. |
| counter   | `ziokafka_producer_records`         | The number of records produced.                                                         |
| histogram | `ziokafka_producer_batch_size`      | The number of records per produce call.                                                 |

### Queue metrics

| Type      | Name                                           | Description                                                     |
|-----------|------------------------------------------------|-----------------------------------------------------------------|
| histogram | `ziokafka_producer_send_queue_size`            | The number of records in the zio-kafka send queue.              |
| histogram | `ziokafka_producer_send_queue_latency_seconds` | Time in send queue, including waiting for capacity, in seconds. |

### Send metrics

| Type    | Name                            | Description                                                                           |
|---------|---------------------------------|---------------------------------------------------------------------------------------|
| counter | `ziokafka_producer_auth_errors` | The number of record sends that resulted in an authentication or authorization error. |

See [ProducerMetrics.scala](https://github.com/zio/zio-kafka/blob/master/zio-kafka/src/main/scala/zio/kafka/producer/internal/ProducerMetrics.scala) for the exact details.

---

## Migrating to zio-kafka 3

Zio-kafka 3.0.0 brings a number of backwards incompatible changes:

1. Removal of all deprecated methods, _including accessor methods,_
2. Kafka 4.x
3. The transactional producer is now much easier to use,
4. Diagnostics set via consumer settings, 
5. `restartStreamOnRebalancing` mode is no longer supported.

This document helps you migrate from zio-kafka 2 to zio-kafka 3.

# 1. Deprecations

Zio-kafka 3.0.0 removes everything that was deprecated in the zio-kafka 2.x series. In particular, this includes
accessor methods. To prepare for zio-kafka 3.0, _you should always first migrate to zio-kafka 2.12.0_ and solve all
deprecation issues, using this page as a guide.

## Renamed methods

Some methods have just been renamed. Read the deprecation message and try the new method name. If it compiles, you're
done. Otherwise, read on.

## Consumer, Producer and TransactionalProducer accessor methods

Accessor methods are little helper methods that look up a service from the environment, and then forward your call to
that service. Accessor methods have not been recommended for some time and are now deprecated. The
[ZIO service pattern](https://zio.dev/reference/service-pattern/) provides a much cleaner approach for accessing
services.

All accessor methods provided by zio-kafka are deprecated in zio-kafka 2.11 and will be removed in zio-kafka 3.0. If
you use these accessor methods follow one of these approaches:

### Use the ZIO Service pattern

This is the best option. For established codebases it may be a lot of work to get here. If you are already follow
this pattern, using it for zio-kafka services as well will be easy. See [ZIO service pattern](https://zio.dev/reference/service-pattern/) for more information.

Here is an example with a `Consumer`, but it works the same with `Producer` and `TransactionalProducer`. We get the
`Consumer` from the environment in the layer with the `ZIO.service` method, and then inject it into the service class.
The service now uses the consumer directly: with `consumer` instead of `Consumer`.

```scala

trait Service {
    def someMethod: ZIO[Any, Throwable, String]
}

object Service {
  
  def layer: ZLayer[Consumer, Throwable, Service] = ZLayer {
      for {
        consumer <- ZIO.service[Consumer]
      } yield ServiceLive(consumer)
  }
  
}

case class ServiceLive(consumer: Consumer) extends Service {

  override def someMethod: ZIO[Any, Throwable, String] = {
    // use `consumer`, not `Consumer`:
    consumer.plainStream(/*...*/).runHead
  }
}
```

Constructing a `Consumer` layer is described in [creating a consumer](creating-a-consumer.md). Constructing a
`Producer` or `TransactionalProducer` layer works in a similar way.

### YOLO, use `ZIO.service` everywhere

The other option is to replace all accessor methods of `Consumer`, `Producer` and `TransactionalProducer` as follows:

- `Consumer.method(...)` => `ZIO.serviceWithZIO[Consumer](_.method(...))`
- `Producer.method(...)` => `ZIO.serviceWithZIO[Producer](_.method(...))`
- `TransactionalProducer.method(...)` => `ZIO.serviceWithZIO[TransactionalProducer](_.method(...))`

Or, alternatively, use `ZIO.service`.

For example: `Consumer.method(...)` is transformed to:
```scala
for {
  consumer <- ZIO.service[Consumer]
  _        <- consumer.method(...)
} yield () 
```

## Zio-test-kit

Zio-kafka provides a `zio-kafka-testkit` library to help you test your code using zio-kafka. Several methods in the
`KafkaTestUtils` class have been replaced:

- The `produce*` and `scheduledProduce*` methods, which get a producer from the environment, have been deprecated.
  Each of these methods now has a variant with an explicit producer parameter.
- All methods that produce a layer (e.g. `consumer`, `producer`) have _not_ been deprecated, but they have been given a
  more convenient alternative: `makeConsumer`, `makeProducer`, etc.
- The `withAdmin`, `withSaslAdmin`, `withSslAdmin` methods are deprecated and are replaced by `makeAdminClient`,
  `makeSaslAdminClient` and `makeSslAdminClient`.

Here is a typical example and the new version:

```scala
// Old
override def spec: Spec[TestEnvironment, Any] =
  suite("old example suite")(
    test("uses a producer") {
      for {
        _ <- KafkaTestUtils.produceOne("topic", "key", "message")
      } yield assertCompletes
    }
  )
    .provideSome[Kafka](KafkaTestUtils.producer)
    .provideShared(Kafka.embedded)

// New
// Added `Scope`:
override def spec: Spec[TestEnvironment & Scope, Any] =
  suite("new example suite")(
    test("uses a producer") {
      for {
        // Make a producer explicitly (instead of via a layer):
        producer <- KafkaTestUtils.makeProducer
        // Notice explicit producer parameter:
        _ <- KafkaTestUtils.produceOne(producer, "topic", "key", "message")
      } yield assertCompletes
    }
    // Producer layer removed.
    .provideSomeShared[Scope](Kafka.embedded)
    // Using `provideSomeShared[Scope]` instead of `provideShared`
  )
```

`KafkaTestUtils.makeProducer` and the other `make*` methods require a `Scope` in the environment. This scope is
normally provided by the test framework (notice how `Scope` was added to the type parameter of `Spec`). If needed, a
smaller scope can be given with the `ZIO.scoped` method. See also [writing tests](writing-tests.md) for details on how
to provide more layers.

Here is an example of how to rewrite code that uses one of the deprecated `with*Admin` methods:

```scala
// Old
KafkaTestUtils.withAdmin { adminClient =>
  adminClient.method() // Use the admin client...
}

// New
for {
  adminClient <- KafkaTestUtils.makeAdminClient
  _           <- adminClient.method() // Use the admin client...
} yield ()
```

# 2. Kafka 4.x

Starting with 3.0.0, zio-kafka is based on the kafka 4.0.0 client. This has a number of consequences:

* Java 11 support is dropped. Even though the kafka client library still supports java 11, zio-kafka's unit tests
  depend on an embedded kafka broker and the kafka brokers no longer support it.
* `ConsumerGroupState` (used in the admin client) was renamed to `GroupState`, unfortunately without a deprecation
  cycle.
* Due to [KAFKA-18818](https://issues.apache.org/jira/browse/KAFKA-18818) we add a 550ms delay after operations like
  `createTopic`, `deleteTopic` and `createACL` in the admin client.
* Two deprecated methods of kafka's admin client were removed. Their equivalent in zio-kafka were unfortunately not
  deprecated yet but had to be removed as well:
  * `AdminClient.alterConfigs` was removed, use `AdminClient.incrementalAlterConfigs` instead.
  * `AdminClient.alterConfigsAsync` was removed, use `AdminClient.incrementalAlterConfigs` instead.

There may be more changes that affect you. See the [Kafka 4.0.0 release announcement](https://kafka.apache.org/blog#apache_kafka_400_release_announcement) for more details.

# 3. Changes in the transactional producer

The transactional producer is now much easier to use. The zio-kafka 2.x transactional API is so complicated that we
expect only a few very experienced users to make use of it. Therefore, only the new API is described in
[transactions](transactions.md).

# 4. Diagnostics set via consumer settings

Diagnostics, an optional callback for key events in the consumer life-cycle, is now set via
`ConsumerSettings.withDiagnostics`. In zio-kafka 2.x it was the second (optional) argument to `Consumer.make`, or an
additional input layer for `Consumer.live`.

### Migrate a layers app without diagnostics

If your program uses ZIO layers, but you don't need diagnostics, it will look like this:

```scala
// zio-kafka 2.x
.provide(
  consumerSettingsLayer,
  ZLayer.succeed(Consumer.NoDiagnostics), // 1
  Consumer.live
)
```

For zio-kafka 3.x simply remove the line marked with `// 1` and you are done.

### Migrate uses of `Consumer.make` with diagnostics

Rewrite your code following this example:

```scala
// zio-kafka 2.x
val diagnostics = ???
val settings = ConsumerSettings(bootstrap)
val consumer = Consumer.make(settings, diagnostics)

// zio-kafka 3.x
val diagnostics = ???
val settings = ConsumerSettings(bootstrap).withDiagnostics(diagnostics)
val consumer = Consumer.make(settings)
```

## Migrate a layers app with diagnostics

In case your application provides diagnostics via a layer, providing the layers looks like this:

```scala
// zio-kafka 2.x
.provide(
  consumerSettingsLayer,
  diagnosticsLayer,
  Consumer.live
)
```

The cleanest solution is to merge the code that constructs `diagnosticsLayer` into the code that constructs
`consumerSettingsLayer`.

The other option is to apply a 'quick fix' that transforms the two layers into a single layer:

```scala
// zio-kafka 3.x
.provide(
  (consumerSettingsLayer ++ diagnosticsLayer) >>> ZLayer {
    for {
      settings <- ZIO.service[ConsumerSettings]
      diagnostics <- ZIO.service[Consumer.ConsumerDiagnostics]
    } yield settings.withDiagnostics(diagnostics)
  },
  Consumer.live
)
```

# 5. `restartStreamOnRebalancing` mode

This mode is no longer available in zio-kafka 3. With `restartStreamOnRebalancing` all streams are ended during a
rebalance, even when the partition for that stream was not revoked. One of its purposes was to enable transactional
consuming. Since zio-kafka 3 however, transactional consuming no longer needs this mode.

If your goal is to [prevent duplicates](preventing-duplicates.md), please use `rebalanceSafeCommits` instead.

# Other changes?

If you find a change that is not documented here then please let us know via a
[new issue](https://github.com/zio/zio-kafka/issues/new).

---

## Partition Assignment And Offset Retrieval

`zio-kafka` offers several ways to control which Kafka topics and partitions are assigned to your application.

| Use case                                           | Method                                                  |
|----------------------------------------------------|---------------------------------------------------------|
| One or more topics, automatic partition assignment | `Subscription.topics("my_topic", "other_topic")`        |
| Topics matching a pattern                          | `Subscription.pattern("topic.*")`                       |
| Manual partition assignment                        | `Subscription.manual("my_topic" -> 1, "my_topic" -> 2)` |

By default `zio-kafka` starts streaming a partition from the last committed offset for the consumer group, or the latest message on the topic if no offset has yet been committed. You can also choose to store offsets outside of Kafka. This can be useful in cases where consistency between data stores and consumer offset is required.

| Use case                                                           | Method                                                                       |
|--------------------------------------------------------------------|------------------------------------------------------------------------------|
| Offsets in Kafka, start at latest message if no offset committed   | `OffsetRetrieval.Auto()`                                                     |
| Offsets in Kafka, start at earliest message if no offset committed | `OffsetRetrieval.Auto(AutoOffsetStrategy.Earliest)`                          |
| Manual/external offset storage                                     | `Manual(getOffsets: Set[TopicPartition] => Task[Map[TopicPartition, Long]])` |

For manual offset retrieval, the `getOffsets` function is called for each topic-partition that is assigned to the consumer, either via Kafka's rebalancing or via a manual assignment.

---

## Preventing duplicates

In zio-kafka processing of records runs asynchronously with partition management. This brings substantial performance
advantages but causes some records to be consumed and processed _twice_ when a rebalance occurs. To prevent this,
since version 2.7.1 zio-kafka supports a new mode in which we prevent duplicates due to rebalances. You can enable it
as follows:

```scala

val consumerSettings: ConsumerSettings =
  ConsumerSettings(List("localhost:9092"))
    .withGroupId("group")
    .withRebalanceSafeCommits(true)       // enable rebalance-safe-commits mode
    .withMaxRebalanceDuration(30.seconds) // defaults to 3 minutes
    .withCommitTimeout(15.seconds)        // defaults to 15 seconds
```

With rebalance-safe-commits mode enabled, rebalances are held up for up to max-rebalance-duration to wait for pending
commits to be completed. Once pending commits are completed, it is safe for another consumer in the group to take over
a partition.

For this to work correctly, your program must process a chunk (of up to `max.poll.records` records) within the
configured max-rebalance-duration. The clock starts the moment the chunk is pushed into the stream and ends when the
commits for these records complete. Zio-kafka uses `commit-timeout` as the worse case time to commit. Therefore, with
the default settings, your program has 2 minutes and 45 seconds until the offsets should be committed.

It is probably clear now that your program should commit the offsets of consumed records. The most straightforward way
is to commit to the kafka brokers. This is done by calling `.commit` on the offset of consumed records (see the consumer
documentation). However, there are more options: external commits and transactional producing.

### Commit to an external system

When you commit to an external system (e.g. by writing to a relational database) the zio-kafka consumer needs to know
about those commits before it can work in rebalance-safe-commits mode. Inform zio-kafka about external commits by
invoking method `Consumer.registerExternalCommits(offsetBatch: OffsetBatch)` (available since zio-kafka 2.10.0).

Here is what this could look like:

```scala

consumer.plainStream(Subscription.topics("topic2000"), Serde.string, Serde.string)
  .mapZIO { record =>
    database.store(record.offset) *> // <-- the external commit
      consumer.registerExternalCommits(OffsetBatch(record.offset))
  }
  .runDrain
```

### Commit with a transactional producer

Transactional producing is described in [transactions](transactions.md).

## Long-running processes / slow consumers

When rebalance-safe-commits mode is enabled, consumers have tighter dead-lines. Besides a suitable `max.poll.records`
and `max.poll.interval.ms` (see [consumer tuning](consumer-tuning.md#long-running-processes--slow-consumers)), a
consumer must also be able to process `max.poll.records` within `maxRebalanceDuration`, with some margin to complete
commits.

With the default values of `500` for `max.poll.records`, 3 minutes for `maxRebalanceDuration` (which is 3/5 of 5
minutes for `max.poll.interval.ms`), and 10 seconds for `commitTimeout`, a consumer must be able to process and commit
`500` records in 2 minutes, 50 seconds (_3 records per second_). **This is a hard threshold.** Change the settings such
that there is sufficient head-room for busy services, network delays, garbage collections or reduced CPU capacity due to
other processes.

In case the consumer can handle less than 10 times the minimum records per second, it is important to use the Range
Assignor (see [PR 1576](https://github.com/zio/zio-kafka/pull/1576) for an extensive explanation).

Settings described in this section can be changed as follows:

```scala
val settings = ConsumerSettings(bootstrapServers)
  .withRebalanceSafeCommits(true)
  .withMaxPollRecords(1)
  .withMaxPollInterval(15.minutes)
  .withMaxRebalanceDuration(4.minutes)
  .withProperty(
    org.apache.kafka.clients.consumer.ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    classOf[org.apache.kafka.clients.consumer.RangeAssignor].getName
  )
  .withCommitTimeout(10.seconds)
  .... etc.
```

## More information

There is more information in the scaladocs of `ConsumerSettings` and the description of
[pull request #1098](https://github.com/zio/zio-kafka/pull/1098) that introduced this feature.
You can also watch the presentation
[Making ZIO-Kafka Safer And Faster](https://www.youtube.com/watch?v=MJoRwEyyVxM). The relevant part starts at 10:24.

---

## Serialization and Deserialization

Zio-kafka deserializes incoming data, and deserializes outgoing data (both keys and values) from byte arrays to any
other type and back. This works by providing a key and value `Deserializer` while constructing a `Consumer`,
and a key and value `Serializer` while constructing the `Producer`.

A `Serde` combines a `Deserializer` and a `Serializer`. Common serdes are provided in the `Serdes` object, e.g.
`Serdes.byteArray`, `Serdes.string` and `Serdes.long`. A serde can be converted to other serdes, or you can create a
custom serde by implementing the `Serde` trait directly.

This document contains:

- Handling failures in a serde
- How to create a custom serde
- How to create and use a custom serde that wraps invalid data
- How to do deserialization in the consumer stream

## Handling failures in a serde

Ideally, a serde can not fail serializing and deserializing. This is for example the case with the provided
`Serdes.byteArray` and `Serdes.string`. This is not the case for any serde that needs to handle invalid input,
(for example `Serdes.long`), or a serde that needs to do a remote lookup.

By default, a consumer stream will fail if it encounters a deserialization error in the serde. Unfortunately, the
resulting failure might not clearly indicate that the cause is in the serde.

There are 2 solutions for improving this:

- Wrap the result of the serde in a `Try` with the `Serde.asTry` method.
- Use `Serdes.byteArray`, put the deserialization code in the consumer stream, or do serialization before handing the
  data to zio-kafka. This way you can handle failures any way you want.

Both approaches are discussed further below.

## Custom Data Type Serdes

Serializers and deserializers for custom data types can be created from scratch, or by converting existing
serdes. For example, to create a serde for an `Instant` from a serde for a `Long`:

```scala

val instantSerde: Serde[Any, Instant] =
  Serdes.long.inmap(java.time.Instant.ofEpochMilli)(_.toEpochMilli)
```

For a complete example of creating a custom serde with zio-json, see the
[Producing and Consuming JSON Data](tutorial.md#producing-and-consuming-json-data) section in the tutorial.

To handle missing data (an empty key or value), you can use the `Serde.asOption` transformer. For example:
`Serdes.string.asOption`. This results in a `None` if the key or value is empty, and in a `Some(string)` otherwise.

## Custom serdes that wraps invalid data

Any `Deserializer[A]` for a given type `A` can be converted into  a `Deserializer[Try[A]]` where deserialization
failures are converted to a `Failure` using the `asTry` method. (Method `asTry` is also available on `Serde`.)

Below is an example of skipping records that fail to deserialize. The offset is passed downstream to be committed.

```scala

val keySerde = Serdes.string
val valueSerde = Serdes.string.asTry   // <-- using `.asTry`

Consumer.make(consumerSettings).flatMap { consumer =>
  consumer
    .plainStream(Subscription.topics("topic150"), keySerde, valueSerde)
    .mapZIO { record => // ⚠️ see section about `mapZIO` below!
      val tryValue: Try[String] = record.record.value()
      val offset: Offset = record.offset

      tryValue match {
        case Success(value) =>
          // Action for successful deserialization
          someEffect(value).as(offset)
        case Failure(exception) =>
          // Possibly log the exception or take alternative action
          ZIO.succeed(offset)
      }
    }
    .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
    .mapZIO(_.commit)
    .runDrain
}
```

## Deserialization in the consumer stream

In this approach we provide zio-kafka with the `Serdes.byteArray` serde (which is a pass-through serde) and do the
deserialization in the consumer stream. The deserialization can be done with regular ZIO operators.

This approach provides more freedom at the cost of having to write more code. It also allows for optimizations such as
operating on chunks of records (see [avoiding chunk-breakers](avoiding-chunk-breakers.md)), and more contextual failure
handling.

Here is an example:

```scala

def deserialize(value: Array[Byte]): ZIO[Any, Throwable, Message] = ???

Consumer.make(consumerSettings).flatMap { consumer =>
  consumer
    .plainStream(Subscription.topics("topic150"), Serde.byteArray, Serde.byteArray)
    .mapZIO { record => // ⚠️ see section about `mapZIO` below!
      val value: Array[Byte] = record.record.value()
      val offset: Offset = record.offset

      deserialize(value)
        // possible action to take if deserialization fails:
        .recoverWith(_ => someEffect(value))
        .flatMap(processMessage)
        .as(offset)
    }
    .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
    .mapZIO(_.commit)
    .runDrain
}
```

## A warning about `mapZIO`

Moved to [avoiding chunk-breakers](avoiding-chunk-breakers.md).

---

## Sharing a Consumer between multiple streams

Often in a single application, you want to consume from multiple Kafka topics and process each topic in a distinct way. With `zio-kafka` you can use a single `Consumer` instance for multiple streams from different topics. It is not only easier to create a single `Consumer` layer instead of one for each topic, but it may be more resource efficient as well. The underlying Apache Kafka consumer, its thread pool and communication with the Kafka brokers will be shared, resulting in less resource consumption compared to when you create a Consumer instance for every topic.

For each of the topics/patterns you subscribe to, you can define a dedicated stream to process the records, a dedicated `Deserializer` for the keys, and a dedicated `Deserializer` for the values. Other settings like poll interval and offset strategy are common to all subscriptions. For example, the value of the `max.poll.records` setting is the maximum number of records returned in each poll for all the subscriptions combined. If you need different settings for each topic/pattern, you need to create a `Consumer` instance per topic/pattern.

```scala

// Create a single Consumer instance
val consumerSettings: ConsumerSettings = ConsumerSettings(List("localhost:9092")).withGroupId("group")
val consumer: ZLayer[Any, Throwable, Consumer] = ZLayer.scoped(Consumer.make(consumerSettings))

// Define two streams, each for a different topic and key-value types
val stream1 = ZIO.serviceWithZIO[Consumer] { consumer =>
  consumer.plainStream(Subscription.topics("topic1"), Serde.string, Serde.string)
    .tap(cr => printLine(s"For topic1, got key: ${cr.record.key}, value: ${cr.record.value}"))
    .map(_.offset)
    .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
    .mapZIO(_.commit)
    .runDrain
}

val stream2 = ZIO.serviceWithZIO[Consumer] { consumer =>
  consumer.plainStream(Subscription.topics("topic2"), Serde.uuid, Serde.int)
    .tap(cr => printLine(s"For topic2, got key: ${cr.record.key}, value: ${cr.record.value}"))
    .map(_.offset)
    .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
    .mapZIO(_.commit)
    .runDrain
}

// Run both streams with the same consumer instance
(stream1 zipPar stream2).provideSomeLayer(consumer)
```

Each Stream can have a different lifetime. Subscriptions are tied to their Stream's scope, so when a Stream completes (successfully, via failure or interruption), the corresponding subscription is removed. 

Consumer sharing is only possible when using the same type of `Subscription`: `Topics`, `Pattern` or `Manual`. Mixing different types will result in a failed stream. Note that every `Topic` subscription can also be written as a `Pattern` subscription, so if you want to mix `Topic` and `Pattern` subscriptions, you can just use `Pattern`. 

If your subscriptions overlap, e.g. `Subscription.topics("topic1", "topic2")` and `Subscription.topics("topic2", "topic3")`, which stream will get the partitions of `topic2` is not deterministic.

:::note
Upon starting or ending subscriptions, Kafka will also reassign all partitions of the _other_ subscriptions. Record fetching will resume from the last committed offset, so records may be processed twice when not all offsets are committed yet. This works the same as for regular rebalancing. To minimize this issue, commit frequently or enable [rebalance safe commits](preventing-duplicates.md).
:::

---

## Transactions

> This document describes Kafka transactions for zio-kafka version 3.0.0 and later.

## What are Kafka Transactions?

[Kafka transactions](https://www.confluent.io/blog/transactions-apache-kafka/) are different from what you may know from
relational databases. In Kafka a transaction means that your program consumes some records, then produces some other
records (to the same broker) and then by committing the _consumed_ records the transaction is committed, and the
produced records become available to other consumers.

I can image that went a bit fast, here is a more gentle introduction with more background:
[Kafka Transactions Explained (Twice!)](https://www.warpstream.com/blog/kafka-transactions-explained-twice).

## A warning about consuming transactional records

By default, produced records become immediately visible for consumers, even if they are produced in a not yet committed
Kafka transaction! To prevent reading transactional records before they are committed, you need to configure every
consumer with the `isolation.level` property set to `read_committed`. For example as follows:

```scala
val settings = ConsumerSettings(bootstrapServers)
  .withGroupId(groupId)
  .withReadCommitted(true) // Only read committed records
.... etc.
```

## Producing transactional records

In order to produce records transactionally, we need a `Consumer` and a `TransactionalProducer` that will work together
to safely commit transactions.

On this page we assume you're using [ZIO service pattern](https://zio.dev/reference/service-pattern/) and that the
`Consumer` and `TransactionalProducer` are created as a layer.

First we create the `Consumer` layer.
Note that rebalance-safe-commits must be enabled (see the background section below for more information).
Since the default `maxRebalanceDuration` is quite high, you may also consider setting it to a lower duration, one in
which you are sure all records of a poll can be processed.

```scala
val bootstrapBrokers = List("localhost:9092")

val consumerLayer: ZLayer[Any, Throwable, Consumer] = ZLayer {
  val consumerSettings = ConsumerSettings(bootstrapBrokers)
    .withGroupId("somegroupid")
    .withRebalanceSafeCommits(true) // required!
    .withMaxRebalanceDuration(30.seconds)
  Consumer.make(consumerSettings)
}
```

Now we can create the `TransactionalProducer` layer.
Note that the producer connects (and must connect) to the same brokers as the consumer.

```scala
val producerLayer: ZLayer[Consumer, Throwable, TransactionalProducer] =
  ZLayer {
    // See text below for a better transactionalId
    val transactionalId = UUID.randomUUID().toString
    val producerSettings = ProducerSettings(bootstrapBrokers)
    val transactionalProducerSettings = TransactionalProducerSettings(producerSettings, transactionalId)

    for {
      consumer <- ZIO.service[Consumer]
      tp       <- TransactionalProducer.make(transactionalProducerSettings, consumer)
    } yield tp
  }
```

Notice that the `producerLayer` requires a `Consumer`. The producer coordinates with the consumer to make sure
that transactions are accepted by the brokers. The consumer does this by holding up rebalances until all records
consumed before the rebalance are committed by the producer. (Again, see the background section below for more
information.)

The `transactionalId` uniquely identifies a producer and is (ideally) retained across restarts. It allows the Kafka
brokers to identify zombies, producers that are accidentally still running. Records produced by a zombie are rejected.
For a more complete description see [Kafka’s transactional.id – your first guess is probably wrong. It’s actually all about zombies](https://community.ibm.com/community/user/integration/blogs/kim-clark1/2024/06/19/kafka-transactionalid).

Ideally, the `transactionalId` is stable and is provided by the environment. For example, the first producer uses a
`transactionalId` that ends with `0`, the second producer uses a `transactionalId` that ends with `1` and so on.

With these in place we can look at the application logic of consuming and producing. In this example we use the
`plainStream` method to consume records with a `Long` value and an `Int` key.

```scala
val consumedRecordsStream = consumer
  .plainStream(Subscription.topics("my-input-topic"), Serde.int, Serde.long)
```

For each consumed records, we then create a `ProducerRecord` with a `String` value containing `Copy of <input value>`.
Note, that we also keep the offset of the consumed record so that we can commit it later.

```scala
val producedRecordsStream = consumedRecordsStream
  .mapChunks { records: Chunk[CommittableRecord[Int, Long]] =>
    records.map { record =>
      val key: Int    = record.record.key()
      val value: Long = record.record.value()
      val newValue: String = "Copy of " + value.toString

      val producerRecord: ProducerRecord[Int, String] =
        new ProducerRecord("my-output-topic", key, newValue)
      (producerRecord, record.offset)
    }
  }
```

Typically, to optimize throughput, we want to produce records in batches. The underlying chunking structure of the
consumer stream is ideal for that because zio-kafka guarantees that each chunk in the stream contains records that
were fetched together from the broker. However, we need to be careful to retain the chunking structure. For example, we
should not use `.mapZIO` because it results in a stream where each chunk contains only a single item. Therefore, we use
`.mapChunksZIO` instead (see also [avoiding chunk-breakers](avoiding-chunk-breakers.md)).

These new records can now be produced. Let's build it up slowly.

```scala
producedRecordsStream
  .mapChunksZIO { recordsAndOffsets: Chunk[(ProducerRecord[Int, String], Offset)] =>
    ???
  }
```

Let's work on those question marks. We need to create a transaction.
Each zio-kafka transaction runs in a ZIO `Scope`; the transaction commits when the scope closes.

```scala
ZIO.scoped {
  ??? // transaction stuff
}
```

We also want to try to complete the transaction, even when the program is shutting down.
Therefore, we add `.uninterruptible` and get:

```scala
ZIO
  .scoped {
    ??? // transaction stuff
  }
  .uninterruptible
```

Now we can fill in the 'transaction stuff': we create the transaction and use it to produce some records.

```scala
for {
  tx <- transactionalProducer.createTransaction
  _  <- {
    val (records, offsets) = recordsAndOffsets.unzip
    tx.produceChunkBatch(
      records,
      Serde.int,
      Serde.string,
      OffsetBatch(offsets)
    )
  }
  // running inside `mapChunksZIO`, we need to return a Chunk.
  // Since we're not using the result, the empty chunk will do.
} yield Chunk.empty
```

Notice how we pass the offsets of the consumed records. Once the transaction completes, these are the offsets that will
be committed.

Putting it all together we get:

```scala

object Transactional extends ZIOAppDefault {

  private val topic = "transactional-test-topic"

  val bootstrapBrokers = List("localhost:9092")

  val consumerSettings = ZLayer.succeed {
    ConsumerSettings(bootstrapBrokers)
      .withGroupId("transactional-example-app")
      .withRebalanceSafeCommits(true) // required for transactional producing
      .withMaxRebalanceDuration(30.seconds)
  }

  val producerSettings = ZLayer.succeed {
    TransactionalProducerSettings(
      bootstrapServers = bootstrapBrokers,
      transactionalId = UUID.randomUUID().toString
    )
  }

  private val runConsumerStream: ZIO[Consumer & TransactionalProducer, Throwable, Unit] =
    for {
      consumer              <- ZIO.service[Consumer]
      transactionalProducer <- ZIO.service[TransactionalProducer]
      _                     <- ZIO.logInfo(s"Consuming messages from topic $topic...")
      _ <- consumer
        .plainStream(Subscription.topics(topic), Serde.int, Serde.long)
        .mapChunks { records: Chunk[CommittableRecord[Int, Long]] =>
          records.map { record =>
            val key: Int         = record.record.key()
            val value: Long      = record.record.value()
            val newValue: String = "Copy of " + value.toString

            val producerRecord: ProducerRecord[Int, String] =
              new ProducerRecord("my-output-topic", key, newValue)
            (producerRecord, record.offset)
          }
        }
        .mapChunksZIO { recordsAndOffsets: Chunk[(ProducerRecord[Int, String], Offset)] =>
          ZIO.scoped {
            for {
              _  <- ZIO.addFinalizer(ZIO.logInfo("Completing transaction"))
              tx <- transactionalProducer.createTransaction
              _ <- {
                val (records, offsets) = recordsAndOffsets.unzip
                tx.produceChunkBatch(
                  records,
                  Serde.int,
                  Serde.string,
                  OffsetBatch(offsets)
                )
              }
            } yield Chunk.empty
          }.uninterruptible
        }
        .runDrain
    } yield ()

  override def run: ZIO[Scope, Any, Any] =
    runConsumerStream
      .provide(
        consumerSettings,
        ZLayer.succeed(Consumer.NoDiagnostics),  // No longer needed in zio-kafka 3.x
        Consumer.live,
        producerSettings,
        TransactionalProducer.live
      )

}
```

## Technical background - How does it all work?

Kafka must ensure that each record is only processed once, even though a partition can be revoked, assigned to another
consumer, all while the original consumer knows nothing about this. In this situation the original consumer will happily
continue producing records even though another consumer is doing the same thing with the same records.

Kafka does this by validating the group epoch. The group epoch is a number that gets increased after every rebalance.
When the transactional producer commits a transaction with the offsets of consumed records, it also includes the group
epoch of when the records were consumed. If the broker detects a commit with a group epoch that is not equal to the
current epoch, it will reject the commit.

In addition, to prevent an old consumer from continuing to poll into the next epoch without participating in the
rebalance, consumers polling within the wrong group epoch are 'fenced'. A fenced consumer dies with a
`FencedInstanceIdException`.

### How zio-kafka helps

With zio-kafka you can use ZIO scopes to define the transaction boundaries. Within the scope you can produce records one
by one, all in 1 batch or in multiple batches. Once the scope closes, all collected offsets are committed.

In addition, zio-kafka prevents failed commits. The `TransactionalProducer` informs the `Consumer` which offsets were
committed. When a rebalance starts, the consumer holds it up until all records consumed so far are committed. (This
feature is called rebalance-safe-commits.) Because of this, every transaction commits in the same group epoch as the
epoch in which the records were consumed, and the brokers can accept the transaction.

When a zio-kafka consumer misses a rebalance and continues polling, there is nothing that can be done. It will die with
a `FencedInstanceIdException`. In these cases the consumer (or perhaps simpler, the whole program) should restart.

---

## Tutorial: How to Produce/Consume Data To/From Kafka Topics?

## Introduction

Kafka is a distributed, fault-tolerant, message-oriented event-store platform. It is used as a message broker for distributed applications. Zio-kafka is a library that provides a way to consume and produce data from Kafka topics, and it also supports the ability to have streaming consumers and producers.

In this tutorial, we will learn how to use zio-streams and zio-kafka to produce and consume data from Kafka topics.

## Running Examples

To access the code examples, you can clone the [ZIO Quickstarts](http://github.com/zio/zio-quickstarts) project:

```bash
$ git clone https://github.com/zio/zio-quickstarts.git
$ cd zio-quickstarts/zio-quickstart-kafka
```

And finally, run the application using sbt:

```bash
$ sbt run
```

## Adding Dependencies to The Project

In this tutorial, we will be using the following dependencies. So, let's add them to the `build.sbt` file:

```scala
libraryDependencies += Seq(
  "dev.zio" %% "zio"         % "2.1.25",
  "dev.zio" %% "zio-streams" % "2.1.25",
  "dev.zio" %% "zio-kafka"   % "3.3.0",
  "dev.zio" %% "zio-json"    % "0.9.1"
)
```

1. **Zio-kafka** is a ZIO native client for Apache Kafka. It provides a high-level streaming API on top of the Java client. So we can produce and consume events using the declarative concurrency model of zio-streams.

2. **Zio-streams** introduces a high-level API for working with streams of values. It is designated to work in a highly concurrent environment. It has seamless integration with ZIO, so we have the ability to use all the features of the ZIO along with the streams, e.g. `Scope`, `Schedule`, `ZLayer`, `Queue`, `Hub` etc. To learn more about zio-stream, there is a comprehensive section on that [here](https://zio.dev/reference/stream/).

3. **Zio-json** is a library to serialize and deserialize data from/to JSON data type. We will be using this library to serialize and deserialize data when reading and writing JSON data from/to Kafka topics.

## Setting Up The Kafka Cluster

Before we start, we need to set up a Kafka cluster. To start a kafka cluster for testing purposes we can use the following `docker-compose.yml` file:

```docker-compose
services:
  broker:
    image: apache/kafka:3.9.0
    container_name: broker
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_BROKER_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
```

Now we can run the `docker compose up -d` command to start the Kafka cluster:

```bash
$ docker compose up -d
```

## Writing a Simple Producer and Consumer Using ZIO Workflows

To write producers and consumers, using the zio-kafka library, we have two choices:

1. Using ZIO workflows
2. Using zio-streams workflows

In this section, we will try the first option.

### 1. Serializing and Deserializing Data

Before we can write a producer and consumer, let's talk about how data is stored in Kafka. Kafka is an event-store platform that stores records, records are key-value pairs as raw bytes. So a Kafka broker knows nothing about its records, it just appends the records to its internal log file.

To produce and consume data from Kafka, we need a way to serialize our data to a byte array and deserialize byte arrays to our data types. This is where the `Serde` data type comes in handy. A `Serde[R, A]` is a `Serializer` and `Deserializer` for values of type `A`, which can use the environment `R` to serialize and deserialize values.

Here is the simplified definition of the `Serde` data type:

```scala
trait Serde[-R, A] {
  def deserialize(data: Array[Byte]): RIO[R, A]
  def serialize(value: A)           : RIO[R, Array[Byte]]
}
```

The companion object of `Serde` trait contains a set of built-in serializers and deserializers for primitive types:

- `Serde.long`
- `Serde.int`
- `Serde.short`
- `Serde.float`
- `Serde.double`
- `Serde.boolean`
- `Serde.string`
- `Serde.byteArray`
- `Serde.byteBuffer`
- `Serde.uuid`

In this example, the type of the `key` is `Int` and the type of the `value` is `String`. So we can use the `Serde.int` for the `key` and the `Serde.string` for the `value`.

### 2. Creating a Producer

Now we can create a `Producer`:

```scala

val producer: ZIO[Scope, Throwable, Producer] =
  Producer.make(
    ProducerSettings(List("localhost:9092"))
  )
```

The `ProducerSettings` as constructed in this code fragment is already sufficient for many applications. However, more configuration options are available on `ProducerSettings`.

Notice that the created producer requires a `Scope` in the environment. When this scope closes, the producer closes its
connection with the Kafka cluster. An explicit scope can be created with the `ZIO.scoped` method.

### 3. Producing records

Zio-kafka has several producers that can be used to produce data on Kafka topics. In this example, we will be using the `Producer.produce` method:

```scala
trait Producer {
  def produce[R, K, V](
    topic: String,
    key: K,
    value: V,
    keySerializer: Serializer[R, K],
    valueSerializer: Serializer[R, V]
  ): RIO[R, RecordMetadata]
}
```

Notice how the type parameters of `produce` method ensure that the value's type, corresponds to the type produced by the value's `Serializer` (and the same for the key and the `Serializer` for the key).

Here is a helper function that shows how we could use the `produce` method:

```scala
def produceRecord(producer: Producer, topic: String, key: Long, value: String): RIO[Any, RecordMetadata] =
  producer.produce[Any, Long, String](
    topic = topic,
    key = key,
    value = value,
    keySerializer = Serde.long,
    valueSerializer = Serde.string
  )
```

### 4. Creating a Consumer

Zio-kafka provides several ways to consume data from Kafka topics. In this example, we will use the `Consumer.consumeWith` function.

The following helper function creates a ZIO workflow that if we run it, runs forever and consumes records from the given topic and prints them to the console, and then commits the offsets of the consumed records:

```scala
def consumeAndPrintEvents(consumer: Consumer, groupId: String, topic: String, topics: String*): RIO[Any, Unit] =
  consumer.consumeWith(
    settings = ConsumerSettings(BOOSTRAP_SERVERS).withGroupId(groupId),
    subscription = Subscription.topics(topic, topics: _*),
    keyDeserializer = Serde.long,
    valueDeserializer = Serde.string,
  ) { (k, v) =>
    Console.printLine(s"Consumed key: $k, value: $v").orDie
  }
```

For performance reasons, records are always consumed in batches. The `consumeWith` method commits the offsets of consumed records, as soon all records of a batch have been processed.

For more options see [consumer tuning](consumer-tuning.md).

### 5. The Complete Example

Now it's time to combine all the above steps to create a ZIO workflow that will produce and consume data from the Kafka cluster:

```scala

/** A simple app that produces and consumes messages from a kafka cluster
 * without using ZIO Streams.
 */
object SimpleKafkaApp extends ZIOAppDefault {
  private val BOOSTRAP_SERVERS = List("localhost:9092")
  private val KAFKA_TOPIC      = "hello"

  def run: ZIO[Scope, Throwable, Unit] = {
    for {
      c <- Consumer
        .consumeWith(
          settings =
            ConsumerSettings(BOOSTRAP_SERVERS).withGroupId("simple-kafka-app"),
          subscription = Subscription.topics(KAFKA_TOPIC),
          keyDeserializer = Serde.long,
          valueDeserializer = Serde.string
        ) { record =>
          Console.printLine(s"Consumed ${record.key()}, ${record.value()}").orDie
        }
        .fork

      producer <- Producer.make(ProducerSettings(BOOSTRAP_SERVERS))
      p <- Clock.currentDateTime
        .flatMap { time =>
          producer.produce[Any, Long, String](
            topic = KAFKA_TOPIC,
            key = time.getHour.toLong,
            value = s"$time -- Hello, World!",
            keySerializer = Serde.long,
            valueSerializer = Serde.string
          )
        }
        .schedule(Schedule.spaced(1.second))
        .fork

      _ <- (c <*> p).join
    } yield ()
  }

}
```

## Zio-kafka With Zio-streams

As we said before, to write producers and consumers using the zio-kafka library, we have two choices:

1. Using ZIO workflows
2. Using zio-streams workflows

In this section we show zio-kafka's seamless integration with zio-streams.

### 1. Streaming Producer API

With zio-kafka's `Producer.produceAll` we get a `ZPipeline`. It takes streams of `ProducerRecord[K, V]`, produces these records to a Kafka topic, and then returns a stream of `RecordMetadata`:

```scala
trait Producer {
  def produceAll[R, K, V](
    keySerializer: Serializer[R, K],
    valueSerializer: Serializer[R, V]
  ): ZPipeline[R, Throwable, ProducerRecord[K, V], RecordMetadata]
}
```

Here is an example that uses `Producer.produceAll`:

```scala
ZStream
  .fromIterator(Iterator.from(0), maxChunkSize = 50)
           // ZStream[Any, Throwable, Int]
  .mapChunksZIO { chunk =>
    chunk.map { i =>
      new ProducerRecord(topic, key = i, value = s"record $i") 
    }
  }        // ZStream[Any, Throwable, ProducerRecord]
  .via(producer.produceAll(Serde.int, Serde.string))
           // ZStream[Any, Throwable, RecordMetadata]
  .runDrain
```

For performance reasons, method `produceAll` produces records in batches, every chunk of the input stream results in a batch. Once all acknowledges are received, the next batch is produced.

See [this page](consuming-producing-and-committing-offsets.md) for tips that can increase producing throughput.

### 2. Creating a Consumer

When we use the streaming API we need to construct a Consumer:

```scala

val consumer: ZIO[Scope, Throwable, Consumer] =
  Consumer.make(
    ConsumerSettings(List("localhost:9092"))
      .withGroupId("streaming-kafka-app")
  )
```

Notice that the consumer requires a `Scope` in the environment. When this scope closes, the consumer closes its
connection with the Kafka cluster. An explicit scope can be created with the `ZIO.scoped` method.

For more options see [creating a consumer](creating-a-consumer.md) and [consumer tuning](consumer-tuning.md).

### 3. Streaming Consumer API

The `Consumer.plainStream` method gives a `ZStream` that, when run, consumes records from a Kafka topic and gives a stream of `CommittableRecord[K, V]`:

```scala
trait Consumer {
  def plainStream[R, K, V](
    subscription: Subscription,
    keyDeserializer: Deserializer[R, K],
    valueDeserializer: Deserializer[R, V]
  ): ZStream[R, Throwable, CommittableRecord[K, V]]
}
```

Parameter `subscription` indicates which topics and partitions should be consumed from. For example `Subscription.topics("events")` to consume from the `events` topic.

Parameters `keyDeserializer` and `valueDeserializer` are the `Serde` that will be used to deserialize the raw record bytes to whatever type you want them to be in.

To indicate that a record has been consumed successfully, and make sure that no other consumer in the same group will consume this record again (even when the application crashes and restarts), we need to commit the offset of the record. This is how it works: first get the `Offset` of the `CommittableRecord` with the `offset` method, then call `commit` on the returned `Offset`.

Here is an example:

```scala

val KAFKA_TOPIC = "my-topic"
val consumer: Consumer = ???

val c: ZIO[Any, Throwable, Unit] =
  consumer
    .plainStream(Subscription.topics(KAFKA_TOPIC), Serde.int, Serde.string)
    .tap(e => Console.printLine(e.value))
    .map(_.offset)      // Get the offset of the record
    .mapZIO(_.commit)   // Commit the offset
    .runDrain
```

While this works, it is not very performant. The problem with this approach is that we are committing offsets for each record separately. This causes a lot of overhead and slows down the consumption of records. To avoid this, we can aggregate offsets into batches and commit them all at once. This can be done by using the `ZStream#aggregateAsyncWithin` along with the `Consumer.collectOffsets` sink:

```scala

val KAFKA_TOPIC = "my-topic"
val consumer: Consumer = ???

val c: ZIO[Any, Throwable, Unit] =
  consumer
    .plainStream(Subscription.topics(KAFKA_TOPIC), Serde.int, Serde.string)
    .tap(e => Console.printLine(e.value))
    .map(_.offset)                           // Get the offset of the record
                                             // Group offsets in an OffsetBatch
    .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
    .mapZIO(_.commit)                        // Commit the batch of offsets
    .runDrain
```

Notice that because we are using `aggregateAsyncWithin`, the commits run asynchronously with the upstream of `aggregateAsyncWithin`. Every 100 milliseconds the next `OffsetBatch` is committed.

:::caution
Keeping the chunking structure intact is important.

In the example so far we have used `tap` to print the records as they are consumed. Unfortunately, methods like `tap` and `mapZIO` destroy the chunking structure and lead to much lower throughput. Please read [a warning about mapZIO](avoiding-chunk-breakers.md) for more details and alternatives.
:::

For more details see [consuming Kafka topics using ZIO Streams](consuming-kafka-topics-using-zio-streams.md).

### 4. The Complete Streaming Example

It's time to create a full working example of zio-kafka with zio-streams:

```scala

object StreamingKafkaApp extends ZIOAppDefault {
  private val BOOSTRAP_SERVERS = List("localhost:9092")
  private val KAFKA_TOPIC      = "streaming-hello"

  private val producerSettings = ProducerSettings(BOOSTRAP_SERVERS)
  private val consumerSettings =
    ConsumerSettings(BOOSTRAP_SERVERS).withGroupId("streaming-kafka-app")

  def run: ZIO[Any, Throwable, Unit] = {
    val p: ZIO[Any, Throwable, Unit] =
      ZIO.scoped {
        for {
          producer <- Producer.make(producerSettings)
          _ <- ZStream
            .repeatZIO(Clock.currentDateTime)
            .schedule(Schedule.spaced(1.second))
            .map { time =>
              new ProducerRecord(
                KAFKA_TOPIC,
                time.getMinute,
                s"$time -- Hello, World!"
              )
            }
            .via(producer.produceAll(Serde.int, Serde.string))
            .runDrain
        } yield ()
      }

    val c: ZIO[Any, Throwable, Unit] =
      ZIO.scoped {
        for {
          consumer <- Consumer.make(consumerSettings)
          _ <- consumer
            .plainStream(
              Subscription.topics(KAFKA_TOPIC),
              Serde.int,
              Serde.string
            )
            // Do not use `tap` in throughput sensitive applications because it
            // destroys the chunking structure and leads to lower performance.
            // See the previous section for more info.
            .tap(r => Console.printLine("Consumed: " + r.value))
            .map(_.offset)
            .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
            .mapZIO(_.commit)
            .runDrain
        } yield ()
      }

    p <&> c
  }

}
```

## Producing and Consuming JSON Data

Until now, we learned how to work with simple types like `Int`, `String`, and how to use their `Serde` instances to serialize and deserialize the data.

In this section, we are going to learn how to serialize/deserialize user-defined data types (like case classes), and in particular how to use the JSON format. We also will learn how to use the `Serde` built-in instances to create more complex `Serde` instances.

### 1. Writing Custom Serializer and Deserializer

In zio-kafka all the built-in serializers/deserializers are instances of the `Serde` trait. All `Serde`s offers several combinators with which we can create new `Serde`s. We take a closer look at 2 of them:

- Method `inmap` transforms the `Serde` with pure transformations from the source type to the target type and back.
- Method `inmapZIO` transforms the `Serde` with effectful transformations from the source type to the target type and back. As it accepts effectful transformations, we can encode any parsing failure with a `ZIO` workflow.

In this example, we use `inmap` to transform the build-in `Serde.long` to a `Serde[Any, Instant]`. The `inmap` method gets 2 pure functions, the first converts from `Long` to `Instant`, the second from `Instant` to `Long`:

```scala

val instantSerde: Serde[Any, Instant] =
  Serde.long.inmap[Instant](Instant.ofEpochMilli)(_.toEpochMilli)
```

In the following example, we will use `inmapZIO` to transform the built-in `Serde.string` to a `Serde[Any, Event]` where `Event` is a case class that is serialized to/deserialized from a String containing JSON.

Let's say we have the `Event` case class with the following fields:

```scala

case class Event(
  uuid: UUID,
  timestamp: OffsetDateTime,
  message: String
)
```

First, we need to define a JSON decoder and encoder for it:

```scala

object Event {
  implicit val encoder: JsonEncoder[Event] =
    DeriveJsonEncoder.gen[Event]

  implicit val decoder: JsonDecoder[Event] =
    DeriveJsonDecoder.gen[Event]
}
```

Then we need to create a `Serde` for the `Event` type. To convert `Event` to JSON and back, we will use the zio-json library. To define a `Serde` for the `Event` type, we will use the `Serde.string.inmapZIO` combinator:

```scala

object EventKafkaSerde {
  val event: Serde[Any, Event] =
    Serde.string.inmapZIO[Any, Event](s =>
      ZIO
        .fromEither(s.fromJson[Event])
        .mapError(e => new RuntimeException(e))
    )(r => ZIO.succeed(r.toJson))
}
```

As we can see, we use the `String#fromJson` to convert the string to an `Event` object, and we also encode any parsing failure with a `RuntimeException` in the `ZIO` workflow.

See [zio-kafka serialization and deserialization](serialization-and-deserialization.md) for more details.

### 2. The Complete JSON Streaming Example

Here is a full working example of producing and consuming JSON data with zio-kafka, zio-streams and zio-json:

```scala

/** This is the data we will be sending to Kafka in JSON format. */
case class Event(uuid: UUID, timestamp: OffsetDateTime, message: String)

/** A zio-json encoder/decoder for [[Event]]. */
object Event {
  implicit val encoder: JsonEncoder[Event] =
    DeriveJsonEncoder.gen[Event]

  implicit val decoder: JsonDecoder[Event] =
    DeriveJsonDecoder.gen[Event]
}

/** A zio-kafka serializer/deserializer for [[Event]]. */
object EventKafkaSerde {
  val event: Serde[Any, Event] =
    Serde.string.inmapZIO[Any, Event](s =>
      ZIO
        .fromEither(s.fromJson[Event])
        .mapError(e => new RuntimeException(e))
    )(r => ZIO.succeed(r.toJson))
}

object JsonStreamingKafkaApp extends ZIOAppDefault {
  private val BOOSTRAP_SERVERS = List("localhost:9092")
  private val KAFKA_TOPIC      = "json-streaming-hello"

  def run: ZIO[Any, Throwable, Unit] = {
    val p: ZIO[Any, Throwable, Unit] =
      ZIO.scoped {
        for {
          producer <- Producer.make(ProducerSettings(BOOSTRAP_SERVERS))
          _ <- ZStream
            .repeatZIO(Random.nextUUID <*> Clock.currentDateTime)
            .schedule(Schedule.spaced(1.second))
            .map { case (uuid, time) =>
              new ProducerRecord(
                KAFKA_TOPIC,
                time.getMinute,
                Event(uuid, time, "Hello, World!")
              )
            }
            .via(producer.produceAll(Serde.int, EventKafkaSerde.event))
            .runDrain
        } yield ()
      }

    val c: ZIO[Any, Throwable, Unit] =
      ZIO.scoped {
        for {
          consumer <- Consumer.make(
            ConsumerSettings(BOOSTRAP_SERVERS).withGroupId("streaming-kafka-app")
          )
          _ <- consumer
            .plainStream(
              Subscription.topics(KAFKA_TOPIC),
              Serde.int,
              EventKafkaSerde.event
            )
            .tap { r =>
              val event: Event = r.value
              Console.printLine(
                s"Event ${event.uuid} was sent at ${event.timestamp} with message ${event.message}"
              )
            }
            .map(_.offset)
            .aggregateAsyncWithin(Consumer.collectOffsets, Schedule.fixed(100.millis))
            .mapZIO(_.commit)
            .runDrain
        } yield ()
      }

    p <&> c
  }

}
```

## Conclusion

In this tutorial we first learned how to create a producer and consumer for Kafka using the ZIO workflow with zio-kafka. Then we learned how to do the same with zio-streams. We also learned how to create a custom serializer and deserializer for the Kafka records and how to produce and consume JSON data using the zio-json library.

All the source code associated with this article is available on the [ZIO Quickstart](http://github.com/zio/zio-quickstarts) project on GitHub.

More information:

- [zio-kafka](https://zio.dev/zio-kafka/)
- [zio-streams](https://zio.dev/reference/stream/)
- [zio-json](https://zio.dev/zio-json/)

---

## Writing Tests with the `zio-kafka-testkit` library

Zio-kafka provides a `zio-kafka-testkit` library to help you test your code using zio-kafka.

To add it in your project, add the following dependency in your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-kafka-testkit" % "<latest-version>" % Test
```

Let's study some examples of tests you can write with the `zio-kafka-testkit` and `zio-test` and let's see what this
library provides you.

## Testing a producer

```scala

object ProducerSpec extends ZIOSpecDefault {
  override def spec: Spec[TestEnvironment & Scope, Any] =
    (
      suite("Producer test suite")(
        test("minimal example") {
          for {
            producer <- KafkaTestUtils.makeProducer // (1)
            _ <- producer.produce(new ProducerRecord("topic", "boo", "baa"), Serde.string, Serde.string) // (3)
          } yield assertCompletes // (3)
        }
        // ... more tests ...
      )
        .provideSomeShared[Scope](Kafka.embedded) // (4) Provide an embedded Kafka instance, shared in the entire suite
      ) @@ withLiveClock @@ timeout(2.minutes) // (5)
}
```

The main entry points for zio-kafka test kit are the classes `KafkaTestUtils` and `Kafka`.

(1) A producer is created with lots of default settings. It knows how to connect to the Kafka broker by getting a
`Kafka` instance from the environment (see (4)).

(2) The producer is used to produce a record.

Take a look at the `produce*` methods in `KafkaTestUtils` when you need to produce a lot of records.

(3) The `assertCompletes` assertion from zio-test is used to check that the effect completes without errors.

(4) Provide a `Kafka` instance as a layer shared between all the tests. In this test we provide an **Embedded Kafka**,
meaning that a complete Kafka cluster is started and stopped inside the current JVM.

See further below for more options, such as connecting to an external Kafka cluster.

Note that any services that are _not_ provided at this line, must be given as type parameter to `provideSomeShared`. In
this case we are not providing a `Scope` (it is provided by the test framework).

(5) Zio-kafka requires a live clock.

### Producer layer

In this example above, we decided to make a new `Producer` in each test. We could have decided to share one instance of
`Producer` between all the tests of this suite. For this purpose we can get a Producer layer using
`KafkaTestUtils.producer`. The result is:

```scala
suite("producer test suite")(
  // ... tests ...
).provideSomeShared[Scope](Kafka.embedded, KafkaTestUtils.producer)
```

### More Kafka sharing options

Kafka is slow to start, so it is better to only start it once and share it between all tests of the suite.

However, if we insist on a separate embedded Kafka _per test_, we can provide the `Kafka.embedded` layer with
`provideSome` (instead of `provideSomeShared`), which looks like this:

```scala
suite("producer test suite")(
  // ... tests ...
).provideSome[Scope](Kafka.embedded)
```

We can also share one embedded `Kafka` instance between different test suites (i.e. between different test files) by
mixing in the `ZIOSpecWithKafka` trait, this looks like this:

```scala
object ProducerSpec extends ZIOSpecWithKafka { // (1)
  override def spec: Spec[TestEnvironment & Scope & Kafka, Any] =
    (
      suite("Producer test suite")(
        // ... tests ...
      )
        .provideSome[Scope & Kafka](/* ...other layers... */) // (2)
      )
}
```

(1) Note the `ZIOSpecWithKafka` trait usage here instead of `ZIOSpecDefault`.

(2) When we need to provide additional layers with `provideSome` or `provideSomeShared`, both `Scope` and `Kafka` are
now provided by the test framework. Therefore, we need to include both in the type parameter.

More details about this `ZIOSpecWithKafka` trait [below](#ziospecwithkafka-trait).

## Considerations for sharing Kafka between tests

Zio tests by default all run concurrently. Tests may interfere with each other through the shared Kafka resource. The
best way to prevent interference is by making sure each test uses a different topic. If this is not feasible, we can use
the `sequential` aspect from zio-test to run the tests one by one.

```scala

suite("test suite")(
 // ... tests ...
) @@ sequential
```

## Testing a consumer

```scala

object ConsumerSpec extends ZIOSpecDefault {

  override def spec: Spec[TestEnvironment & Scope, Any] =
    (
      suite("Consumer test suite")(
        test("minimal example") {
          // Records (as key/value pairs) to produce and consume
          val kvs: List[(String, String)] = (1 to 5).toList.map(i => (s"key-$i", s"msg-$i"))
          for {
            topic <- Random.nextUUID.map("topic-" + _.toString) // (1)
            client <- Random.nextUUID.map("client-" + _.toString)
            group <- Random.nextUUID.map("group-" + _.toString)

            _ <- KafkaTestUtils.createCustomTopic(topic, partitionCount = 3) // (2)

            producer <- KafkaTestUtils.makeProducer
            _ <- KafkaTestUtils.produceMany(producer, topic, kvs) // (3)

            consumer <- KafkaTestUtils.makeConsumer(clientId = client, groupId = Some(group)) // (4)
            records <- consumer
              .plainStream(Subscription.topics(topic), Serde.string, Serde.string)
              .take(5)
              .runCollect // (5)
            consumed = records.map(r => (r.record.key, r.record.value)).toList
          } yield assert(consumed)(hasSameElements(kvs))
        }
      )
        .provideSomeShared[Scope](Kafka.embedded) // (6)
      ) @@ withLiveClock @@ timeout(2.minutes)
}
```

(1) Using random values for these parameters is important to avoid conflicts between tests as we share one Kafka
instance between all the tests of the suite.

(2) Here we create a topic with 3 partitions. Note that by default Kafka auto-creates topics. Therefore, creating
a topic explicitly is only needed when we want to control the number of partitions, or when your Kafka cluster does
not allow auto-created topics.

(3) A producer is constructed and 5 records are produced with `KafkaTestUtils.produceMany`.

(4) A consumer is constructed with `KafkaTestUtils.makeConsumer`.

(5) The consumer reads 5 records from the topic.

Be careful with the `take` method, due to pre-fetching, the consumer may have fetched more from the topic than expected.

(6) Similarly as in the producer test above, we provide the `Kafka.embedded` layer.

## More consumer options

If we want to share a consumer between tests, we can use the `KafkaTestUtils.consumer` layer and provide it with the
`provideSomeShared` method (see the producer example above for more details).

## Utilities provided by the `zio-kafka-testkit` library

### `Kafka` service

This trait represents a Kafka instance in your tests. It is used to provide the bootstrap servers to the constructor
methods in `KafkaTestUtils`.

```scala
trait Kafka {
  def bootstrapServers: List[String]

  def stop(): UIO[Unit]
}
```

The companion object provides a few layers to provide a Kafka instance in your tests:

```scala
object Kafka {
  /**
   * Creates an in-memory Kafka instance with a random port.
   */
  val embedded: ZLayer[Any, Throwable, Kafka]

  /**
   * Will connect to a Kafka instance running on localhost:9092 (with Docker, for example).
   */
  val local: ULayer[Kafka]

  /**
   * Creates an in-memory Kafka instance with a random port and SASL authentication configured.
   */
  val saslEmbedded: ZLayer[Any, Throwable, Kafka.Sasl]

  /**
   * Creates an in-memory Kafka instance with a random port and SSL authentication configured.
   */
  val sslEmbedded: ZLayer[Any, Throwable, Kafka]
}
```

The in-memory Kafka instances are created using [embedded-kafka](https://github.com/embeddedkafka/embedded-kafka).

### `KafkaTestUtils` utilities

This object provides several utilities to simplify writing your tests, like constructing a `Producer`, a `Consumer`,
or an `AdminClient`.

It also provides several functions to produce records, and more.

Each utility function is documented in the source code. Please have a look at the source code for more details.     
You can also look at `zio-kafka` tests in the `zio-kafka-test` module to have examples on how to use these utilities.

### `ZIOSpecWithKafka` trait

This trait can be used if you want to share one Kafka instance between different test suites.    
This allows you to speed up your tests by booting a Kafka instance only once for all your test suites using this trait.

Usage example:

```scala
// In `src/test/scala/io/example/producer/ProducerSpec.scala`
object ProducerSpec extends ZIOSpecWithKafka { // Note `ZIOSpecWithKafka`
  override def spec: Spec[TestEnvironment & Scope & Kafka, Any] =
    suite("Producer test suite")(
      // ... tests ...
    ) @@ timeout(2.minutes)
}

// In `src/test/scala/io/example/consumer/ConsumerSpec.scala`
object ConsumerSpec extends ZIOSpecWithKafka { // Note `ZIOSpecWithKafka`
  override def spec: Spec[TestEnvironment & Scope & Kafka, Any] =
    suite("Consumer test suite")(
      // ... tests ...
    ) @@ timeout(2.minutes)
}
```

This is a capability offered by ZIO2.    
See related zio-test documentation: https://zio.dev/reference/test/sharing-layers-between-multiple-files/

### `KafkaRandom` trait

The `KafkaRandom` trait provides a few methods to generate random values.
To use it, you need to mix it in your test suite, like this:

```scala

object MyServiceSpec extends ZIOSpecDefault with KafkaRandom {
  // Required when mixing in the `KafkaRandom` trait
  // The best is to use a different prefix for each test suite
  override def kafkaPrefix: String = "my-service"

  override def spec: Spec[TestEnvironment & Scope, Any] =
    suite("MyService")(
      test("minimal example") {
        for {
          topic    <- randomTopic  // Comes from `KafkaRandom`
          clientId <- randomClient // Comes from `KafkaRandom`
          groupId  <- randomGroup  // Comes from `KafkaRandom`
          // ... 
        } yield assertCompletes
      }
    ).provideSomeShared[Scope](Kafka.embedded)
}
```

---

## Introduction to ZIO Lambda

A ZIO-based AWS Custom Runtime compatible with GraalVM Native Image.

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-lambda/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-lambda_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-lambda_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-lambda_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-lambda_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-lambda-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-lambda-docs_2.13) [![ZIO Lambda](https://img.shields.io/github/stars/zio/zio-lambda?style=social)](https://github.com/zio/zio-lambda)

## Installation

```scala
libraryDependencies += "dev.zio" %% "zio-json" % "0.6.2"
libraryDependencies += "dev.zio" %% "zio-lambda" % "1.0.5"

// Optional dependencies
libraryDependencies += "dev.zio" %% "zio-lambda-event"    % "1.0.5"
libraryDependencies += "dev.zio" %% "zio-lambda-response" % "1.0.5"
```

## Usage

Create your Lambda function by providing it to `ZLambdaRunner.serve(...)` method.

```scala

object SimpleHandler extends ZIOAppDefault {

   def app(request: KinesisEvent, context: Context) = for {
      _ <- printLine(event.message)
   } yield "Handler ran successfully"

   override val run =
      ZLambdaRunner.serve(app)
}
```

zio-lambda depends on [**zio-json**](https://github.com/zio/zio-json) for decoding any event you send to it and enconding any response you send back to the Lambda service. You can either create your own data types or use the ones that are included in **zio-lambda-event** and **zio-lambda-response**.

The last step is to define the way your function will be invoked. There are three ways, detailed below:

## Lambda layer

Upload zio-lambda as a [Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html)
Each release will contain a zip file ready to be used as a lambda layer) and your function. Instructions coming soon!

## Direct deployment of native image binary

1. Create an AWS Lambda function and choose the runtime where you provide your own bootstrap on Amazon Linux 2

   ![create-lambda](https://user-images.githubusercontent.com/14280155/164102664-3686e415-20be-4dd9-8979-ea6098a7a4b9.png)
2. Run `sbt GraalVMNativeImage/packageBin`, we'll find the binary present under the `graalvm-native-image` folder:

   ![binary-located](https://user-images.githubusercontent.com/14280155/164103337-6645dfeb-7fc4-4f7f-9b13-8005b0cddead.png)

3. Create the following bootstap file (which calls out to the binary) and place it in the same directory alongside the binary:
    ```bash
    #!/usr/bin/env bash
    
    set -euo pipefail
    
    ./zio-lambda-example
    ```

   ![bootstrap-alongside-native-binary](https://user-images.githubusercontent.com/14280155/164103935-0bf7a6cb-814d-4de1-8fa1-4d0d54fb6e88.png)

4. Now we can zip both these files up:
    ```log
    > pwd
    /home/cal/IdeaProjects/zio-lambda/lambda-example/target/graalvm-native-image                                                                                                                                
    > zip upload.zip bootstrap zio-lambda-example
    ```

5. Take `upload.zip` and upload it to AWS Lambda and test your function:

   ![lambda-ui](https://user-images.githubusercontent.com/14280155/164104747-039ec584-d3e2-4b47-884d-ff88977e2b53.png)

6. Test everything out to make sure everything works:

   ![test-ui](https://user-images.githubusercontent.com/14280155/164104858-a720ac55-b9bb-47ec-af70-c4bd5eb5bed3.png)

## Deployment of native image binary in a Docker container

Following the steps from `Direct deployment of native image binary` to produce your native image binary, we can package
up the native binary into a Docker image and deploy it like that to AWS Lambda.

```Dockerfile
FROM gcr.io/distroless/base-debian12
COPY lambda-example/target/graalvm-native-image/zio-lambda-example /app/zio-lambda-example
CMD ["/app/zio-lambda-example"]
```

**NOTE:** This Dockerfile is meant to build the lambda-example located in the zio-lambda project and the Dockerfile is
placed in the zio-lambda-repository. You will need to adjust this Dockerfile to match your project needs.

Now we can build and tag the Docker image:

```shell
docker build -t native-image-binary .
```

Take this image and push it to AWS ECR:

```bash
pass=$(aws ecr get-login-password --region us-east-1) 
docker login --username AWS --password $pass <your_AWS_ECR_REPO>   
docker tag native-image-binary <your-particular-ecr-image-repository>:<your-tag>
docker push <your-particular-ecr-image-repository>:<your-tag>
```

Here is an example:

![image-uploaded](https://user-images.githubusercontent.com/14280155/164120591-68a78d19-c56b-4793-96b8-cfe567443063.png)

Create a Lambda function and choose container image:

![lambda-create-container-image](https://user-images.githubusercontent.com/14280155/164120637-9c827736-26a8-4c65-92d4-2919157bbda6.png)

![image](https://user-images.githubusercontent.com/14280155/164120764-2c736a46-29e3-488c-ba6a-e2b69ef51792.png)

Please note that because you incur the overhead of your native binary residing within a Docker container, there is more overhead than the other approach of deploying the binary straight to AWS Lambda

---

## Console Logger

logger layer with configuration from config provider:

```scala

val configProvider: ConfigProvider = ???

val logger = Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> consoleLogger()
```

logger layer with given configuration:

```scala

val config: ConsoleLoggerConfig = ???

val logger = Runtime.removeDefaultLoggers >>> consoleLogger(config)
```

there are other versions of console loggers:
* `zio.logging.consoleJsonLogger` - output in json format
* error console:
  * `zio.logging.consoleErrLogger` - output in string format
  * `zio.logging.consoleErrJsonLogger` - output in json format

## Configuration

the configuration for console logger (`zio.logging.ConsoleLoggerConfig`) has the following configuration structure:

```
logger {
  # log format, default value: LogFormat.default
  format = "%label{timestamp}{%fixed{32}{%timestamp}} %label{level}{%level} %label{thread}{%fiberId} %label{message}{%message} %label{cause}{%cause}"
  
  # log filter
  filter {
    # see filter configuration
    rootLevel = INFO
  }
}
```

see also [log format configuration](formatting-log-records.md#log-format-configuration) and [filter configuration](log-filter.md#configuration)

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### Colorful Console Logger With Log Filtering

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object ConsoleColoredApp extends ZIOAppDefault {

  val configString: String =
    s"""
       |logger {
       |
       |  format = "%highlight{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %cause}"
       |
       |  filter {
       |    mappings {
       |      "zio.logging.example.LivePingService" = "DEBUG"
       |    }
       |  }
       |}
       |""".stripMargin

  val configProvider: ConfigProvider = TypesafeConfigProvider.fromHoconString(configString)

  override val bootstrap: ZLayer[Any, Config.Error, Unit] =
    Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> consoleLogger()

  private def ping(address: String): URIO[PingService, Unit] =
    PingService
      .ping(address)
      .foldZIO(
        e => ZIO.logErrorCause(s"ping: $address - error", Cause.fail(e)),
        r => ZIO.logInfo(s"ping: $address - result: $r")
      )

  override def run: ZIO[Scope, Any, ExitCode] =
    (for {
      _ <- ping("127.0.0.1")
      _ <- ping("x8.8.8.8")
    } yield ExitCode.success).provide(LivePingService.layer)

}
```

Expected console output:

```
2023-03-05T12:24:05+0100 DEBUG   [zio-fiber-4] zio.logging.example.LivePingService:37 ping: /127.0.0.1 
2023-03-05T12:24:05+0100 INFO    [zio-fiber-4] zio.logging.example.ConsoleColoredApp:43 ping: 127.0.0.1 - result: true 
2023-03-05T12:24:05+0100 ERROR   [zio-fiber-4] zio.logging.example.LivePingService:36 ping: x8.8.8.8 - invalid address error Exception in thread "zio-fiber-4" java.net.UnknownHostException: x8.8.8.8: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1378)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at java.base/java.net.InetAddress.getByName(InetAddress.java:1256)
	at zio.logging.example.LivePingService.$anonfun$ping$2(PingService.scala:35)
	at zio.ZIOCompanionVersionSpecific.$anonfun$attempt$1(ZIOCompanionVersionSpecific.scala:100)
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1519)
	at java.net.InetAddress.getAllByName(InetAddress.java:1378)
	at java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at java.net.InetAddress.getByName(InetAddress.java:1256)
	at zio.logging.example.LivePingService.ping(PingService.scala:35)
	at zio.logging.example.LivePingService.ping(PingService.scala:36)
	at zio.logging.example.LivePingService.ping(PingService.scala:33)
	at zio.logging.example.ConsoleColoredApp.ping(ConsoleColoredApp.scala:41)
	at zio.logging.example.ConsoleColoredApp.run(ConsoleColoredApp.scala:49)
	at zio.logging.example.ConsoleColoredApp.run(ConsoleColoredApp.scala:50)
2023-03-05T12:24:05+0100 ERROR   [zio-fiber-4] zio.logging.example.ConsoleColoredApp:42 ping: x8.8.8.8 - error Exception in thread "zio-fiber-" java.net.UnknownHostException: x8.8.8.8: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1378)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at java.base/java.net.InetAddress.getByName(InetAddress.java:1256)
	at zio.logging.example.LivePingService.$anonfun$ping$2(PingService.scala:35)
	at zio.ZIOCompanionVersionSpecific.$anonfun$attempt$1(ZIOCompanionVersionSpecific.scala:100)
```

### JSON Console Logger 

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object ConsoleJsonApp extends ZIOAppDefault {

  final case class User(firstName: String, lastName: String) {
    def toJson: String = s"""{"first_name":"$firstName","last_name":"$lastName"}""".stripMargin
  }

  private val userLogAnnotation = LogAnnotation[User]("user", (_, u) => u, _.toJson)
  private val uuid              = LogAnnotation[UUID]("uuid", (_, i) => i, _.toString)

  val logFormat =
    "%label{timestamp}{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ}} %label{level}{%level} %label{fiberId}{%fiberId} %label{message}{%message} %label{cause}{%cause} %label{name}{%name} %kvs"
  
  val configProvider: ConfigProvider = ConfigProvider.fromMap(
    Map(
      "logger/format"           -> logFormat,
      "logger/filter/rootLevel" -> LogLevel.Info.label
    ),
    "/"
  )

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> consoleJsonLogger()

  private val uuids = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.foreachPar(uuids) { uId =>
        {
          ZIO.logInfo("Starting operation") *>
            ZIO.sleep(500.millis) *>
            ZIO.logInfo("Stopping operation")
        } @@ userLogAnnotation(User("John", "Doe")) @@ uuid(uId)
      } @@ LogAnnotation.TraceId(traceId)
      _       <- ZIO.logInfo("Done")
    } yield ExitCode.success

}
```

Expected console output:

```
{"timestamp":"2023-03-08T19:36:04+0100","level":"INFO","fiberId":"zio-fiber-5","message":"Starting operation","name":"zio.logging.example.ConsoleJsonApp","trace_id":"f30c2e89-006c-4f7c-adfa-497e3bba1b98","uuid":"3b971312-cb3e-420e-8c12-7fb96add2224","user":{"first_name":"John","last_name":"Doe"}}
{"timestamp":"2023-03-08T19:36:04+0100","level":"INFO","fiberId":"zio-fiber-6","message":"Starting operation","name":"zio.logging.example.ConsoleJsonApp","trace_id":"f30c2e89-006c-4f7c-adfa-497e3bba1b98","uuid":"93b87ff0-410a-4fc1-9cc7-b7eb7c655f60","user":{"first_name":"John","last_name":"Doe"}}
{"timestamp":"2023-03-08T19:36:05+0100","level":"INFO","fiberId":"zio-fiber-5","message":"Stopping operation","name":"zio.logging.example.ConsoleJsonApp","trace_id":"f30c2e89-006c-4f7c-adfa-497e3bba1b98","uuid":"3b971312-cb3e-420e-8c12-7fb96add2224","user":{"first_name":"John","last_name":"Doe"}}
{"timestamp":"2023-03-08T19:36:05+0100","level":"INFO","fiberId":"zio-fiber-6","message":"Stopping operation","name":"zio.logging.example.ConsoleJsonApp","trace_id":"f30c2e89-006c-4f7c-adfa-497e3bba1b98","uuid":"93b87ff0-410a-4fc1-9cc7-b7eb7c655f60","user":{"first_name":"John","last_name":"Doe"}}
{"timestamp":"2023-03-08T19:36:05+0100","level":"INFO","fiberId":"zio-fiber-4","message":"Done","name":"zio.logging.example.ConsoleJsonApp"}
```

---

## File Logger

logger layer with configuration from config provider:

```scala

val configProvider: ConfigProvider = ???

val logger = Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> fileLogger()
```

logger layer with given configuration:

```scala

val config: FileLoggerConfig = ???

val logger = Runtime.removeDefaultLoggers >>> fileLogger(config)
```

there are other versions of file loggers:
* `zio.logging.fileJsonLogger` - output in json format
* file async:
    * `zio.logging.fileAsynLogger` - output in string format
    * `zio.logging.fileAsyncJsonLogger` - output in json format

## Configuration

the configuration for file logger (`zio.logging.FileLoggerConfig`) has the following configuration structure:

```
logger {
  # log format, default value: LogFormat.default
  format = "%label{timestamp}{%fixed{32}{%timestamp}} %label{level}{%level} %label{thread}{%fiberId} %label{message}{%message} %label{cause}{%cause}"
  
  # URI to file
  path = "file:///tmp/console_app.log"
    
  # charset configuration, default value: UTF-8
  charset = "UTF-8"

  # auto flush batch size, default value: 1
  autoFlushBatchSize = 1

  # if defined, buffered writer is used, with given buffer size
  # bufferedIOSize = 8192
  
  # if defined, file log rolling policy is used
  rollingPolicy {
    type = TimeBasedRollingPolicy # time based file rolling policy based on date - currently only this one is supported
  }
  
  # log filter
  filter {
    # see filter configuration
    rootLevel = INFO
  }
}
```

see also [log format configuration](formatting-log-records.md#log-format-configuration) and [filter configuration](log-filter.md#configuration)

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### File Logger 

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object FileApp extends ZIOAppDefault {

  val configString: String =
    s"""
       |logger {
       |  format = "%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %cause"
       |  path = "file:///tmp/file_app.log"
       |  rollingPolicy {
       |    type = TimeBasedRollingPolicy
       |  }
       |}
       |""".stripMargin

  val configProvider: ConfigProvider = TypesafeConfigProvider.fromHoconString(configString)

  override val bootstrap: ZLayer[Any, Config.Error, Unit] =
    Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> fileLogger()

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _ <- ZIO.logInfo("Start")
      _ <- ZIO.fail("FAILURE")
      _ <- ZIO.logInfo("Done")
    } yield ExitCode.success
}
```

Expected file name: `file_app-2023-03-05.log`

Expected file content:

```
2023-03-05T12:37:31+0100 INFO    [zio-fiber-4] zio.logging.example.FileApp:40 Start
2023-03-05T12:37:31+0100 ERROR   [zio-fiber-0] zio-logger:  Exception in thread "zio-fiber-4" java.lang.String: FAILURE
        at zio.logging.example.FileApp.run(FileApp.scala:41)
```

---

## Formatting Log Records

A `LogFormat` represents a DSL to describe the format of text log messages.

[//]: # (TODO: make snippet type-checked using mdoc)

```scala

val myLogFormat = timestamp.fixed(32) |-| level |-| label("message", quoted(line))
val myConsoleLogger = console(myLogFormat)
```

`LogFormat.filter` returns a new log format that produces the same result, if `LogFilter` is satisfied.

```scala

LogFormat.label("cause", LogFormat.cause).filter(LogFilter.causeNonEmpty)
```

## Log Format Configuration

String representation of LogFormat:

| format                                   | description                                                                                                                              |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| %timestamp\{date-format\}%timestamp | Timestamp, where date format is `java.time.format.DateTimeFormatter`,example: `%timestamp{yyyy-MM-dd'T'HH:mm:ssZ}`                  |
| %fiberId                                 | Fiber Id                                                                                                                                 |
| %fixed\{size}\{format\}                  | Fixed size for format,example: `%fixed{7}{%level}`                                                                                  |
| %label\{name}\{format\}                  | Labeled format,example: `%label{abcSpan}{%span{abc}}`                                                                               |
| %color\{LogColor}\{format\}              | Colored format, log color values: `RED`, `BLUE`, `YELLOW`, `CYAN`, `GREEN`, `MAGENTA`, `WHITE`, example: `%color{CYAN}{%timestamp}` |
| %level                                   | Log level                                                                                                                                |
| %name                                    | Logger name (from logger name annotation or Trace)                                                                                       |
| %line                                    | Log/trace line (from Trace)                                                                                                              |
| %message                                 | Log message                                                                                                                              |
| %cause                                   | Cause                                                                                                                                    |
| %kvs                                     | Key-values - all annotations                                                                                                             |
| %kv\{key\}                               | Key-value - annotation with given key                                                                                                    |
| %spans                                   | All log spans                                                                                                                            |
| %span\{span\}                            | Log spans with key                                                                                                                       |
| %highlight\{format\}                     | Highlight given format with colors based on log Level                                                                                    |
| %%                                       | % character                                                                                                                              |
| %\{                                      | \{ character                                                                                                                             |
| %\}                                      | \} character                                                                                                                             |

examples:

```
%timestamp %level [%fiberId] %name:%line %message %cause

%highlight{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %cause}

%label{timestamp}{%fixed{32}{%timestamp}} %label{level}{%level} %label{thread}{%fiberId} %label{message}{%message} %label{cause}{%cause}
```

## LogFormat and LogAppender

A `LogFormat` represents a DSL to describe the format of text log messages.

A `LogAppender` is a low-level interface designed to be the bridge between, ZIO Logging and logging backends, such as
Logback.
This interface is slightly higher-level than a string builder, because it allows for structured logging,
and preserves all ZIO-specific information about runtime failures.

`LogFormat` may be created by following function:

```scala
object LogFormat {
  def make(format: (LogAppender, Trace, FiberId, LogLevel, () => String, Cause[Any], FiberRefs, List[LogSpan], Map[String, String]) => Any): LogFormat
}
```

format function arguments can be split to two sections:

* LogAppender
* all others - all log inputs provided by ZIO core logging:
    * Trace - current trace (`zio.Trace`)
    * FiberId - fiber id (`zio.FiberId`)
    * LogLevel - log level (`zio.LogLevel`)
    * () => String - log message
    * Cause[Any] - cause (`zio.Cause`)
    * FiberRefs - fiber refs (`zio.FiberRefs`), collection of `zio.FiberRef` - ZIO's equivalent of Java's ThreadLocal
    * List[LogSpan] - log spans  (`zio.LogSpan`)
    * Map[String, String] - ZIO core log annotations values, where key is annotation key/name, and value is annotation
      value

essential `LogAppender` functions, which are used in predefined log formats:

* `def appendCause(cause: Cause[Any])` - appends a `zio.Cause` to the log, some logging backends may have special
  support for logging failures
* `def appendNumeric[A](numeric: A)` - appends a numeric value to the log
* `def appendText(text: String)` - appends unstructured text to the log
* `def appendKeyValue(key: String, value: String)` - appends a key/value string pair to the log

then it depends on the specific logging backend how these functions are implemented with respect to the backend output,
for example:

* [slf4j v1](slf4j1.md) logging backend - key/value is appended to
  slf4j [MDC context](https://logback.qos.ch/manual/mdc.html), Cause is transformed to Throwable and placed to slf4j
  throwable section, all other text and numeric parts are added to slf4j log message
* console logging backend - in general all values are added to log line, `Cause.prettyPrint` is used to log cause
  details

example of some predefined log formats implementations:

```scala
def annotation(name: String): LogFormat =
  LogFormat.make { (builder, _, _, _, _, _, _, _, annotations) =>
    annotations.get(name).foreach { value =>
      builder.appendKeyValue(name, value)
    }
  }

val cause: LogFormat =
  LogFormat.make { (builder, _, _, _, _, cause, _, _, _) =>
    if (!cause.isEmpty) {
      builder.appendCause(cause)
    }
  }

def text(value: => String): LogFormat =
  LogFormat.make { (builder, _, _, _, _, _, _, _, _) =>
    builder.appendText(value)
  }
```

---

## Introduction to ZIO Logging

[ZIO Logging](https://github.com/zio/zio-logging) is simple logging for ZIO apps, with correlation, context, and pluggable backends out of the box with integrations for common logging backends.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-logging/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-logging_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-logging_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-logging_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-logging_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-logging-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-logging-docs_2.13) [![zio-logging](https://img.shields.io/github/stars/zio/zio-logging?style=social)](https://github.com/zio/zio-logging)

## Introduction

When we are writing our applications using ZIO effects, to log easy way we need a ZIO native solution for logging.

Key features of ZIO Logging:

- **ZIO Native** — Other than it is a type-safe and purely functional solution, it leverages ZIO's features.
- **Multi-Platform** - It supports JS, JVM and Native platforms for the core zio-logging module, all other modules are only available for JVM.
- **Composable** — Loggers are composable together via contraMap.
- **Pluggable Backends** — Support multiple backends like ZIO Console, SLF4j, JPL.
- **Logger Context** — It has a first citizen _Logger Context_ implemented on top of `FiberRef`. The Logger Context maintains information like logger name, filters, correlation id, and so forth across different fibers. It supports _Mapped Diagnostic Context (MDC)_ which manages contextual information across fibers in a concurrent environment.
- Richly integrated into ZIO 2's built-in logging facilities
- ZIO Console, SLF4j, and other backends

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
// ZIO Logging backends
libraryDependencies += "dev.zio" %% "zio-logging" % "2.5.3"
```

The main module contains the following features: 
* [console loggers](console-logger.md)
* [file loggers](file-logger.md)
* [log metrics](metrics.md)

Other modules:

* SLF4J backend - if you like to use [`SLF4J`](https://www.slf4j.org/) logging backends (e.g. java.util.logging, logback, log4j), add the one of following lines to your `build.sbt` file:

    ```scala
    // SLF4j v1 integration
    libraryDependencies += "dev.zio" %% "zio-logging-slf4j" % "2.5.3"
    
    // SLF4j v2 integration
    libraryDependencies += "dev.zio" %% "zio-logging-slf4j2" % "2.5.3"
    ```
   When to use this module: you are already using SLF4J logger in some other project, and you like to have same log outputs. 
   See SLF4J [v2](slf4j2.md) or [v1](slf4j1.md) section for more details.

* SLF4J bridge - with this logging bridge, it is possible to use `zio-logging` for SLF4J loggers (usually third-party non-ZIO libraries), add the one of following lines to your `build.sbt` file: 

    ```scala
    // Using ZIO Logging for SLF4j v1 loggers, usually third-party non-ZIO libraries
    libraryDependencies += "dev.zio" %% "zio-logging-slf4j-bridge" % "2.5.3"
    
    // Using ZIO Logging for SLF4j v2 loggers, usually third-party non-ZIO libraries
    libraryDependencies += "dev.zio" %% "zio-logging-slf4j2-bridge" % "2.5.3"
    ```

    When to use this module: you want to use some zio-logger implementation, but also you are using some java library which using SLF4J interface for logging.
    See SLF4J bridge [v2](slf4j2-bridge.md) or [v1](slf4j1-bridge.md) section for more details.

* Java Platform/System Logger backend - if you like to use  [`Java Platform/System Logger`](https://openjdk.org/jeps/264) logging backend, add the following line to your `build.sbt` file:

    ```scala
    // JPL integration
    libraryDependencies += "dev.zio" %% "zio-logging-jpl" % "2.5.3"
    ```

    When to use this module: you are already using Java Platform/System Logger in some other project, and you like to have same log outputs.
    See [Java Platform/System Logger](jpl.md) section for more details.

* java.util.logging bridge - with this logging bridge, it is possible to use `zio-logging` for JUL loggers (usually third-party non-ZIO libraries), add the one of following lines to your `build.sbt` file:

    ```scala
    // JUL bridge
    libraryDependencies += "dev.zio" %% "zio-logging-jul-bridge" % "2.5.3"
    ```

    When to use this module: you are already using JUL logger in some other project, and you like to have same log outputs.
    See [java.util.logging bridge](jul-bridge.md) section for more details.

## Example

Let's try an example of ZIO Logging which demonstrates a simple application of ZIO logging.

The recommended place for setting the logger is application boostrap. In this case, custom logger will be set for whole application runtime (also application failures will be logged with specified logger).

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object SimpleApp extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleLogger()

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _ <- ZIO.logInfo("Start")
      _ <- ZIO.fail("FAILURE")
      _ <- ZIO.logInfo("Done")
    } yield ExitCode.success

}
```

Expected console output:

```
timestamp=2023-03-15T08:36:24.421098+01:00 level=INFO thread=zio-fiber-4 message="Start"
timestamp=2023-03-15T08:36:24.440181+01:00 level=ERROR thread=zio-fiber-0 message="" cause=Exception in thread "zio-fiber-4" java.lang.String: FAILURE
	at zio.logging.example.SimpleApp.run(SimpleApp.scala:29)
```

You can find the source code of examples [here](https://github.com/zio/zio-logging/tree/master/examples)

---

## Java Platform/System Logger

[`Java Platform/System Logger`](https://openjdk.org/jeps/264) is logging API which was introduced in Java 9.

In order to use this logging backend, we need to add the following line in our build.sbt file:

```scala
libraryDependencies += "dev.zio" %% "zio-logging-jpl" % "2.5.3"
```

Logger layer:

[//]: # (TODO: make snippet type-checked using mdoc)

```scala

val logger = Runtime.removeDefaultLoggers >>> JPL.jpl
```

Default `JPL` logger setup:
* logger name (by default)  is extracted from `zio.Trace`
    * for example, trace `zio.logging.example.JplSimpleApp.run(JplSimpleApp.scala:17)` will have `zio.logging.example.JplSimpleApp` as logger name
    * NOTE: custom logger name may be set by `zio.logging.loggerName` aspect
* all annotations (logger name annotation is excluded) are placed at the beginning of log message
* cause is logged as throwable

See also [LogFormat and LogAppender](formatting-log-records.md#logformat-and-logappender)

Custom logger name set by aspect:

```scala
ZIO.logInfo("Starting user operation") @@ zio.logging.loggerName("zio.logging.example.UserOperation")
```

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### Java Platform/System logger name and annotations

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object JplSimpleApp extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] = Runtime.removeDefaultLoggers >>> JPL.jpl

  private val users = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _       <- ZIO.logInfo("Start")
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.foreachPar(users) { uId =>
        {
          ZIO.logInfo("Starting user operation") *>
            ZIO.sleep(500.millis) *>
            ZIO.logInfo("Stopping user operation")
        } @@ ZIOAspect.annotated("user", uId.toString)
      } @@ LogAnnotation.TraceId(traceId) @@ zio.logging.loggerName("zio.logging.example.UserOperation")
      _       <- ZIO.logInfo("Done")
    } yield ExitCode.success

}
```

Expected Console Output:
```
Oct 28, 2022 1:47:01 PM zio.logging.backend.JPL$$anon$1 $anonfun$closeLogEntry$1
INFO: Start
Oct 28, 2022 1:47:01 PM zio.logging.backend.JPL$$anon$1 $anonfun$closeLogEntry$1
INFO: user=59c114fd-676d-4df9-a5a0-b8e132468fbf trace_id=7d3e3b84-dd3b-44ff-915a-04fb2d135e28 Starting user operation
Oct 28, 2022 1:47:01 PM zio.logging.backend.JPL$$anon$1 $anonfun$closeLogEntry$1
INFO: user=e1ebf0cc-2f61-484f-afcd-de7e20ec7829 trace_id=7d3e3b84-dd3b-44ff-915a-04fb2d135e28 Starting user operation
Oct 28, 2022 1:47:02 PM zio.logging.backend.JPL$$anon$1 $anonfun$closeLogEntry$1
INFO: user=e1ebf0cc-2f61-484f-afcd-de7e20ec7829 trace_id=7d3e3b84-dd3b-44ff-915a-04fb2d135e28 Stopping user operation
Oct 28, 2022 1:47:02 PM zio.logging.backend.JPL$$anon$1 $anonfun$closeLogEntry$1
INFO: user=59c114fd-676d-4df9-a5a0-b8e132468fbf trace_id=7d3e3b84-dd3b-44ff-915a-04fb2d135e28 Stopping user operation
Oct 28, 2022 1:47:02 PM zio.logging.backend.JPL$$anon$1 $anonfun$closeLogEntry$1
INFO: Done
```

## Feature changes

### Version 2.2.0

Deprecated log annotation with key `jpl_logger_name` (`JPL.loggerNameAnnotationKey`) removed,
only common log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`) for logger name is supported now.

---

## java.util.logging bridge

It is possible to use `zio-logging` for included `java.util.logging` Loggers (do not confuse with `java.platform.logging`),
usually third-party non-ZIO libraries (most notable: OpenTelemetry used by ZIO-telemetry). To do so, import the `zio-logging-jul-bridge` module
```scala
libraryDependencies += "dev.zio" %% "zio-logging-jul-bridge" % "2.5.3"
```

and use one of the `JULBridge` layers when setting up logging

```scala

program.provideCustom(JULBridge.init())
```

`JULBridge` layers:
* `JULBridge.init(configPath: NonEmptyChunk[String] = logFilterConfigPath)` - setup with `LogFilter` from [filter configuration](log-filter.md#configuration), default configuration path: `logger.filter`, default `LogLevel` is `INFO`
* `JULBridge.init(filter: LogFilter[Any])` - setup with given `LogFilter`
* `JULBridge.initialize` - setup without filtering

Need for log filtering in JUL bridge: filtering in JUL is made on higher level than `jul-bridge` (on `Logger` level and not `Handler` level - which `JULBridge` is). Due to that the whole
filtering in JUL is disabled and is implemented in JULBridge. This may cause degraded performance and much more logs when using other Handlers.

JUL logger name is stored in log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`), following log format

```scala

val loggerName = LoggerNameExtractor.loggerNameAnnotationOrTrace
val loggerNameFormat = loggerName.toLogFormat()
```
may be used to get logger name from log annotation or ZIO Trace.

This logger name extractor is used by default in log filter, which applying log filtering by defined logger name and level:

```scala
val logFilterConfig = LogFilter.LogLevelByNameConfig(
  LogLevel.Info,
  "zio.logging.jul  " -> LogLevel.Debug,
  "JUL-LOGGER"        -> LogLevel.Warning
)

val logFilter: LogFilter[String] = logFilterConfig.toFilter
```

JUL bridge with custom logger can be setup:

```scala

val logger = Runtime.removeDefaultLoggers >>> consoleJsonLogger() >+> JULBridge.init()
```

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### JUL bridge with JSON console logger

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object JULBridgeExampleApp extends ZIOAppDefault {

  private val julLogger = java.util.logging.Logger.getLogger("JUL-LOGGER")

  private val logFilterConfig = LogFilter.LogLevelByNameConfig(
    LogLevel.Info,
    "zio.logging.slf4j" -> LogLevel.Debug,
    "SLF4J-LOGGER"      -> LogLevel.Warning
  )

  private val logFormat = LogFormat.label(
    "name",
    LoggerNameExtractor.loggerNameAnnotationOrTrace.toLogFormat()
  ) + LogFormat.logAnnotation(LogAnnotation.UserId) + LogFormat.logAnnotation(
    LogAnnotation.TraceId
  ) + LogFormat.default

  private val loggerConfig = ConsoleLoggerConfig(logFormat, logFilterConfig)

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleJsonLogger(loggerConfig) >+> JULBridge.init(loggerConfig.toFilter)

  private val uuids = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _ <- ZIO.logInfo("Start")
      _ <- ZIO.foreachPar(uuids) { u =>
        ZIO.succeed(julLogger.info("Test INFO!")) *> ZIO.succeed(
          julLogger.warning("Test WARNING!")
        ) @@ LogAnnotation.UserId(
          u.toString
        )
      } @@ LogAnnotation.TraceId(UUID.randomUUID())
      _ <- ZIO.logDebug("Done")
    } yield ExitCode.success

}
```

Expected console output:
```
{"name":"zio.logging.example.JULbridgeExampleApp","timestamp":"2024-05-26T13:50:20.6832831+02:0","level":"INFO","thread":"zio-fiber-1143120685","message":"Start"}
{"name":"JUL-LOGGER","trace_id":"08e9e10a-d3c5-4f90-8627-2ae4ddee1522","timestamp":"2024-05-26T13:50:20.7112909+02:0","level":"INFO","thread":"zio-fiber-1683803358","message":"Test INFO!"}
{"name":"JUL-LOGGER","trace_id":"08e9e10a-d3c5-4f90-8627-2ae4ddee1522","timestamp":"2024-05-26T13:50:20.7112909+02:0","level":"INFO","thread":"zio-fiber-71852457","message":"Test INFO!"}
{"name":"JUL-LOGGER","user_id":"85f762cc-e62c-4576-9f14-6a3ad0918d99","trace_id":"08e9e10a-d3c5-4f90-8627-2ae4ddee1522","timestamp":"2024-05-26T13:50:20.7142882+02:0","level":"WARN","thread":"zio-fiber-1911711828","message":"Test WARNING!"}
{"name":"JUL-LOGGER","user_id":"47850c02-bb60-4b6a-9c0f-0aa095066d10","trace_id":"08e9e10a-d3c5-4f90-8627-2ae4ddee1522","timestamp":"2024-05-26T13:50:20.7142882+02:0","level":"WARN","thread":"zio-fiber-1801412106","message":"Test WARNING!"}
```

---

## Log Filter

A `LogFilter` represents function/conditions for log filtering.

Following filter

[//]: # (TODO: make snippet type-checked using mdoc)

```scala

val filter = LogFilter.logLevelByName(
    LogLevel.Debug,
    "io.netty" -> LogLevel.Info, 
    "io.grpc.netty" -> LogLevel.Info,
    "org.my.**.ServiceX" -> LogLevel.Trace, // glob-like (any paths) filter
    "org.my.X*Layers" -> LogLevel.Info // glob-like (single or partial path) filter
)
```

will use the `Debug` log level for everything except for log events with the logger name
prefixed by either `List("io", "netty")` or `List("io", "grpc", "netty")` or `List("org", "my", "**", "ServiceX")` or `List("org", "my", "X*Layers")`.
Logger name is extracted from log annotation or `zio.Trace`.

`LogFilter.filter` returns a version of `zio.ZLogger` that only logs messages when this filter is satisfied.

## Configuration

the configuration for filter (`zio.logging.LogFilter.LogLevelByNameConfig`) has the following configuration structure:

```
{
    # LogLevel values: ALL, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, OFF
    
    # root LogLevel, default value: INFO
    rootLevel = DEBUG 
    
    # LogLevel configurations for specific logger names, or prefixes, default value: empty
    mappings {
      "io.netty" = "INFO"
      "io.grpc.netty" = "INFO"
    }
}
```

this configuration is equivalent to following:

```scala

val config =
  LogFilter.LogLevelByNameConfig(LogLevel.Debug, Map("io.netty" -> LogLevel.Info, "io.grpc.netty" -> LogLevel.Info))

val filter = LogFilter.logLevelByName(config)    
```

---

## Logger Context and Annotations

The `logContext` fiber reference is used to store typed, structured log
annotations, which can be utilized by backends to enrich log messages.

Because `logContext` is an ordinary `zio.FiberRef`, it may be get, set,
and updated like any other fiber reference. However, the idiomatic way to
interact with `logContext` is by using `zio.logging.LogAnnotation`.

For example:

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
myResponseHandler(request) @@ LogAnnotation.UserId(request.userId)
```

This code would add the structured log annotation `LogAnnotation.UserId`
to all log messages emitted by the `myResponseHandler(request)` effect.

The user of the library is allowed to add a custom `LogAnnotation`:

```scala

val customLogAnnotation = LogAnnotation[Int]("custom_annotation", _ + _, _.toString)
```

---

## Log Metrics

Log metrics collecting metrics related to ZIO logging (all `ZIO.log*` functions).
As ZIO core supporting multiple loggers, this logging metrics collector is implemented as specific `ZLogger`
which is responsible just for collecting metrics of all logs - `ZIO.log*` functions.

The Metrics layer

```scala
val layer = zio.logging.logMetrics
```

will add a default metric named `zio_log_total` with the label `level` which will be
incremented for each log message with the value of `level` being the corresponding log level label in lower case.

Metrics:

* `LogLevel.All` -> `all`
* `LogLevel.Fatal` -> `fatal`
* `LogLevel.Error` -> `error`
* `LogLevel.Warning` -> `warn`
* `LogLevel.Info` -> `info`
* `LogLevel.Debug` -> `debug`
* `LogLevel.Trace` -> `trace`
* `LogLevel.None` -> `off`

Custom names for the metric and label can be set via:

```scala
val layer = zio.logging.logMetricsWith("log_counter", "log_level")
```

## Examples

You can find the source
code [here](https://github.com/zio/zio-logging/tree/master/examples)

### Console logger with metrics

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object MetricsApp extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> (consoleLogger() ++ logMetrics)

  override def run: ZIO[Scope, Any, ExitCode] =
    (for {
      _            <- ZIO.logInfo("Start")
      _            <- ZIO.logWarning("Some warning")
      _            <- ZIO.logError("Some error")
      _            <- ZIO.logError("Another error")
      _            <- ZIO.sleep(1.second)
      metricValues <- ZIO.serviceWithZIO[PrometheusPublisher](_.get)
      _            <- Console.printLine(metricValues)
      _            <- ZIO.logInfo("Done")
    } yield ExitCode.success)
      .provideLayer((ZLayer.succeed(MetricsConfig(200.millis)) ++ publisherLayer) >+> prometheusLayer)

}
```

Expected Console Output:

```
timestamp=2023-03-15T08:44:39.93193+01:00  level=INFO thread=zio-fiber-6 message="Start"
timestamp=2023-03-15T08:44:39.951764+01:00 level=WARN thread=zio-fiber-6 message="Some warning"
timestamp=2023-03-15T08:44:39.95388+01:00  level=ERROR thread=zio-fiber-6 message="Some error"
timestamp=2023-03-15T08:44:39.954738+01:00 level=ERROR thread=zio-fiber-6 message="Another error"
# TYPE zio_log_total counter
# HELP zio_log_total
zio_log_total{level="error",} 2.0 1678866280778
# TYPE zio_log_total counter
# HELP zio_log_total
zio_log_total{level="warn",} 1.0 1678866280778
# TYPE zio_log_total counter
# HELP zio_log_total
zio_log_total{level="info",} 1.0 1678866280778
timestamp=2023-03-15T08:44:40.972877+01:00 level=INFO thread=zio-fiber-6 message="Done"
```

---

## Reconfigurable Logger

`ReconfigurableLogger` is adding support for updating logger configuration in application runtime. 

logger layer with configuration from `ConfigProvider` (example with [Console Logger)](console-logger.md)):

```scala

val configProvider: ConfigProvider = ???

val logger = Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> ReconfigurableLogger
        .make[Any, Nothing, String, Any, ConsoleLoggerConfig](
          ConsoleLoggerConfig.load().orDie, // loading current configuration
          (config, _) => makeConsoleLogger(config), // make logger from loaded configuration
          Schedule.fixed(5.second) // default is 10 seconds 
        )
        .installUnscoped[Any]
```

`ReconfigurableLogger`, based on given `Schedule` and load configuration function, will recreate logger if configuration changed.

**NOTE:** consider if you need this feature in your application, as there may be some performance impacts (see [benchmarks](https://github.com/zio/zio-logging/blob/master/benchmarks/src/main/scala/zio/logging/FilterBenchmarks.scala)).

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### Console Logger With Re-configuration From Configuration File In Runtime

[//]: # (TODO: make snippet type-checked using mdoc)

Example of application where logger configuration is updated at runtime when logger configuration file is changed.

Configuration:

```
logger {
  format = "%highlight{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %kv{trace_id} %kv{user_id} %cause}"
  filter {
    rootLevel = "INFO"
    mappings {
      "zio.logging.example" = "DEBUG"
    }
  }
}
```

Application:

```scala
package zio.logging.example

object LoggerReconfigureApp extends ZIOAppDefault {

  def configuredLogger(
            loadConfig: => ZIO[Any, Config.Error, ConsoleLoggerConfig]
    ): ZLayer[Any, Config.Error, Unit] =
    ReconfigurableLogger
            .make[Any, Config.Error, String, Any, ConsoleLoggerConfig](
              loadConfig,
              (config, _) => makeConsoleLogger(config),
              Schedule.fixed(500.millis)
            )
            .installUnscoped

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> configuredLogger(
      for {
        config       <- ZIO.succeed(ConfigFactory.load("logger.conf"))
        _            <- Console.printLine(config.getConfig("logger")).orDie
        loggerConfig <- ConsoleLoggerConfig.load().withConfigProvider(TypesafeConfigProvider.fromTypesafeConfig(config))
      } yield loggerConfig
    )

  def exec(): ZIO[Any, Nothing, Unit] =
    for {
      ok      <- Random.nextBoolean
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.logDebug("Start") @@ LogAnnotation.TraceId(traceId)
      userIds <- ZIO.succeed(List.fill(2)(UUID.randomUUID().toString))
      _       <- ZIO.foreachPar(userIds) { userId =>
        {
          ZIO.logDebug("Starting operation") *>
                  ZIO.logInfo("OK operation").when(ok) *>
                  ZIO.logError("Error operation").when(!ok) *>
                  ZIO.logDebug("Stopping operation")
        } @@ LogAnnotation.UserId(userId)
      } @@ LogAnnotation.TraceId(traceId)
      _       <- ZIO.logDebug("Done") @@ LogAnnotation.TraceId(traceId)
    } yield ()

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _ <- exec().repeat(Schedule.fixed(500.millis))
    } yield ExitCode.success

}
```

When configuration for `logger/filter/mappings/zio.logging.example` change from `DEBUG` to `WARN`:

```
Config(SimpleConfigObject({"filter":{"mappings":{"zio.logging.example":"DEBUG"},"rootLevel":"INFO"},"format":"%highlight{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %kv{trace_id} %kv{user_id} %cause}"}))
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-5] zio.logging.example.LoggerReconfigureApp:51 Start trace_id=87ead38c-8b42-43ea-9905-039d0263026d  
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-36] zio.logging.example.LoggerReconfigureApp:55 Starting operation trace_id=87ead38c-8b42-43ea-9905-039d0263026d user_id=dfa05247-ec27-46f7-a4e0-bb86f2d501e9 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-37] zio.logging.example.LoggerReconfigureApp:55 Starting operation trace_id=87ead38c-8b42-43ea-9905-039d0263026d user_id=19693c77-0896-4fae-a830-67d5fe370b05 
2023-12-26T10:10:26+0100 ERROR   [zio-fiber-36] zio.logging.example.LoggerReconfigureApp:57 Error operation trace_id=87ead38c-8b42-43ea-9905-039d0263026d user_id=dfa05247-ec27-46f7-a4e0-bb86f2d501e9 
2023-12-26T10:10:26+0100 ERROR   [zio-fiber-37] zio.logging.example.LoggerReconfigureApp:57 Error operation trace_id=87ead38c-8b42-43ea-9905-039d0263026d user_id=19693c77-0896-4fae-a830-67d5fe370b05 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-36] zio.logging.example.LoggerReconfigureApp:58 Stopping operation trace_id=87ead38c-8b42-43ea-9905-039d0263026d user_id=dfa05247-ec27-46f7-a4e0-bb86f2d501e9 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-37] zio.logging.example.LoggerReconfigureApp:58 Stopping operation trace_id=87ead38c-8b42-43ea-9905-039d0263026d user_id=19693c77-0896-4fae-a830-67d5fe370b05 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-5] zio.logging.example.LoggerReconfigureApp:61 Done trace_id=87ead38c-8b42-43ea-9905-039d0263026d  
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-5] zio.logging.example.LoggerReconfigureApp:51 Start trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c  
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-39] zio.logging.example.LoggerReconfigureApp:55 Starting operation trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c user_id=a62a153f-6a91-491e-8c97-bab94186f0a2 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-38] zio.logging.example.LoggerReconfigureApp:55 Starting operation trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c user_id=8eeb6442-80a9-40e5-b97d-a12876702a65 
2023-12-26T10:10:26+0100 ERROR   [zio-fiber-39] zio.logging.example.LoggerReconfigureApp:57 Error operation trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c user_id=a62a153f-6a91-491e-8c97-bab94186f0a2 
2023-12-26T10:10:26+0100 ERROR   [zio-fiber-38] zio.logging.example.LoggerReconfigureApp:57 Error operation trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c user_id=8eeb6442-80a9-40e5-b97d-a12876702a65 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-39] zio.logging.example.LoggerReconfigureApp:58 Stopping operation trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c user_id=a62a153f-6a91-491e-8c97-bab94186f0a2 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-38] zio.logging.example.LoggerReconfigureApp:58 Stopping operation trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c user_id=8eeb6442-80a9-40e5-b97d-a12876702a65 
2023-12-26T10:10:26+0100 DEBUG   [zio-fiber-5] zio.logging.example.LoggerReconfigureApp:61 Done trace_id=c6d4c770-8db1-4ea8-91eb-548c6a99a90c  
Config(SimpleConfigObject({"filter":{"mappings":{"zio.logging.example":"WARN"},"rootLevel":"INFO"},"format":"%highlight{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %kv{trace_id} %kv{user_id} %cause}"}))
2023-12-26T10:10:27+0100 ERROR   [zio-fiber-40] zio.logging.example.LoggerReconfigureApp:57 Error operation trace_id=e2d8bbb4-5ad0-4952-b035-03da7689ab56 user_id=0f6452da-3f7e-40ff-b8b4-6b4c731903fb 
2023-12-26T10:10:27+0100 ERROR   [zio-fiber-41] zio.logging.example.LoggerReconfigureApp:57 Error operation trace_id=e2d8bbb4-5ad0-4952-b035-03da7689ab56 user_id=c4a86b38-90d7-4bb6-9f49-73bc5701e1ef  
```

### Console Logger With Configuration By Http APIs

[//]: # (TODO: make snippet type-checked using mdoc)

Example of application where logger configuration can be changed by Http APIs.

Logger configurations APIs:
* get logger configurations 
 ```bash 
  curl -u "admin:admin" 'http://localhost:8080/example/logger'
 ```
* get `root` logger configuration
 ```bash 
  curl -u "admin:admin" 'http://localhost:8080/example/logger/root'
 ```
* set `root` logger configuration
 ```bash 
  curl -u "admin:admin" --location --request PUT 'http://localhost:8080/example/logger/root' --header 'Content-Type: application/json' --data-raw '"WARN"'
 ```
* get `zio.logging.example` logger configuration
 ```bash 
  curl -u "admin:admin" --location --request PUT 'http://localhost:8080/example/logger/zio.logging.example' --header 'Content-Type: application/json' --data-raw '"WARN"'
 ```

Configuration:

```
logger {
  format = "%highlight{%timestamp{yyyy-MM-dd'T'HH:mm:ssZ} %fixed{7}{%level} [%fiberId] %name:%line %message %kv{trace_id} %kv{user_id} %cause}"
  filter {
    rootLevel = "INFO"
    mappings {
      "zio.logging.example" = "DEBUG"
    }
  }
}
```

Application:

```scala
package zio.logging.example

object ConfigurableLoggerApp extends ZIOAppDefault {

  def configurableLogger(): ZLayer[Any, Config.Error, Unit] =
    ConsoleLoggerConfig
      .load()
      .flatMap { config =>
        makeSystemOutLogger(config.format.toLogger).map { logger =>
          ConfigurableLogger.make(logger, config.filter)
        }
      }
      .install

  val configProvider: ConfigProvider = TypesafeConfigProvider.fromTypesafeConfig(ConfigFactory.load("logger.conf"))

  override val bootstrap: ZLayer[Any, Config.Error, Unit] =
    Runtime.removeDefaultLoggers >>> Runtime.setConfigProvider(configProvider) >>> configurableLogger()

  def exec(): ZIO[Any, Nothing, Unit] =
    for {
      ok      <- Random.nextBoolean
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.logDebug("Start") @@ LogAnnotation.TraceId(traceId)
      userIds <- ZIO.succeed(List.fill(2)(UUID.randomUUID().toString))
      _       <- ZIO.foreachPar(userIds) { userId =>
        {
          ZIO.logDebug("Starting operation") *>
            ZIO.logInfo("OK operation").when(ok) *>
            ZIO.logError("Error operation").when(!ok) *>
            ZIO.logDebug("Stopping operation")
        } @@ LogAnnotation.UserId(userId)
      } @@ LogAnnotation.TraceId(traceId)
      _       <- ZIO.logDebug("Done") @@ LogAnnotation.TraceId(traceId)
    } yield ()

  val httpApp: Routes[LoggerConfigurer with Any, Nothing] =
    ApiHandlers.routes("example" :: Nil) @@ Middleware.basicAuth("admin", "admin")

  override def run: ZIO[Scope, Any, ExitCode] =
    (for {
      _ <- Server.serve(httpApp).fork
      _ <- exec().repeat(Schedule.fixed(500.millis))
    } yield ExitCode.success).provide(LoggerConfigurer.layer ++ Server.default)

}
```

**NOTE:** `ConfigurableLogger` and `ApiHandlers` are currently implemented in examples,
it will be considered in the future, if they will be moved to official `zio-logging` implementation
(once there will be official stable `zio-http` release).
If you like to use them in your app, you can copy them.

When configuration for `logger/filter/mappings/zio.logging.example` change from `DEBUG` to `WARN`:

```bash
curl -u "admin:admin" --location --request PUT 'http://localhost:8080/example/logger/zio.logging.example' --header 'Content-Type: application/json' --data-raw '"WARN"'
```

```
2023-12-26T10:49:27+0100 DEBUG   [zio-fiber-73] zio.logging.example.ConfigurableLoggerApp:62 Starting operation trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045 user_id=5d35778f-78ff-48a8-aa6a-73114ec719b5 
2023-12-26T10:49:27+0100 DEBUG   [zio-fiber-72] zio.logging.example.ConfigurableLoggerApp:62 Starting operation trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045 user_id=9e17bd97-aa18-4b09-a423-9de28241a20b 
2023-12-26T10:49:27+0100 INFO    [zio-fiber-73] zio.logging.example.ConfigurableLoggerApp:63 OK operation trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045 user_id=5d35778f-78ff-48a8-aa6a-73114ec719b5 
2023-12-26T10:49:27+0100 INFO    [zio-fiber-72] zio.logging.example.ConfigurableLoggerApp:63 OK operation trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045 user_id=9e17bd97-aa18-4b09-a423-9de28241a20b 
2023-12-26T10:49:27+0100 DEBUG   [zio-fiber-73] zio.logging.example.ConfigurableLoggerApp:65 Stopping operation trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045 user_id=5d35778f-78ff-48a8-aa6a-73114ec719b5 
2023-12-26T10:49:27+0100 DEBUG   [zio-fiber-72] zio.logging.example.ConfigurableLoggerApp:65 Stopping operation trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045 user_id=9e17bd97-aa18-4b09-a423-9de28241a20b 
2023-12-26T10:49:27+0100 DEBUG   [zio-fiber-4] zio.logging.example.ConfigurableLoggerApp:68 Done trace_id=dcf30228-dc00-4c1f-ab94-20c9f8116045  
2023-12-26T10:49:28+0100 ERROR   [zio-fiber-77] zio.logging.example.ConfigurableLoggerApp:64 Error operation trace_id=7da8765e-2e27-42c6-8834-16d15d21c72c user_id=4395d188-5971-4839-a721-278d07a2881b 
2023-12-26T10:49:28+0100 ERROR   [zio-fiber-78] zio.logging.example.ConfigurableLoggerApp:64 Error operation trace_id=7da8765e-2e27-42c6-8834-16d15d21c72c user_id=27af6873-2f7b-4b9a-ad2b-6bd12479cace  
```

---

## SLF4J v1 bridge

It is possible to use `zio-logging` for SLF4J loggers, usually third-party non-ZIO libraries. To do so, import the `zio-logging-slf4j-bridge` module for SLF4J v1 (working with JDK8):

```scala
libraryDependencies += "dev.zio" %% "zio-logging-slf4j-bridge" % "2.5.3"
```

and use one of the `Slf4jBridge` layers when setting up logging:

```scala

program.provideCustom(Slf4jBridge.init())
```

`Slf4jBridge` layers:
* `Slf4jBridge.init(configPath: NonEmptyChunk[String] = logFilterConfigPath)` - setup with `LogFilter` from [filter configuration](log-filter.md#configuration), default configuration path: `logger.filter`, default `LogLevel` is `INFO`
* `Slf4jBridge.init(filter: LogFilter[Any])` - setup with given `LogFilter`
* `Slf4jBridge.initialize` - setup without filtering

Need for log filtering in slf4j bridge: libraries with slf4j loggers, may have conditional logic for logging,
which using functions like [org.slf4j.Logger.isTraceEnabled()](https://github.com/qos-ch/slf4j/blob/master/slf4j-api/src/main/java/org/slf4j/Logger.java#L170). 
logging parts may contain message and log parameters construction, which may be expensive and degrade performance of application.

SLF4J logger name is stored in log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`), following log format

```scala

val loggerName = LoggerNameExtractor.loggerNameAnnotationOrTrace
val loggerNameFormat = loggerName.toLogFormat()
```
may be used to get logger name from log annotation or ZIO Trace.

SLF4J bridge with custom logger can be setup:

```scala

val logger = Runtime.removeDefaultLoggers >>> consoleJsonLogger() >+> Slf4jBridge.init()
```

**NOTE:** You should either use `zio-logging-slf4j` to send all ZIO logs to an SLF4j provider (such as logback, log4j etc) OR `zio-logging-slf4j-bridge` to send all SLF4j logs to
ZIO logging. Enabling both causes circular logging and makes no sense.

## Examples

### SLF4J bridge with JSON console logger

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object Slf4jBridgeExampleApp extends ZIOAppDefault {

  private val slf4jLogger = org.slf4j.LoggerFactory.getLogger("SLF4J-LOGGER")

  private val logFilterConfig = LogFilter.LogLevelByNameConfig(
    LogLevel.Info,
    "zio.logging.slf4j" -> LogLevel.Debug,
    "SLF4J-LOGGER"      -> LogLevel.Warning
  )

  private val logFormat = LogFormat.label(
    "name",
    LoggerNameExtractor.loggerNameAnnotationOrTrace.toLogFormat()
  ) + LogFormat.logAnnotation(LogAnnotation.UserId) + LogFormat.logAnnotation(
    LogAnnotation.TraceId
  ) + LogFormat.default

  private val loggerConfig = ConsoleLoggerConfig(logFormat, logFilterConfig)

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleJsonLogger(loggerConfig) >+> Slf4jBridge.init(loggerConfig.toFilter)

  private val uuids = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _ <- ZIO.logInfo("Start")
      _ <- ZIO.foreachPar(uuids) { u =>
        ZIO.succeed(slf4jLogger.info("Test {}!", "INFO")) *> ZIO.succeed(
          slf4jLogger.warn("Test {}!", "WARNING")
        ) @@ LogAnnotation.UserId(
          u.toString
        )
      } @@ LogAnnotation.TraceId(UUID.randomUUID())
      _ <- ZIO.logDebug("Done")
    } yield ExitCode.success

}
```

Expected Console Output:
```
{"name":"zio.logging.example.Slf4jBridgeExampleApp","timestamp":"2024-02-16T08:10:45.373807+01:00","level":"INFO","thread":"zio-fiber-6","message":"Start"}
{"name":"SLF4J-LOGGER","user_id":"d13f90ad-6b0a-45fd-bf94-1db7a0d8c0b7","trace_id":"561300a9-e6f1-4f61-8dcc-dfef476dab20","timestamp":"2024-02-16T08:10:45.421448+01:00","level":"WARN","thread":"zio-fiber-10","message":"Test WARNING!"}
{"name":"SLF4J-LOGGER","user_id":"0f28521f-ac8f-4d8e-beeb-13c85c90c041","trace_id":"561300a9-e6f1-4f61-8dcc-dfef476dab20","timestamp":"2024-02-16T08:10:45.421461+01:00","level":"WARN","thread":"zio-fiber-9","message":"Test WARNING!"}
{"name":"zio.logging.example.Slf4jBridgeExampleApp","timestamp":"2024-02-16T08:10:45.428162+01:00","level":"DEBUG","thread":"zio-fiber-6","message":"Done"}
```

## Feature changes

### Version 2.1.9

SLF4J logger name is stored in log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`), 
in previous versions, logger name was stored in log annotation with key `slf4j_logger_name` (`Slf4jBridge.loggerNameAnnotationKey`),
for backward compatibility, if there is need to use legacy annotation key, it can be done with following initialisation

```scala

program.provideCustom(Slf4jBridge.initialize(Slf4jBridge.loggerNameAnnotationKey))
```

NOTE: this feature may be removed in future versions

Following log format

```scala

val loggerName = LoggerNameExtractor.annotationOrTrace(Slf4jBridge.loggerNameAnnotationKey)
val loggerNameFormat = loggerName.toLogFormat()
```
may be used to get logger name from log annotation or ZIO Trace.

This logger name extractor can be used also in log filter, which applying log filtering by defined logger name and level:

```scala
val logFilter: LogFilter[String] = LogFilter.logLevelByGroup(
  LogLevel.Info,
  loggerName.toLogGroup(),
  "zio.logging.slf4j" -> LogLevel.Debug,
  "SLF4J-LOGGER"      -> LogLevel.Warning
)
```

### Version 2.2.0

Deprecated log annotation with key `slf4j_logger_name` (`Slf4jBridge.loggerNameAnnotationKey`) removed, 
only common log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`) for logger name is supported now.

---

## SLF4J v1

The Simple Logging Facade for Java ([`SLF4J v1`](https://www.slf4j.org/) - working with JDK8) serves as a simple facade or abstraction for various logging frameworks (e.g. java.util.logging, logback, log4j).

In order to use this logging backend, we need to add the following line in our build.sbt file:

```scala
libraryDependencies += "dev.zio" %% "zio-logging-slf4j" % "2.5.3"
```

>**_NOTE:_** SLF4J v1 implementation is similar to [v2](slf4j2.md), 
however there are some differences, v1 using [MDC context](https://logback.qos.ch/manual/mdc.html), working with JDK8, 
v2 using [key-value pairs](https://www.slf4j.org/manual.html#fluent), working with JDK9+. 
It is recommended to use v2, as it is the latest version 
and also there may be MDC (using [ThreadLocal](https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html)) issues,
where in the case of parallel executions it can happen that MDC will be reset by different Fiber before the message is logged.

Logger layer:

```scala

val logger = Runtime.removeDefaultLoggers >>> SLF4J.slf4j
```

Default `SLF4J` logger setup:
* logger name (by default)  is extracted from `zio.Trace`
    * for example, trace `zio.logging.example.Slf4jSimpleApp.run(Slf4jSimpleApp.scala:17)` will have `zio.logging.example.Slf4jSimpleApp` as logger name
    * NOTE: custom logger name may be set by `zio.logging.loggerName` aspect
* all annotations (logger name and log marker name annotations are excluded) are placed into [MDC context](https://logback.qos.ch/manual/mdc.html)
* cause is logged as throwable

See also [LogFormat and LogAppender](formatting-log-records.md#logformat-and-logappender)

Custom logger name set by aspect:

```scala
ZIO.logInfo("Starting user operation") @@ zio.logging.loggerName("zio.logging.example.UserOperation")
```

Log marker name set by aspect:

```scala
ZIO.logInfo("Confidential user operation") @@ SLF4J.logMarkerName("CONFIDENTIAL")
```

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### SLF4J logger name and annotations

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object Slf4jSimpleApp extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] = Runtime.removeDefaultLoggers >>> SLF4J.slf4j

  private val users = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _       <- ZIO.logInfo("Start")
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.foreachPar(users) { uId =>
        {
          ZIO.logInfo("Starting user operation") *>
            ZIO.logInfo("Confidential user operation") @@ SLF4J.logMarkerName("CONFIDENTIAL") *>
            ZIO.sleep(500.millis) *>
            ZIO.logInfo("Stopping user operation")
        } @@ ZIOAspect.annotated("user", uId.toString)
      } @@ LogAnnotation.TraceId(traceId) @@ zio.logging.loggerName("zio.logging.example.UserOperation")
      _       <- ZIO.logInfo("Done")
    } yield ExitCode.success

}
```

Logback configuration:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>%d{HH:mm:ss.SSS} [%thread] [%mdc] %-5level %logger{36} %msg%n</Pattern>
        </layout>
    </appender>
    <turboFilter class="ch.qos.logback.classic.turbo.MarkerFilter">
        <Name>CONFIDENTIAL_FILTER</Name>
        <Marker>CONFIDENTIAL</Marker>
        <OnMatch>DENY</OnMatch>
    </turboFilter>
    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>
```

Expected Console Output:
```
12:16:21.951 [ZScheduler-Worker-8] [] INFO  zio.logging.example.Slf4jSimpleApp Start
12:16:22.024 [ZScheduler-Worker-12] [user=0e3bd69c-ee62-4096-82b2-593760d3fb19, trace_id=6e956bcf-d534-4c16-9402-fb6bca13c9ab] INFO  zio.logging.example.UserOperation Starting user operation
12:16:22.024 [ZScheduler-Worker-10] [user=869ed4c7-924d-4c02-ab5c-c30c1996a139, trace_id=6e956bcf-d534-4c16-9402-fb6bca13c9ab] INFO  zio.logging.example.UserOperation Starting user operation
12:16:22.592 [ZScheduler-Worker-14] [user=869ed4c7-924d-4c02-ab5c-c30c1996a139, trace_id=6e956bcf-d534-4c16-9402-fb6bca13c9ab] INFO  zio.logging.example.UserOperation Stopping user operation
12:16:22.592 [ZScheduler-Worker-1] [user=0e3bd69c-ee62-4096-82b2-593760d3fb19, trace_id=6e956bcf-d534-4c16-9402-fb6bca13c9ab] INFO  zio.logging.example.UserOperation Stopping user operation
12:16:22.598 [ZScheduler-Worker-14] [] INFO  zio.logging.example.Slf4jSimpleApp Done
```

### Custom tracing annotation

Following application has custom aspect `currentTracingSpanAspect` implementation which taking current span from `Tracing` service 
which then is added to logs by log annotation.

```scala
package zio.logging.example

trait Tracing {
  def getCurrentSpan(): UIO[String]
}

final class LiveTracing extends Tracing {
  override def getCurrentSpan(): UIO[String] = ZIO.succeed(UUID.randomUUID().toString)
}

object LiveTracing {
  val layer: ULayer[Tracing] = ZLayer.succeed(new LiveTracing)
}

object CustomTracingAnnotationApp extends ZIOAppDefault {

  private def currentTracingSpanAspect(key: String): ZIOAspect[Nothing, Tracing, Nothing, Any, Nothing, Any] =
    new ZIOAspect[Nothing, Tracing, Nothing, Any, Nothing, Any] {
      def apply[R <: Tracing, E, A](zio: ZIO[R, E, A])(implicit trace: Trace): ZIO[R, E, A] =
        ZIO.serviceWithZIO[Tracing] { tracing =>
          tracing.getCurrentSpan().flatMap { span =>
            ZIO.logAnnotate(key, span)(zio)
          }
        }
    }

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] = Runtime.removeDefaultLoggers >>> SLF4J.slf4j

  private val users = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    (for {
      _ <- ZIO.foreachPar(users) { uId =>
        {
          ZIO.logInfo("Starting operation") *>
            ZIO.sleep(500.millis) *>
            ZIO.logInfo("Stopping operation")
        } @@ ZIOAspect.annotated("user", uId.toString)
      } @@ currentTracingSpanAspect("trace_id")
      _ <- ZIO.logInfo("Done")
    } yield ExitCode.success).provide(LiveTracing.layer)

}
```

Expected Console Output:
```
15:53:20.145 [ZScheduler-Worker-9] [user=1abd8458-aefd-4780-88ec-cccd1310d4c8, trace_id=71436dd4-22d5-4e06-aaa7-f3ff7b108037] INFO  z.l.e.CustomTracingAnnotationApp Starting operation
15:53:20.145 [ZScheduler-Worker-13] [user=878689e0-da30-49f8-8923-ed915c00db9c, trace_id=71436dd4-22d5-4e06-aaa7-f3ff7b108037] INFO  z.l.e.CustomTracingAnnotationApp Starting operation
15:53:20.688 [ZScheduler-Worker-15] [user=1abd8458-aefd-4780-88ec-cccd1310d4c8, trace_id=71436dd4-22d5-4e06-aaa7-f3ff7b108037] INFO  z.l.e.CustomTracingAnnotationApp Stopping operation
15:53:20.688 [ZScheduler-Worker-11] [user=878689e0-da30-49f8-8923-ed915c00db9c, trace_id=71436dd4-22d5-4e06-aaa7-f3ff7b108037] INFO  z.l.e.CustomTracingAnnotationApp Stopping operation
15:53:20.691 [ZScheduler-Worker-15] [] INFO  z.l.e.CustomTracingAnnotationApp Done
```

## Feature changes

### Version 2.2.0

Deprecated log annotation with key `slf4j_logger_name` (`SLF4J.loggerNameAnnotationKey`) removed,
only common log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`) for logger name is supported now.

---

## SLF4J v2 bridge

It is possible to use `zio-logging` for SLF4J loggers, usually third-party non-ZIO libraries. To do so, import  the `zio-logging-slf4j2-bridge` module for [SLF4J v2](https://www.slf4j.org/faq.html#changesInVersion200) (using JDK9+ module system ([JPMS](http://openjdk.java.net/projects/jigsaw/spec/)))

```scala
libraryDependencies += "dev.zio" %% "zio-logging-slf4j2-bridge" % "2.5.3"
```

and use one of the `Slf4jBridge` layers when setting up logging:

```scala

program.provideCustom(Slf4jBridge.init())
```

`Slf4jBridge` layers:
* `Slf4jBridge.init(configPath: NonEmptyChunk[String] = logFilterConfigPath)` - setup with `LogFilter` from [filter configuration](log-filter.md#configuration), default configuration path: `logger.filter`, default `LogLevel` is `INFO`
* `Slf4jBridge.init(filter: LogFilter[Any])` - setup with given `LogFilter`
* `Slf4jBridge.initialize` - setup without filtering

SLF4J v2 [key-value pairs](https://www.slf4j.org/manual.html#fluent) are propagated like ZIO log annotations.

Need for log filtering in slf4j bridge: libraries with slf4j loggers, may have conditional logic for logging,
which using functions like [org.slf4j.Logger.isTraceEnabled()](https://github.com/qos-ch/slf4j/blob/master/slf4j-api/src/main/java/org/slf4j/Logger.java#L170).
logging parts may contain message and log parameters construction, which may be expensive and degrade performance of application.

SLF4J logger name is stored in log annotation with key `logger_name` (`zio.logging.loggerNameAnnotationKey`), following log format

```scala

val loggerName = LoggerNameExtractor.loggerNameAnnotationOrTrace
val loggerNameFormat = loggerName.toLogFormat()
```
may be used to get logger name from log annotation or ZIO Trace. 

This logger name extractor is used by default in log filter, which applying log filtering by defined logger name and level:

```scala
val logFilterConfig = LogFilter.LogLevelByNameConfig(
  LogLevel.Info,
  "zio.logging.slf4j" -> LogLevel.Debug,
  "SLF4J-LOGGER"      -> LogLevel.Warning
)

val logFilter: LogFilter[String] = logFilterConfig.toFilter
```

SLF4J bridge with custom logger can be setup:

```scala

val logger = Runtime.removeDefaultLoggers >>> consoleJsonLogger() >+> Slf4jBridge.init()
```

**NOTE:** You should either use `zio-logging-slf4j` to send all ZIO logs to an SLF4j provider (such as logback, log4j etc) OR `zio-logging-slf4j-bridge` to send all SLF4j logs to
ZIO logging. Enabling both causes circular logging and makes no sense.

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### SLF4J bridge with JSON console logger

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object Slf4jBridgeExampleApp extends ZIOAppDefault {

  private val slf4jLogger = org.slf4j.LoggerFactory.getLogger("SLF4J-LOGGER")

  private val logFilterConfig = LogFilter.LogLevelByNameConfig(
    LogLevel.Info,
    "zio.logging.slf4j" -> LogLevel.Debug,
    "SLF4J-LOGGER"      -> LogLevel.Warning
  )

  private val logFormat = LogFormat.label(
    "name",
    LoggerNameExtractor.loggerNameAnnotationOrTrace.toLogFormat()
  ) + LogFormat.allAnnotations(Set(zio.logging.loggerNameAnnotationKey)) + LogFormat.default

  private val loggerConfig = ConsoleLoggerConfig(logFormat, logFilterConfig)

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleJsonLogger(loggerConfig) >+> Slf4jBridge.init(loggerConfig.toFilter)

  private val uuids = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _ <- ZIO.logInfo("Start")
      _ <- ZIO.foreachPar(uuids) { u =>
        ZIO.succeed(slf4jLogger.info("Test {}!", "INFO")) *> ZIO.succeed(
          slf4jLogger.atWarn().addArgument("WARNING").log("Test {}!")
        ) @@ LogAnnotation.UserId(
          u.toString
        )
      } @@ LogAnnotation.TraceId(UUID.randomUUID())
      _ <- ZIO.logDebug("Done")
    } yield ExitCode.success

}
```

Expected Console Output:
```
{"name":"zio.logging.example.Slf4jBridgeExampleApp","timestamp":"2024-05-28T08:05:48.056029+02:00","level":"INFO","thread":"zio-fiber-735317063","message":"Start"}
{"name":"SLF4J-LOGGER","trace_id":"416f3886-717c-4359-9f6e-c260e49fd158","user_id":"580376aa-f851-43a2-8b17-9456ce005e93","timestamp":"2024-05-28T08:05:48.065133+02:00","level":"WARN","thread":"zio-fiber-1749629949","message":"Test WARNING!"}
{"name":"SLF4J-LOGGER","trace_id":"416f3886-717c-4359-9f6e-c260e49fd158","user_id":"76d49e0d-7c28-47fd-b5a1-d05afb487cfe","timestamp":"2024-05-28T08:05:48.065174+02:00","level":"WARN","thread":"zio-fiber-697454589","message":"Test WARNING!"}
{"name":"zio.logging.example.Slf4jBridgeExampleApp","timestamp":"2024-05-28T08:05:48.068127+02:00","level":"DEBUG","thread":"zio-fiber-735317063","message":"Done"}
```

---

## SLF4J v2

The Simple Logging Facade for Java ([`SLF4J v2`](https://www.slf4j.org/) - using JDK9+ module system [JPMS](http://openjdk.java.net/projects/jigsaw/spec/)) serves as a simple facade or abstraction for various logging frameworks (e.g. java.util.logging, logback, log4j).

In order to use this logging backend, we need to add the following line in our build.sbt file:

```scala
libraryDependencies += "dev.zio" %% "zio-logging-slf4j2" % "2.5.3"
```

>**_NOTE:_** SLF4J v2 implementation is similar to [v1](slf4j1.md),
however there are some differences, v1 using [MDC context](https://logback.qos.ch/manual/mdc.html), working with JDK8,
v2 using [key-value pairs](https://www.slf4j.org/manual.html#fluent), working with JDK9+.

Logger layer:

```scala

val logger = Runtime.removeDefaultLoggers >>> SLF4J.slf4j
```

Default `SLF4J` logger setup:
* logger name (by default)  is extracted from `zio.Trace`
    * for example, trace `zio.logging.example.Slf4jSimpleApp.run(Slf4jSimpleApp.scala:17)` will have `zio.logging.example.Slf4jSimpleApp` as logger name
    * NOTE: custom logger name may be set by `zio.logging.loggerName` aspect
* all annotations (logger name and log marker name annotations are excluded) are passed like [key-value pairs](https://www.slf4j.org/manual.html#fluent)
* cause is logged as throwable

See also [LogFormat and LogAppender](formatting-log-records.md#logformat-and-logappender)

Custom logger name set by aspect:

```scala
ZIO.logInfo("Starting user operation") @@ zio.logging.loggerName("zio.logging.example.UserOperation")
```

Log marker name set by aspect:

```scala
ZIO.logInfo("Confidential user operation") @@ SLF4J.logMarkerName("CONFIDENTIAL")
```

## Examples

You can find the source code [here](https://github.com/zio/zio-logging/tree/master/examples)

### SLF4J logger name and annotations

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object Slf4jSimpleApp extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] = Runtime.removeDefaultLoggers >>> SLF4J.slf4j

  private val users = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    for {
      _       <- ZIO.logInfo("Start")
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.foreachPar(users) { uId =>
        {
          ZIO.logInfo("Starting user operation") *>
            ZIO.logInfo("Confidential user operation") @@ SLF4J.logMarkerName("CONFIDENTIAL") *>
            ZIO.sleep(500.millis) *>
            ZIO.logInfo("Stopping user operation")
        } @@ ZIOAspect.annotated("user", uId.toString)
      } @@ LogAnnotation.TraceId(traceId) @@ zio.logging.loggerName("zio.logging.example.UserOperation")
      _       <- ZIO.logInfo("Done")
    } yield ExitCode.success

}
```

Logback configuration:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>%d{HH:mm:ss.SSS} [%thread] [%kvp] %-5level %logger{36} %msg%n</Pattern>
        </layout>
    </appender>
    <turboFilter class="ch.qos.logback.classic.turbo.MarkerFilter">
        <Name>CONFIDENTIAL_FILTER</Name>
        <Marker>CONFIDENTIAL</Marker>
        <OnMatch>DENY</OnMatch>
    </turboFilter>
    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>
```

Expected Console Output:
```
12:13:17.495 [ZScheduler-Worker-8] [] INFO  zio.logging.example.Slf4j2SimpleApp Start
12:13:17.601 [ZScheduler-Worker-11] [trace_id="7826dd28-7e37-4b54-b84d-05399bb6cbeb" user="7b6a1af5-bad2-4846-aa49-138d31b40a24"] INFO  zio.logging.example.UserOperation Starting user operation
12:13:17.601 [ZScheduler-Worker-10] [trace_id="7826dd28-7e37-4b54-b84d-05399bb6cbeb" user="4df9cdbc-e770-4bc9-b884-756e671a6435"] INFO  zio.logging.example.UserOperation Starting user operation
12:13:18.167 [ZScheduler-Worker-13] [trace_id="7826dd28-7e37-4b54-b84d-05399bb6cbeb" user="7b6a1af5-bad2-4846-aa49-138d31b40a24"] INFO  zio.logging.example.UserOperation Stopping user operation
12:13:18.167 [ZScheduler-Worker-15] [trace_id="7826dd28-7e37-4b54-b84d-05399bb6cbeb" user="4df9cdbc-e770-4bc9-b884-756e671a6435"] INFO  zio.logging.example.UserOperation Stopping user operation
12:13:18.173 [ZScheduler-Worker-13] [] INFO  zio.logging.example.Slf4j2SimpleApp Done
```

---

## Testing

ZIO 2 test library has test logger implementation for testing:

```scala
libraryDependencies += "dev.zio" %% "zio-test" % ZioVersion % Test
```

Test logger layer:

```scala
zio.test.ZTestLogger.default
```

You can find the source code of examples [here](https://github.com/zio/zio-logging/tree/master/examples/src/test/scala/zio/logging/example)

Test example:

[//]: # (TODO: make snippet type-checked using mdoc)

```scala
package zio.logging.example

object LoggingSpec extends ZIOSpecDefault {

  override def spec: Spec[TestEnvironment, Any] = suite("LoggingSpec")(
    test("start stop log output") {
      val users = Chunk.fill(2)(UUID.randomUUID())
      for {
        traceId      <- ZIO.succeed(UUID.randomUUID())
        _            <- ZIO.foreach(users) { uId =>
                          {
                            ZIO.logInfo("Starting operation") *> ZIO.sleep(500.millis) *> ZIO.logInfo("Stopping operation")
                          } @@ ZIOAspect.annotated("user", uId.toString)
                        } @@ LogAnnotation.TraceId(traceId)
        _            <- ZIO.logInfo("Done")
        loggerOutput <- ZTestLogger.logOutput
      } yield assertTrue(loggerOutput.size == 5) && assertTrue(
        loggerOutput.forall(_.logLevel == LogLevel.Info)
      ) && assert(loggerOutput.map(_.message()))(
        equalTo(
          Chunk(
            "Starting operation",
            "Stopping operation",
            "Starting operation",
            "Stopping operation",
            "Done"
          )
        )
      ) && assert(loggerOutput.map(_.context.get(logContext).flatMap(_.asMap.get(LogAnnotation.TraceId.name))))(
        equalTo(
          Chunk.fill(4)(Some(traceId.toString)) :+ None
        )
      ) && assert(loggerOutput.map(_.annotations.get("user")))(
        equalTo(users.flatMap(u => Chunk.fill(2)(Some(u.toString))) :+ None)
      )
    }
  ).provideLayer(
    Runtime.removeDefaultLoggers >>> ZTestLogger.default
  ) @@ TestAspect.withLiveClock
}
```

---

## Introduction to ZIO Meta

Meta-programming helpers

---

## Getting Started(Zio-metrics-connectors)

ZIO Metrics lets you observe everything that goes on in your ZIO application.

Tests are important for verifying the correctness of the code we write. However, when we go into production conditions
are inevitably different than in testing. Still, we do need to know what is going on in our application in real time so
that we can diagnose and address problems.

Logging is part of the solution, but it is not enough. We either only log in certain areas, which are often not the
areas where an issue actually occurs, or we log everywhere, significantly impacting application performance.

Diagnostics and metrics are the other pieces of the puzzle, but there has not been a solution for integrating existing
metrics backends with functional effect systems before, forcing users to develop their own workarounds. These solutions
are not connected with the runtime of the effect system, so they have no ability to provide insights on what is going
on at the level of the runtime such as the activity of individual fibers.

ZIO Metrics solves this problem. ZIO Metrics can be added to any application with only a few lines of code and provides
two types of insights:

- **Diagnostics** — Diagnostics are pre-defined data on the behavior of the application as it is running directly by the
  ZIO runtime, such as the number of fibers in the application, distribution of fiber lifetimes, and a breakdown of
  reasons for fiber termination.
- **[Metrics](metrics/metric-reference.md)** — Metrics are used-defined data, such as the number of times a certain
  section of code was executed, that are tracked by ZIO Metrics and made available either directly or through third
  party metrics solutions such as Prometheus, StatsD or Micrometer.

See the corresponding sections for more information on how to use each of these pieces of functionality.

## Installation

Choose and include ZIO Metrics connector in your project by adding the following to your `build.sbt` (for example
prometheus connector):

```
libraryDependencies += "dev.zio" %% "zio-metrics-connectors-prometheus" % "2.5.4"
```

---

## Introduction to ZIO Metrics Connectors

ZIO Metrics Connectors provides integration between ZIO metrics and industry-leading monitoring and observability systems such as
  * [Prometheus](https://prometheus.io/)
  * [Datadog](https://docs.datadoghq.com/)
  * [New Relic](https://newrelic.com)
  * [StatsD](https://github.com/statsd/statsd)
  * [Micrometer](https://micrometer.io)

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-metrics-connectors/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-metrics-connectors_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-metrics-connectors_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-metrics-connectors_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-metrics-connectors_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-metrics-connectors-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-metrics-connectors-docs_2.13) [![ZIO Metrics Connectors](https://img.shields.io/github/stars/zio/zio-metrics-connectors?style=social)](https://github.com/zio/zio-metrics-connectors)

Get started by clicking [here](metrics/index.md#adding-zio-metrics-connectors-to-your-project)

---

## Datadog Client

In a normal DataDog setup we will find a DataDog agent with an open UDP port where applications send their
metrics to. The format of the metrics is defined by the [DogStatsD protocol](https://docs.datadoghq.com/developers/dogstatsd/datagram_shell/?tab=metrics).

With the DataDog client zio metrics creates the relevant DogStatsD datagrams and sends them via UDP.

> The instrumented code is exactly the same as for the Prometheus instrumentation. The only difference is that
> another client is provided when the App is configured.

## ZIO metrics in DogStatsD

DogStatsD normally has its own definition how histograms and summaries are configured. In the default setup this
is defined in the config file of the statsd agent. Furthermore, a StatsD Histogram is more or less the
equivalent of a Prometheus summary.

However, whenever the desired quantiles need to change, the config must be adjusted and the agent restarted.

Therefore, the ZIO Metrics Datadog client maps the more complex metrics to a set of related gauges. This allows us to achieve
the same visualization without the need to adjust any of the agents config files.

### Counter

Counters are reported periodically within a configurable time interval.

```
countAll:1|c
```

### Gauge

Gauges are reported periodically within a configurable time interval.

```
adjustGauge:32.2787317766752|g
```

### Histogram

A histogram is reported as a datadog distribution whenever a new value is submitted. The bucket configuration is 
ignored in this case, as DataDog expects raw values to be recorded. 

```
myHistogram:3|d
```

### Summary

A summary is periodically reported as a set of related gauges, within a configurable time interval. Each quantile will 
be reported within its own gauge. The additional labels are `quantile` to address the quantile and `error` to display 
the configured error margin.

```
mySummary:123|g|#quantile:0.1,error:0.03
mySummary:254|g|#quantile:0.5,error:0.03
mySummary:441|g|#quantile:0.9,error:0.03
```

### Sets

Sets are also reported as sets of related gauges. An additional label is used to differentiate the distinct
values in the set. The name of the label is as configured in the aspect used to capture the set.

```
mySet:6|g|#token:myKey-18
mySet:1|g|#token:myKey-19
mySet:2|g|#token:myKey-13
mySet:3|g|#token:myKey-14
mySet:1|g|#token:myKey-16
mySet:3|g|#token:myKey-10
mySet:2|g|#token:myKey-11
mySet:1|g|#token:myKey-12
```

## The ZIO metrics DataDog example

```scala

val instrumentedSample = new InstrumentedSample() {}
```

For DataDog we do need to spin up our own server. Rather we need to provide a client that can send datagrams
to a specified UDP destination or a unix domain socket.

Again we need an effect that runs our instrumented code until the user presses any key:

```scala 
val execute =
  for {
   fiber <- instrumentedSample.program.fork
    _ <- putStrLn("Press Any Key") *> getStrLn.orDie 
    _ <- fiber.interrupt
  } yield ExitCode.success
```  

Now, we can override the `run` method of our ZIO `App` and simply provide a `datadog.datadogLayer`.

```scala 
  override def run: ZIO[ZIOAppArgs with Scope, Any, Any] =
    execute
      .provide(
        ZLayer.succeed(MetricsConfig(100.millis)),
        ZLayer.succeed(statsd.StatsdConfig("localhost", 8125)),
        ZLayer.succeed(datadog.DatadogPublisherConfig()),
        statsd.statsdUDP,
        datadog.live,
      )
      .orDie
```

Alternatively, we can use the `statsdClient.liveDatagram` layer which is a convenience layer that provides an ability to send datagrams
via a UDS socket this can be used with datadog agent starting with version 6.0.0 on the linux platform.

```scala 
  override def run: ZIO[ZIOAppArgs with Scope, Any, Any] =
    execute
      .provide(
        ZLayer.succeed(MetricsConfig(100.millis)),
        ZLayer.succeed(statsd.DatagramSocketConfig("/var/run/datadog/datadog.sock")),
        ZLayer.succeed(datadog.DatadogPublisherConfig()),
        statsd.statsdUDS,
        datadog.live,
      )
      .orDie
```

## A simple StatsD / Datadog setup

The following steps describe how to set up a ZIO application reporting to [Datadog](https://www.datadoghq.com/) using a free Datadog account
with limited functionality. The local setup is Windows 10 with WSL and Docker installed running Ubuntu 18.04 within WSL.

In principle the setup is as follows:

1. The ZIO application sends datagrams to `/var/run/datadog/datadog.sock` via the datagram unix domain socket.
1. Run a Datadog Collector within a docker image exposing a Unix socket for datagrams.
1. Configure a dashboard in Datadog to visualize the metrics.

### Get and run the docker based Datadog collector

Upon registration with datadoghq.com you will get an API key which is required to configure the agents collecting data. If you are planning
to experiment with different agents, take a note of your API key for further reference. In the steps below the API key will be referred to
as `$APIKEY`

For our example we have chosen to use the docker based collector and use a unix socket to report our datagrams to that agent.

You can start the agent from the command line with

```
docker run --name dd-agent -e DD_API_KEY=$APIKEY -e DD_SITE=datadoghq.eu \
  -e DD_DOGSTATSD_SOCKET=/var/run/datadog/datadog.sock \
  -v /var/run/datadog:/var/run/datadog \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  datadog/agent
```

As you can see, we require that the directory `/var/run/datadog` exists so that we can use it as a volume within the agent's docker container. The environment variable `DD_DOGSTATSD_SOCKET` tells the agent to use a unix socket to listen for datagrams. The socket file must reside within the mounted volume.

This will start the datadog collector within docker and we have a unix socket to report our datagrams to.

### Run the Datadog example

Now, the Datadog example can be started from within the project checkout directory with

```
sbt sampleApp/run
```

### Visualize the metrics

1. Log in to your datadog account
1. From the menu on left hand side select `Dashboards/New Dashboard'
1. In the upper right corner, click on the dashboard settings and select 'Import Dashboard JSON'
1. From the filesystem, select `$DIR/examples/statsd/ZIOMetrics.json`
1. Confirm to override the dashboard configuration
1. Save the just imported dashboard
1. From the dashboard list select _ZIO metrics_
1. The Datadog dashboard is displayed

### Datadog dashboard

![A simple Datadog Dashboard](../img/Datadog.png)

---

## Metrics

ZIO Metrics allows you to apply special aspects to the workflows of any ZIO based application. This is called
instrumentation. The type of the original ZIO workflow will not change by adding one or more aspects to it. (From here
on, we'll use the word _effect_ instead of workflow.)

Whenever an instrumented effect executes, all the aspects will be executed as well. Each of the aspects will capture
some data of interest and update some Metric internal state. Which data will be captured and how it can be used later on
is dependent on the metric type associated with the aspect. Capturing metrics data is usually called 'measuring'.

Metrics are normally collected to be displayed in an application like [Grafana](https://grafana.com/) or a cloud based
platform like [DatadogHQ](https://docs.datadoghq.com/) or [NewRelic](https://newrelic.com).
In order to support such a range of different platforms, the metric state is kept in an internal data structure which is
optimized to update as efficiently as possible. The data required by the collecting platforms is generated only when it
is required.

ZIO Metrics Connectors also allows us to dump current metric states (in one of the next minor releases) out of the box
to analyze the metrics in the development phase before the decision for a metric platform has been made or in cases when
the platform might not be feasible to use in development.

> Changing which platform the metrics are reported to has no impact on the application at all. Once instrumented
> properly, the decision where to report to happens 'at the end of the world', that is, in the ZIO application's main
> entry point, by providing one or more of the available reporting clients.

Currently, ZIO Metrics Connectors provides integrations with the following reporting clients:
* [Prometheus](https://prometheus.io/)
* [Datadog](https://docs.datadoghq.com/)
* [New Relic](https://newrelic.com)
* [StatsD](https://github.com/statsd/statsd)
* [Micrometer](https://micrometer.io)

### Adding ZIO Metrics Connectors to your project

Import the corresponding dependency based on your reporting infrastructure:
```sbt
libraryDependencies ++= {
  Seq(
    "dev.zio" %% "zio-metrics-connectors"             % "2.5.4", // core library
    "dev.zio" %% "zio-metrics-connectors-prometheus"  % "2.5.4", // Prometheus client
    "dev.zio" %% "zio-metrics-connectors-datadog"     % "2.5.4", // DataDog client
    "dev.zio" %% "zio-metrics-connectors-newrelic"    % "2.5.4", // NewRelic client
    "dev.zio" %% "zio-metrics-connectors-statsd"      % "2.5.4", // StatsD client
    "dev.zio" %% "zio-metrics-connectors-micrometer"  % "2.5.4"  // Micrometer client
  )
}
```

Please refer to:

* [Metrics Reference](metric-reference.md) for more information on the metrics currently supported
* [Prometheus Client](prometheus-client.md) to learn more about the mapping from ZIO Metrics to Prometheus
* [StatsD Client](statsd-client.md) to learn more about the mapping from ZIO Metrics to StatsD
* [DataDog Client](datadog-client.md) to learn more about the mapping from ZIO Metrics to DataDog
* [Micrometer](micrometer-connector.md) to learn more about the mapping from ZIO Metrics to Micrometer

---

## Instrumentation Examples

The trait below is used in the
[zio metrics sample application](https://github.com/zio/zio-metrics-connectors/blob/docs/sample-app/src/main/scala/sample/SamplePrometheusStatsDApp.scala)
that is provided in the zio-metrics-connectors repository on GitHub. It shows how the individual metrics can be
configured and used.

Please refer to [StatsD](statsd-client.md) and [Prometheus](prometheus-client.md) to see how the captured metrics can be
visualized in the supported back ends.

```scala
trait InstrumentedSample {

  // Create a gauge, it can be applied to effects yielding a Double
  val aspGauge1 = Metric.gauge("gauge1", "Description of gauge1")
    .tagged("tag1", "tag value")
    .tagged("tag2", "anoter tag value")

  val aspGauge2 = Metric.gauge("gauge2")

  // Create a histogram with 12 buckets: 0..100 in steps of 10, Infinite
  // It also can be applied to effects yielding a Double
  val aspHistogram =
    Metric.histogram("my_histogram", MetricKeyType.Histogram.Boundaries.linear(0.0d, 10.0d, 11))

  // Create a summary that can hold 100 samples, the max age of the samples is 1 day.
  // The summary should report th 10%, 50% and 90% Quantile
  // It can be applied to effects yielding an Int
  val aspSummary =
    Metric.summary("my_summary", 1.day, 100, 0.03d, Chunk(0.1, 0.5, 0.9)).contramap[Int](_.toDouble)

  // Create a Set to observe the occurrences of unique Strings
  // It can be applied to effects yielding a String
  val aspSet = Metric.frequency("my_set")

  // Create a counter applicable to any effect
  val aspCountAll = Metric.counter("count_all").contramap[Any](_ => 1L)

  private lazy val gaugeSomething = for {
    _ <- Random.nextDoubleBetween(0.0d, 100.0d) @@ aspGauge1 @@ aspCountAll
    _ <- Random.nextDoubleBetween(-50d, 50d) @@ aspGauge2 @@ aspCountAll
  } yield ()

  // Record something into a histogram
  private lazy val observeNumbers = for {
    _ <- Random.nextDoubleBetween(0.0d, 120.0d) @@ aspHistogram @@ aspCountAll
    _ <- Random.nextIntBetween(100, 500) @@ aspSummary @@ aspCountAll
  } yield ()

  // Observe Strings in order to capture unique values
  private lazy val observeKey = for {
    _ <- Random.nextIntBetween(10, 20).map(v => s"my_key_$v") @@ aspSet @@ aspCountAll
  } yield ()

  def program: ZIO[Any, Nothing, Unit] = for {
    _ <- gaugeSomething.schedule(Schedule.spaced(200.millis)).forkDaemon
    _ <- observeNumbers.schedule(Schedule.spaced(150.millis)).forkDaemon
    _ <- observeKey.schedule(Schedule.spaced(300.millis)).forkDaemon
  } yield ()
}
```

---

## ZIO Metric Reference

## Measuring

All ZIO metrics are defined in the form of aspects that can be applied with `@@` to measure an effect. Applying an
aspect to an effect does not change the type of the effect. For example:

```scala
val metric = Metric.counter("effect_counter")
val effect: ZIO[Any, Throwable, Double] = ???
val effectWithMeasurement: ZIO[Any, Throwable, Double] = effect @@ metric
```

### Name, description and labels

Every metric has a name and optionally a description. The name may be augmented by zero or more `Label`s (sometimes
called `tag`s). Labels are useful, because some reporting platforms support them and provide aggregation mechanisms
for them.

> For example, think of a counter that counts how often a particular service has been invoked. If the application is
> deployed across several hosts, we can model our counter with the name `my_service` and an additional label
> `(host, ${hostname})`. With this definition we can see the number of executions for each host, but we can also
> create a query in Grafana or DatadogHQ to visualize the aggregated value over all hosts. Using more than one label
> allows us to create visualizations across any combination of the labels.

Name and description are always given as part of the metric constructor, labels can be added via one of the `tagged`
methods. Here is an example:

```scala
val countRequests = Metric.counter("count_requests", "Description of the counter").tagged("hostname", hostname)
val effect: ZIO[Any, Throwable, Double] = ???
val effectWithTaggedMeasurement = effect @@ countRequests.tagged("path", path)
```

We recommend you follow the [Prometheus best practices](https://prometheus.io/docs/practices/naming/) guide for the metric name and labels.

### Supported type

Every `Metric` has the type parameter `In` which indicates what types of values can be measured by the metric. When
applying the metric as an aspect to an effect, the metric's `In` type must be compatible with the output type of the
effect (the `A` in `ZIO[R, E, A]`). For example, a `Metric.Counter[Any]` can be applied to any effect while a
`Metric.Counter[Double]` can only be applied to an effect that produces a `Double`.

In the underlying implementation, each metric type supports only one base type. _Counters_, _Gauges_, _Histograms_ and
_Summaries_ all support `Double` values, while a _Frequency_ supports `String` values. To support a different type, we
can use the `contramap` method. For example, a Gauge that supports `Long`s is written as:

```scala
val longGauge: Metrics.Gauge[Long] =
  Metric.Gauge("my_metric").contramap[Long](_.toDouble)
```

Method `fromConst` works like `contramap` but always returns the same value. For example a counter that always increases
with the same value is written as:

```scala
val occurrenceCounter: Metric.Counter[Any] =
  Metric.counter("my_metric").fromConst(1L)
```

### Measuring durations

To measure how long an effect takes, we can use methods `trackDuration` and `trackDurationWith`. See the histogram
section below for an example.

### Measuring failures

Normally, we only measure effects that succeed. To measure an effect that failed, we can use methods like `trackAll`,
`trackDefectWith`, `trackErrorWith` and `trackSuccessWith`. For example `trackError` lets you measure the error output
of an effect:

```scala
val countAllErrors =
  Metric.counter("all_errors").fromConst(1L).trackError

// counts the errors of `effect`
val effectWithMeasurement = effect @@ countAllErrors
```

# Metric types

## Counter

A counter is a metric of which the value can only increase over time. A counter can be created with one of these
methods:

```scala
val longCounter: Metric.Counter[Long] = Metric.counter(name = "my_metric")
val intCounter: Metric.Counter[Int] = Metric.counterInt(name = "my_metric")
val doubleCounter: Metric.Counter[Double] = Metric.counterDouble(name = "my_metric")
```

Besides using a counter as an aspect (with the `@@` operator), counter metrics can also measure values with their
`increment` and `incrementBy` methods. For example:

```scala
longCounter.incrementBy(10L)
```

### Examples

Create a counter named `count_all` which is incremented by `1` every time it is invoked. Since the metric type is `Any`
the aspect can be applied to effect of any type.

```scala 
val aspCountAll: Metric.Counter[Any] =
  Metric.counter("count_all").fromConst(1L)
```

The same aspect can be applied to more than one effect. In the following example we count the sum of executions of both
effects in the for comprehension:

```scala 
val countAll = for {
  _ <- ZIO.unit @@ aspCountAll
  _ <- ZIO.unit @@ aspCountAll
} yield ()
```  

Create a counter named `count_bytes` that can be applied to effects having the output type `Double`.

```scala
val aspCountBytes = Metric.counterDouble("count_bytes")
```

Now we can apply it to effects producing `Double` (in a real application the value might be the number of bytes read
from a stream or something similar):

```scala 
val countBytes = nextDoubleBetween(0.0d, 100.0d) @@ aspCountBytes
```

## Gauges

A gauge is a metric that can change over time. A gauge can be created as follows:

```scala
val doubleGauge: Metric.Gauge[Double] = Metric.gauge(name = "my_metric")
```

Besides using a gauge as an aspect (with the `@@` operator), gauge metrics can also measure values with their `set`,
`increment`, `incrementBy`, `decrement` and `decrementBy` methods. For example:

```scala
for {
  _ <- doubleGauge.set(10.0) // sets the gauge to 10.0
  _ <- doubleGauge.decrement // decreases the gauge with 1.0
} yield ()
```

### Examples

Create a gauge that can be applied to effects that produce a Double:

```scala
val aspGauge = Metric.gauge("set_gauge")
```

Now we can apply the metric to effects that produce `Double`s. Note that we can instrument an effect with any number of
aspects if the type constraints are satisfied.

```scala 
val measuredEffect = nextDoubleBetween(0.0d, 100.0d) @@ aspGauge @@ aspCountAll
```

## Histograms

A histogram is a metric that counts the measured values in buckets. Each bucket is defined by an upper boundary. The
count in a bucket with upper boundary `b` increases by `1` if an observed value `v` is less or equal to `b`. As a
consequence, all buckets that have a boundary `b1` with `b1 > b` will increase by `1` after observing `v`.

A histogram also keeps track of the overall count of observed values and the sum of all observed values.

The last bucket is always defined as `Double.MaxValue`, so that the count of observed values in the last bucket is
always equal to the overall count of observed values within the histogram.

The used histogram model is inspired by [Prometheus](https://prometheus.io/docs/concepts/metric_types/#histogram).

Boundaries can be created as follows:

```scala
// given boundaries: 100.0, 200.0, 300.0, 400.0, MaxValue
MetricKeyType.Histogram.Boundaries.fromChunk(Chunk(100.0, 200.0, 300.0, 400.0))
// linear boundaries: 100.0, 200.0, 300.0, 400.0, 500.0, 600.0, 700.0, 800.0, 900.0, 1000.0, MaxValue
MetricKeyType.Histogram.Boundaries.linear(100.0, 100.0, 10)
// exponential boundaries: 100.0, 200.0, 400.0, 800.0, 1600.0, 3200.0, 6400.0, 12800.0, 25600.0, 51200.0, MaxValue
MetricKeyType.Histogram.Boundaries.exponential(100.0, 2.0, 10)
```

A histogram can be created with the `Metric.histogram(name, boundaries)` method. For example:

```scala
val latencyHistogram: Metric.Histogram[Double] =
  Metric.histogram("queue_size", boundaries)
```

### Examples

Create a histogram with 10 buckets: `0..100` in steps of `10` and `Double.MaxValue`. It can be applied to effects
yielding a `Double`.

```scala 
val aspHistogram =
  Metric.histogram("my_histogram", Histogram.Boundaries.linear(0.0d, 10.0d, 10))
```

Now we can apply the histogram to effects producing `Double`:

```scala 
val measuredEffect = nextDoubleBetween(0.0d, 120.0d) @@ aspHistogram 
```

Create a histogram that observes `Duration`s in seconds. Use `trackDuration` to track the duration of an effect.
Note that using seconds as unit is the recommended convention in many metric platforms such as Prometheus and
OpenTelemetry.

```scala
val latencyHistogram: Metric.Histogram[Duration] =
  Metric.histogram(
      "my_latency_seconds",
      "The latency in seconds.",
      MetricKeyType.Histogram.Boundaries.exponential(0.01, 2.0, 10)
    )
    .contramap[Duration](_.toNanos.toDouble / 1e9) // convert to seconds

val effect: ZIO[Any, Throwable, Unit] = ???

effect @@ latencyHistogram.trackDuration
```

## Summaries

Similar to a histogram a summary also observes `Double` values. While a histogram directly modifies the bucket counters
and does not keep the individual samples, the summary keeps the observed samples in its internal state. To avoid the set
of samples grow uncontrolled, the summary need to be configured with a maximum age `t` and a maximum size `n`. To
calculate the statistics, maximal `n` samples will be used, all of which are not older than `t`.

Essentially the set of samples is a sliding window over the last observed samples matching the conditions above.

A summary is used to calculate a set of quantiles over the current set of samples. A quantile is defined by a `Double`
value `q` with `0 <= q <= 1` and resolves to a `Double` as well.

The value of a given quantile `q` is the maximum value `v` out of the current sample buffer with size `n` where at most
`q * n` values out of the sample buffer are less or equal to `v`.

Typical quantiles for observation are `0.5` (the median) and the `0.95`. Quantiles are very good for monitoring Service
Level Agreements.

The ZIO Metrics API also allows summaries to be configured with an error margin `e`. The error margin is applied to the
count of values, so that a quantile `q` for a set of size `s` resolves to value `v` if the number `n` of values less or
equal to `v` is `(1 -e)q * s <= n <= (1+e)q`.

A summary can be created with the `summary` method:

```scala
Metric.summary(
  name: String,
  maxAge: Duration,
  maxSize: Int,
  error: Double,
  quantiles: Chunk[Double]
): Metric.Summary[Double]
```

### Examples

Create a summary that can hold 100 samples, the max age of the samples is `1 day` and the error margin is `3%`. The
summary should report the `10%`, `50%` and `90%` quantiles. It can be applied to effects producing an `Int`.

```scala
val aspSummary =
  Metric.summary("my_summary", 1.day, 100, 0.03d, Chunk(0.1, 0.5, 0.9)).contramap[Int](_.toDouble)
```

Now we can apply this aspect to an effect producing an `Int`:

```scala
val summary = nextIntBetween(100, 500) @@ aspSummary
```

## Frequencies

Frequencies are used to count the occurrences of distinct string values. For example an application that uses logical
names for its services, the number of invocations for each service can be tracked.

Essentially, a Frequency is a set of related counters sharing the same name and tags. The counters are set apart from
each other by an additional configurable tag. The values of the tag represent the observed distinct values.

To configure a frequency aspect, the name of the tag holding the distinct values must be configured.

A frequency can be created with the `frequency` method:

```scala
Metric.frequency(name: String): Metric.Frequency[String]
```

### Examples

Create a Frequency to observe the occurrences of unique Strings. It can be applied to effects producing a String.

```scala
val aspSet = Metric.frequency("my_set")
```  

Now we can generate some keys within an effect and start counting the occurrences
for each value.

```scala 
val set = nextIntBetween(10, 20).map(v => s"my_key_$v") @@ aspSet
```

---

## Micrometer Connector

ZIO Metrics has an integration with [Micrometer](https://micrometer.io/), a powerful metrics instrumentation library.
By combining these two frameworks, developers can benefit from comprehensive and flexible metrics monitoring capabilities
within their ZIO applications. This integration allows for efficient monitoring, gathering, and reporting of key
performance indicators (KPIs) and other metrics for enhanced observability.

## Benefits

Micrometer integration offers a range of benefits that make it an excellent choice for integrating with ZIO Metrics.
Here are some key advantages:

### 1. Comprehensive Metrics Support

Micrometer provides a vendor-neutral facade for various monitoring systems, including Prometheus, Datadog, Graphite,
and more. The same like slf4j for logs but for observability. It offers a unified API for recording metrics, allowing
developers to easily integrate with multiple backend systems and switch between different monitoring systems without
major code changes for its ZIO applications. You can choose whatever you want as monitoring tool - Grafana,
Prometheus, and Jaeger and so on.

### 2. Real-time Metrics Updates

By leveraging the low-level integration with the ZIO Metrics core client listener, the module provides immediate
updates to Micrometer whenever metric changes occur within the ZIO application. This real-time synchronization
ensures that the metrics reported by Micrometer accurately reflect the current state of the application, enabling
timely monitoring and analysis.

### 3. Always on trend of standards

Integration with Micrometer ensures that developers are always on trend of standards with metrics format changes and
can take advantage of the latest updates and improvements in the monitoring ecosystem. By relying on Micrometer as
the integration layer, developers gain compatibility with the latest monitoring systems, future-proof their
monitoring infrastructure, benefit from community-driven updates and support, and simplify maintenance and upgrades.
This integration provides a reliable and scalable solution for metrics monitoring in ZIO-based applications, allowing
developers to focus on their core business logic while staying at the forefront of monitoring technology.

## Example of usage

1. Import Dependencies

   In your project's build configuration, add the following dependency to import the ZIO Metrics Micrometer module:

```
   libraryDependencies += "dev.zio" %% "zio-metrics-connectors-micrometer" % latest
```

2. Choose Micrometer Backend

   Decide on the backend for the Micrometer registry. You can use the built-in
   'SimpleMeterRegistry', which stores all metrics in memory (commonly used for testing), or select another backend for
   external integration. In this example, we'll use Prometheus as the backend.

```
   "io.micrometer" % "micrometer-registry-prometheus" % latest
```

3. Provide Micrometer and its backend Layer to your main ZIO effect

   `zio-metrics-connectors-micrometer` gives you `micrometer.micrometerLayer` which initializes a bridge between ZIO Metrics and
   Micrometer.
   You should also provide a layer with a backend for micrometer.

4. Optionally enable Core ZIO Metrics and Default JVM Metrics

   Enable core ZIO Metrics by providing `Runtime.enableRuntimeMetrics` to your main application's ZIO effect. This
   ensures that basic ZIO metrics, such as fiber counts and execution times, are collected. Additionally, include
   `DefaultJvmMetrics.live` in your layer composition to enable default JVM metrics collection.

Here's an example of how example may look like:

```scala

object SampleApp extends ZIOAppDefault {

  def program[R, E, T]: ZIO[R, E, T] = ???

  override def run: ZIO[Environment & ZIOAppArgs & Scope, Any, Any] = (for {
    _ <- program
  } yield ())
    .provide(
      micrometer.micrometerLayer,
      ZLayer.succeed(new PrometheusMeterRegistry(PrometheusConfig.DEFAULT)),
      Runtime.enableRuntimeMetrics,
      DefaultJvmMetrics.live.unit,
    )
}

```

---

## Prometheus Client

In a normal prometheus setup we will find prometheus agents which query configured endpoints at regular intervals. The
endpoints are HTTP endpoints serving the current metric state in an encoding defined by
[prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format).

ZIO Metrics Connectors provides the Prometheus encoding for the captured metrics out of the box. To avoid enforcing a
particular HTTP implementation, an instrumented application needs to expose the encoded format as an endpoint with the
HTTP server of its choice.

## ZIO Metrics in Prometheus

Most of the ZIO metrics have a direct representation in the Prometheus encoding.

### Counter

A counter is represented as a prometheus counter.

```scala
Metrics.counter("count_all", "The counter description.")
```

```
# TYPE count_all counter
# HELP count_all The counter description.
count_all 460.0 1623586224730
```

### Gauge

A gauge is represented as a prometheus gauge.

```scala
Metrics.gauge("adjust_gauge", "The gauge description.")
```

```
# TYPE adjust_gauge gauge
# HELP adjust_gauge The gauge description.
adjust_gauge -1.2485836762095701 1623586224730
```

### Histogram

A histogram is represented as a prometheus histogram.

```scala
Metrics.histogram(
  "my_histogram",
  "My histogram description.",
  MetricKeyType.Histogram.Boundaries.linear(0.0, 10.0, 11)
)
```

```
# TYPE my_histogram histogram
# HELP my_histogram My histogram description.
my_histogram{le="0.0"} 0.0 1623586224730
my_histogram{le="10.0"} 8.0 1623586224730
my_histogram{le="20.0"} 18.0 1623586224730
my_histogram{le="30.0"} 30.0 1623586224730
my_histogram{le="40.0"} 44.0 1623586224730
my_histogram{le="50.0"} 51.0 1623586224730
my_histogram{le="60.0"} 59.0 1623586224730
my_histogram{le="70.0"} 65.0 1623586224730
my_histogram{le="80.0"} 76.0 1623586224730
my_histogram{le="90.0"} 88.0 1623586224730
my_histogram{le="100.0"} 95.0 1623586224730
my_histogram{le="+Inf"} 115.0 1623586224730
my_histogram_sum 6828.578655207023 1623586224730
my_histogram_count 115.0 1623586224730
```

### Summary

A summary is represented as a prometheus summary.

```scala
Metric.summary(
  "my_summary",
  "The summary description.",
  maxAge = ???,
  maxSize = ???,
  error = ???,
  quantiles = Chunk(0.1, 0.5, 0.9),
)
```

```
# TYPE my_summary summary
# HELP my_summary The summary description. 
my_summary{quantile="0.1",error="0.03"} 147.0 1623589839194
my_summary{quantile="0.5",error="0.03"} 286.0 1623589839194
my_summary{quantile="0.9",error="0.03"} 470.0 1623589839194
my_summary_sum 42582.0 1623589839194
my_summary_count 139.0 1623589839194
```

### Frequency

A frequency is represented by a set of prometheus counters, distinguished from each other with a label.

```scala
val aspSet = Metric.frequency("my_set", "My set description.")
ZIO.succeed("my_key_10") @@ aspSet
ZIO.succeed("my_key_11") @@ aspSet
// etc.
```

```
# TYPE my_set counter
# HELP my_set My set description.
my_set{token="my_key_17"} 7.0 1623589839194
my_set{token="my_key_18"} 9.0 1623589839194
my_set{token="my_key_19"} 12.0 1623589839194
my_set{token="my_key_13"} 6.0 1623589839194
my_set{token="my_key_14"} 4.0 1623589839194
my_set{token="my_key_15"} 6.0 1623589839194
my_set{token="my_key_16"} 5.0 1623589839194
my_set{token="my_key_10"} 10.0 1623589839194
my_set{token="my_key_11"} 1.0 1623589839194
my_set{token="my_key_12"} 10.0 1623589839194
```

## Serving Prometheus metrics through HTTP

ZIO Metrics Connectors provides a prometheus client that can be used to produce the prometheus encoded metric state upon
request. The state is encoded and stored in the `PrometheusPublisher` class. To retrieve the prometheus encoded state,
the application can use the `PrometheusPublisher#get` method:

```scala
val encodedContent: ZIO[PrometheusPublisher, Nothing, String] =
   ZIO.serviceWithZIO[PrometheusPublisher](_.get)
```

The resulting string can then be used in an HTTP response that has a `Content-Type` header with value
`text/plain; version=0.0.4` (just `text/plain` works for as long as Prometheus does not define a newer format).

## Example HTTP server using zio-http

This section shows some code fragments from the
[example application](https://github.com/zio/zio-metrics-connectors/blob/docs/sample-app/src/main/scala/sample/SamplePrometheusStatsDApp.scala)
that is provided in the zio-metrics-connectors repository on GitHub. In the example we
use [zio-http](https://github.com/zio/zio-http) (version `3.0.0-RC4`) to serve the metrics.

First we define the metrics route:

```scala
val prometheusRoute: Route[PrometheusPublisher, Nothing] =
  Method.GET / "metrics" -> handler {
    ZIO.serviceWithZIO[PrometheusPublisher](_.get).map { response =>
      Response(
        status = Status.Ok,
        headers = Headers(Header.Custom(Header.ContentType.name, "text/plain; version=0.0.4")),
        body = Body.fromString(response, StandardCharsets.UTF_8)
      )
    }
  }
```

We convert the route to an `HttpApp` and create an HTTP server:

```scala
val httpApp = prometheusRoute.toHttpApp
val runHttp = (Server.serve(httpApp) *> ZIO.never).forkDaemon
val serverConfig = ZLayer.succeed(Server.Config.default.port(8080))
```

Now we can wire everything together in a ZIO app:

```scala
object SamplePrometheusApp extends ZIOAppDefault {
  // How often should the prometheus publisher update the response:
  private val metricsConfig = ZLayer.succeed(MetricsConfig(5.seconds))

   override def run: ZIO[Scope, Any, Any] = (for {
      f <- runHttp
      _ <- ZIO.sleep(1.minute) // Your program logic here
      _ <- f.join
   } yield ())
     .provide(
        serverConfig,
        Server.live,

        // The prometheus reporting layer
        metricsConfig,
        prometheus.publisherLayer,
        prometheus.prometheusLayer,

        // Enable the ZIO internal metrics and the default JVM metricsConfig
        // Do NOT forget the .unit for the JVM metrics layer
        Runtime.enableRuntimeMetrics,
        DefaultJvmMetrics.live.unit,
     )
}
```

## Running the full example app

Any of the examples can be run from a command line within the project checkout directory with:

```
sbt sampleApp/run
```

Out of the choices, select the option corresponding to `sample.SamplePrometheusStatsDApp`.

If everything works, we should be able to use a web browser and go to `http://localhost:8080/metrics` to see something
like:

```
# TYPE my_counter counter
# HELP my_counter
my_counter{effect="count2"} 46.0 1608982756235
# TYPE set_gauge gauge
# HELP set_gauge
set_gauge 8.66004641453435 1608982756235
# TYPE change_gauge gauge
# HELP change_gauge
change_gauge 90.7178906485008 1608982756235
# TYPE my_counter counter
# HELP my_counter
my_counter{effect="count1"} 92.0 1608982756235
```

Once we see the metrics being served from our example, we can set up Prometheus and Grafana to create a dashboard
displaying our metrics.

## Prometheus setup

The following steps have been tested on Ubuntu 18.04 running inside the Windows Subsystem for Linux. Essentially, you
need to download the prometheus binaries for your environment and start the server with our sample configuration located
at

```
${PROJECT_HOME}/examples/prometheus/promcfg.yml
``` 

This will configure a prometheus job that regular polls `http://localhost:8080/metrics` for prometheus encoded metrics.

In addition, you need to download the Grafana binaries for your installation, start the Grafana server and configure
prometheus as a single data source.

Finally, you can import our example dashboard at `examples/prometheus/ZIOMetricsDashboard.json` and enjoy the results.

> These steps are not intended to replace the Prometheus or Grafana documentation. Please refer to their websites for
> guidelines towards a more sophisticated setup or an installation on different platforms.

---

### Download and configure Prometheus

In the steps below the project checkout directory will be referred to as `$DIR`.

1. [Download](https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz)
   prometheus
1. Extract the downloaded archive to a directory of your choice, this will be referred to as `$PROMDIR`.
1. Within `$PROMDIR` execute
   ```
   ./prometheus --config.file $DIR/examples/prometheus/promcfg.yml
   ```
   This will start the prometheus server which regularly polls the HTTP endpoint of the example above for its metrics.

### Download and configure Grafana

1. [Download](https://dl.grafana.com/oss/release/grafana-7.3.6.linux-amd64.tar.gz) grafana
1. Extract the downloaded archive to a directory of your choice, this will be referred to as `$GRAFANADIR`.
1. Within `$GRAFANADIR` execute
   ```
   ./bin/grafana-server
   ```
   This will start a Grafana server.
1. Now you should be able to login to Grafana at `http://localhost:3000' with the default user `admin` with the default
   password `admin`.

   Upon the first login you will be asked to change the default password.
1. Within the Grafana menu on the left hand side you will find `Manage Dashboards` within that page, select `Import`.
1. You can now either install a dashboard from grafana.com or use a text field to paste JSON.
1. Paste the content of `$DIR/examples/prometheus/ZIOMetricsDashboard.json` into the text field and select `Load`.

   This will import our dashboard example.
1. Now, under `Manage Dashboards` the just imported ZIO dashboard should be visible.
1. Navigate to the dashboard.

### Grafana dashboard

Here is a screenshot of the Grafana dashboard produced with the setup above.

![A simple Grafana Dashboard](../img/ZIOMetrics-Grafana.png)

---

## StatsD Client

```scala

```

In a normal StatsD setup we will find a StatsD agent with an open UDP port where applications send their 
metrics to. The format of the metrics is defined in simple [datagrams](https://docs.datadoghq.com/developers/dogstatsd/datagram_shell/?tab=metrics). 

With the StatsD client creates the relevant StatsD datagrams and sends them via UDP. 

> The instrumented code is exactly the as for the Prometheus instrumentation. The only difference is that 
> another client is provided when the App is configured. 

## ZIO metrics in StatsD

StatsD normally has its own definition how histograms and summaries are configured. In the default setup this 
is defined in the config file of the statsd agent. Furthermore, a StatsD Histogram is more or less the
equivalent of a Prometheus summary. 

However, whenever the desired quantiles need to change, the config must be adjusted and the agent restarted. 

Therefore, the client maps the more complex metrics to a set of related gauges. This allows us to achieve 
the same visualization without the need to adjust any of the agents config files. 

### Counter 

Upon the change of a counter a datagram is sent with the delta to the previous value reported. 

```
countAll:1|c
```

### Gauge 

Whenever a gauge changes, a datagram with the current absolute value of the gauge is reported.
```
adjustGauge:32.2787317766752|g
```

### Histogram 

A histogram is reported in a datagram built from related gauges. The individual records share the same name
and the buckets are encoded in an extra label `le`. 

```
myHistogram:0|g|#le:0.0
myHistogram:3|g|#le:10.0
myHistogram:6|g|#le:20.0
myHistogram:10|g|#le:30.0
myHistogram:13|g|#le:40.0
myHistogram:16|g|#le:50.0
myHistogram:19|g|#le:60.0
myHistogram:22|g|#le:70.0
myHistogram:24|g|#le:80.0
myHistogram:27|g|#le:90.0
myHistogram:29|g|#le:100.0
myHistogram:37|g|#le:Inf
```

### Summary 

A summary is also reported as a set of related gauges. Each quantile will be reported within its own gauge. 
The additional labels are `quantile` to address the quantile and `error` to display the configured error margin. 

```
mySummary:123|g|#quantile:0.1,error:0.03
mySummary:254|g|#quantile:0.5,error:0.03
mySummary:441|g|#quantile:0.9,error:0.03
```

### Sets 

Sets are also reported as sets of related gauges. An additional label is used to differentiate the distinct 
values in the set. The name of the label is as configured in the aspect used to capture the set.

```
mySet:6|g|#token:myKey-18
mySet:1|g|#token:myKey-19
mySet:2|g|#token:myKey-13
mySet:3|g|#token:myKey-14
mySet:1|g|#token:myKey-16
mySet:3|g|#token:myKey-10
mySet:2|g|#token:myKey-11
mySet:1|g|#token:myKey-12
```

## The ZIO Metrics StatsD example

```scala

val instrumentedSample = new InstrumentedSample() {}
```

For StatsD we do need to spin up our own server. Rather we need to provide a client that can send datagrams 
to a specified UDP destination. 

Again we need an effect that runs our instrumented code until the user presses any key:

```scala 
val execute =
  for {
   fiber <- instrumentedSample.program.fork
    _ <- putStrLn("Press Any Key") *> getStrLn.orDie 
    _ <- fiber.interrupt
  } yield ExitCode.success
```  

Now, we can override the `run` method of our ZIO `App` and simply provide a `StatsDListener`. 

```scala 
def run(args: List[String]): URIO[ZEnv, ExitCode] =
  execute.provideCustomLayer(statsdLayer).orDie
```

> It is a listener because it listens to changes in the ZIO Metrics internal state and reports them 
> to StatsD by sending out appropriate datagrams. 

## A simple StatsD / Datadog setup 

The following steps describe how to set up a ZIO Metrics application reporting to [Datadog](https://www.datadoghq.com/) using a free Datadog account 
with limited functionality. The local setup is Windows 10 with WSL and Docker installed running Ubuntu 18.04 within WSL. 

In principle the setup is as follows:

1. The ZIO application sends datagrams to `localhost:8125` via UDP, so we need a component picking up those datagrams. 
1. Run a Datadog Collector within a docker image exposing a Unix socket for datagrams.
1. Run `socat` to listen on the UDP socket 8125 and forward incoming traffic to the Unix socket. 
1. Configure a dashboard in Datadog to visualize the metrics.

### Get and run the docker based Datadog collector 

Upon registration with datadoghq.com you will get an API key which is required to configure the agents collecting data. If you are planning 
to experiment with different agents, take a note of your API key for further reference. In the steps below the API key will be referred to 
as `$APIKEY`

For our example we have chosen to use the docker based collector and use a unix socket to report our datagrams to that agent. 

You can start the agent from the command line with 

```
docker run --name dd-agent -e DD_API_KEY=$APIKEY -e DD_SITE=datadoghq.eu \
  -e DD_DOGSTATSD_SOCKET=/var/run/datadog/datadog.sock \
  -v /var/run/datadog:/var/run/datadog \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  datadog/agent
```

As you can see, we require that the directory `/var/run/datadog` exists so that we can use it as a volume within the agent's docker container. The environment variable `DD_DOGSTATSD_SOCKET` tells the agent to use a unix socket to listen for datagrams. The socket file must reside within the mounted volume. 

This will start the datadog collector within docker and we have a unix socket to report our datagrams to. 

The next step is to create a UDP socket where our application can report its datagrams to. For our example we have chosen `socat` to forward 
UDP traffic to our unix socket:

```
sudo socat -s -u udp-recv:8125 unix-sendto:/var/run/datadog/datadog.sock
```

> You could add the `-v` parameter to the socat call to run socat in verbose mode. That would print
> all traffic being forwarded to the console as well. This is useful to determine whether 
> datagrams are actually sent. 

### Run the Datadog example

Now, the Datadog example can be started from within the project checkout directory with 

```
sbt sampleApp/run
```

### Visualize the metrics

1. Log in to your datadog account 
1. From the menu on left hand side select `Dashboards/New Dashboard'
1. In the upper right corner, click on the dashboard settings and select 'Import Dashboard JSON' 
1. From the filesystem, select `$DIR/examples/statsd/ZIOMetrics.json`
1. Confirm to override the dashboard configuration 
1. Save the just imported dashboard 
1. From the dashboard list select _ZIO metrics_
1. The Datadog dashboard is displayed

### Datadog dashboard 

![A simple Datadog Dashboard](../img/Datadog.png)

---

## Introduction to ZIO Parser

Library for constructing parsers and pretty printers based on invertible syntax descriptions

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-parser/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-parser_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-parser_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-parser_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-parser_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-parser-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-parser-docs_2.13) [![ZIO Parser](https://img.shields.io/github/stars/zio/zio-parser?style=social)](https://github.com/zio/zio-parser)

## Introduction

[![Zymposium - ZIO Parser](https://i.ytimg.com/vi/DEPpL9LBiyA/maxresdefault.jpg)](https://www.youtube.com/watch?v=DEPpL9LBiyA)

## Installation

Start by adding `zio-parser` as a dependency to your project:
  
```scala
libraryDependencies += "dev.zio" %% "zio-parser" % "0.1.9"
```

[//]: # (TODO: Add example section)
[//]: # (## Example)

---

## ZIO Prelude Abstraction Diagrams

## Absorption

```mermaid
classDiagram
  Absorption~A~ <|-- DistributiveAbsorption~A~
  Absorption~A~ <|-- Noncontradiction~A~
  Absorption~A~ <|-- ExcludedMiddle~A~
  ExcludedMiddle~A~ <|-- Involution~A~
  Noncontradiction~A~ <|-- Involution~A~
  class Absorption~A~{
    () or(=> A, => A): A
    () and(=> A, => A): A
  }
  class DistributiveAbsorption~A~{
    Boolean
    Set[A]
  }
  class Noncontradiction~A~{
    () complement(=> A): A
    () bottom: A
  }
  class ExcludedMiddle~A~{
    () complement(=> A): A
    () top: A
  }
  class Involution~A~{
    Boolean
  }
```

## Associative

```mermaid
classDiagram
  Associative~A~ <|-- Commutative~A~
  Associative~A~ <|-- Idempotent~A~
  Associative~A~ <|-- Identity~A~
  Identity~A~ <|-- PartialInverse~A~
  PartialInverse~A~ <|-- Inverse~A~
  class Associative~A~{
    Either[E, A: Associative]
    F[A: Associative]: Derive[_, Associative]
    First[A]
    Last[A]
    NonEmptyChunk[A]
    NonEmptyList[A]
    These[A: Associative, B: Associative]
    ❨T1: Associative, ..., T22: Associative❩
    Validation[E, A: Associative]
    ZNonEmptySet[A, B: Associative]

    () combine(=> A, => A): A
  }
  class Commutative~A~{
    And
    F[A: Commutative]: Derive[_, Commutative]
    Either[E: Commutative, A: Commutative]
    Or
    Map[K, V: Commutative]
    Max[A: Ord]
    Max[Boolean]
    Max[Byte/Char/Double/Float/Int/Long/Short]
    Min[A: Ord]
    Min[Boolean]
    Min[Byte/Char/Double/Float/Int/Long/Short]
    NonEmptySet[A]
    Option[A: Commutative]
    Prod[Boolean]
    Prod[Byte/Char/Double/Float/Int/Long/Short]
    Set[A]
    Sum[Boolean]
    Sum[Byte/Char/Double/Float/Int/Long/Short]
    These[A: Commutative, B: Commutative]
    ❨T1: Commutative, ..., T22: Commutative❩
    Validation[E, A: Commutative]
    ZSet[A, B: Commutative]
    ZNonEmptySet[A, B: Commutative]

    () commute: Commutative[A]
  }
  class Idempotent~A~{
    And
    F[A: Idempotent]: Derive[_, Idempotent]
    Or
    Map[K, V: Idempotent]
    Max[Boolean]
    Max[Byte/Char/Double/Float/Int/Long/Short]
    Min[Boolean]
    Min[Byte/Char/Double/Float/Int/Long/Short]
    NonEmptySet[A]
    Option[A: Idempotent]
    Prod[Boolean]
    Set[A]
    Sum[Boolean]
    These[A: Idempotent, B: Idempotent]
    ❨T1: Idempotent, ..., T22: Idempotent❩
    Validation[E, A: Idempotent]
    ZSet[A, B: Idempotent]
    ZNonEmptySet[A, B: Idempotent]

    () combineIdempotent(=> A, => A)(Equal[A]): A
    () idempotent(Equal[A]): Idempotent[A]
  }
  class Identity~A~{
    F[A: Identity]: Derive[_, Identity]
    Chunk[A]
    Either[E, A: Identity]
    List[A]
    Map[K, V: Associative]
    Max[Boolean]
    Max[Byte/Char/Double/Float/Int/Long/Short]
    Min[Boolean]
    Min[Byte/Char/Double/Float/Int/Long/Short]
    Option[A: Associative]
    String
    ❨T1: Identity, ..., T22: Identity❩
    Validation[E, A: Identity]
    Vector[A]
    ZSet[A, B: Associative]

    () identity: A
  }
  class PartialInverse~A~{
    F[A: PartialInverse]: Derive[_, PartialInverse]
    Prod[Byte/Char/Double/Float/Int/Long/Short]
    ❨T1: PartialInverse, ..., T22: PartialInverse❩
    () inverseOption(=> A, => A): Option[A]
  }
  class Inverse~A~{
    And
    F[A: Inverse]: Derive[_, Inverse]
    Or
    Prod[Boolean]
    Set[A]
    Sum[Boolean]
    Sum[Byte/Char/Double/Float/Int/Long/Short]
    ❨T1: Inverse, ..., T22: Inverse❩

    () inverse(=> A, => A): A
  }
```

## AssociativeBoth

```mermaid
classDiagram
  AssociativeBoth~F<_>~ <|-- CommutativeBoth~F<_>~
  AssociativeBoth~F<_>~ <|-- IdentityBoth~F<_>~
  class AssociativeBoth~F<_>~{
    Exit[E, +*]
    Fiber[E, +*]
    NonEmptyChunk[+*]
    ZSink[R, E, I, L, +*]
    ZStream[R, E, +*]

    () both[A,B](=> F[A], => F[B]): F[(A,B)]
  }
  class CommutativeBoth~F<_>~{
    Chunk[+*]
    Exit[E, +*]
    Id[+*]
    List[+*]
    NonEmptyChunk[+*]
    Option[+*]
    AndF[Schedule[R, E, +*]]
    OrF[Schedule[R, E, +*]]
    Vector[+*]
    ZIO[R, E, +*]
    Failure[ZIO[R, E, +*]]
    ZLayer[R, E, +*]
    ZManaged[R, E, +*]
    Failure[ZManaged[R, E, +*]]
    ZSink[R, E, I, L, +*]
    ZStream[R, E, +*]
  }
  class IdentityBoth~F<_>~{
    Chunk[+*]
    Config[+*]
    Either[L, +*]
    Failure[Either[+*, R]]
    Option[+*]
    Optional[+*]
    Future[+*]
    Id[+*]
    List[+*]
    STM[R, E, +*]
    Try[+*]
    Vector[+*]

    () any: F[Any]
  }
```

## AssociativeCompose

```mermaid
classDiagram
  AssociativeCompose~=>:[-_, +_]~ <|-- IdentityCompose~=>:[-_, +_]~
  AssociativeCompose~=>:[-_, +_]~ <|-- BothCompose~=>:[-_, +_]~
  AssociativeCompose~=>:[-_, +_]~ <|-- EitherCompose~=>:[-_, +_]~
  BothCompose <|-- ApplicationCompose~=>:[-_, +_]~
  class AssociativeCompose~=>:[-_, +_]~{
    () compose[A, B, C](B =>: C, A =>: B): A =>: C
  }
  class IdentityCompose~=>:[-_, +_]~{
    Function[-*, +*]
    URIO[-*, +*]

    () identity[A]: A =>: A
  }
  class BothCompose~=>:[-_, +_]~{
    URIO[-*, +*] [ :*: = Tuple2 ]
  
    (type) :*:[+_, +_]
    () fromFirst[A]: (A :*: Any) =>: A
    () fromSecond[B]: (Any :*: B) =>: B
    () toBoth[A, B, C](A =>: B)(A =>: C): A =>: (B :*: C)
  }
  class ApplicationCompose~=>:[-_, +_]~{
    Function[-*, +*] [ :*: = Tuple2 ; -->: = Function ]
    
    (type) -->:[-_, +_]
    () application[A, B]: ((A -->: B) :*: A) =>: B
    () curry[A, B, C]((A :*: B) =>: C): A =>: (B -->: C)
    () uncurry[A, B, C](A =>: (B -->: C)): (A :*: B) =>: C
  }
  class EitherCompose~=>:[-_, +_]~{
    Function[-*, +*] [ :+: = Either ]
    URIO[-*, +*] [ :+: = Either ]
    
    (type) :+:[+_, +_]
    () toLeft[A]: A =>: (A :+: Nothing)
    () toRight[B]: B =>: (Nothing :+: B)
    () fromEither[A, B, C](=> A =>: C)(=> B =>: C): (A :+: B) =>: C
  }
```

## AssociativeEither

```mermaid
classDiagram
  AssociativeEither~F<_>~ <|-- CommutativeEither~F<_>~
  AssociativeEither~F<_>~ <|-- IdentityEither~F<_>~
  class AssociativeEither~F<_>~{
    Either[L, +*]
    Exit[E, +*]
    Fiber[E, +*]
    Schedule[R, E, +*]
    Try[+*]
    ZLayer[R, E, +*]
    ZManaged[R, E, +*]
    
    () either[A, B](=> F[A], => F[B]): F[Either[A, B]]
  }
  class CommutativeEither~F<_>~{
    Future[+*]
    ZIO[R, E, +*] 
    ZSink[R, E, I, L, +*]
    ZStream[R, E, +*]
    Equal[-*]
  }
  class IdentityEither~F<_>~{
    Option[+*]
    Equal[-*]
    Hash[-*]
    Ord[-*]

    () none: F[Nothing]
  }
```

## AssociativeFlatten

```mermaid
classDiagram
  AssociativeFlatten~F<+_>~ <|-- IdentityFlatten~F<+_>~
  class AssociativeFlatten~F<+_>~{
    Map[K, +*]
    
    () flatten[A](F[F[A]]): F[A]
  }
  class IdentityFlatten~F<+_>~{
    Cause[+*]
    Chunk[+*]
    Either[L, +*]
    Exit[E, +*]
    Future[+*]
    Id[+*]
    List[+*]
    NonEmptyChunk[+*]
    Option[+*]
    Try[+*]
    Vector[+*]
    ZIO[R, E, +*]
    ZManaged[R, E, +*]
    ZStream[R, E, +*]

    () any: F[Any]
  }
```

# DistributiveProd

```mermaid
classDiagram
  DistributiveProd~A~ <|-- Annihilation~A~
  class DistributiveProd~A~{
    Cause[A]
    ParSeq[Unit, A]
    
    () Sum: Associative[Sum[A]]
    () Prod: Associative[Prod[A]]
    () sum(=> A, => A): A
    () prod(=> A, => A): A
  }
  class Annihilation~A~{
    Byte/Char/Double/Float/Int/Long/Short
  }
```

## Equal

```mermaid
classDiagram
  Equal~-A~ <|-- Hash~-A~
  Equal~-A~ <|-- PartialOrd~-A~
  PartialOrd~-A~ <|-- Ord~-A~
  class Equal~A~{
    Chunk[A: Equal]
    Either[A: Equal, B: Equal]
    Exit[E: Equal, A: Equal]
    F[A: Equal]: DeriveEqual[_]
    List[A: Equal]
    NonEmptyChunk[A: Equal]
    NonEmptyList[A: Equal]
    Option[A: Equal]
    ParSeq[A: Equal]
    These[A: Equal, B: Equal]
    Try[A: Equal]
    ❨T1: Equal, ..., T22: Equal❩
    Validation[E, A: Equal]
    Vector[A: Equal]
    ZNonEmptySet[A, B: Equal]
    ZSet[A, B: Equal]

    () both[B](=> Equal[B]): Equal[(A, B)]
    () bothWith[B, C](=> Equal[B])(C => (A, B)): Equal[C]
    () contramap[B](B => A): Equal[B]
    () either[B](=> Equal[B]): Equal[Either[A, B]]
    () eitherWith[B, C](=> Equal[B])(C => Either[A, B]): Equal[C]
    () equal(A, A): Boolean
    () notEqual(A, A): Boolean
    () toScala[A1 <: A]: scala.math.Equiv[A1]
  }
  class Hash~-A~ {
    Boolean
    Byte
    Cause[A]
    Char
    Chunk[A: Hash]
    Class[_]
    Double
    Either[A: Hash, B: Hash]
    F[A: Hash]: Derive[_, Hash]
    Fiber.Id
    Float
    Int
    List[A: Hash]
    Long
    Map[A, B: Hash]
    NonEmptyChunk[A: Hash]
    NonEmptyList[A: Hash]
    NonEmptySet[A]
    Nothing
    Option[A: Hash]
    Ordering
    ParMap[A, B: Hash]
    ParSeq[A: Hash]
    ParSet[A]
    PartialOrdering
    Set[A]
    Short
    String
    These[A: Hash, B: Hash]
    ❨T1: Hash, ..., T22: Hash❩
    Unit
    Validation[E: Hash, A: Hash]
    Vector[A: Hash]
    ZNonEmptySet[A, B: Hash]
    ZSet[A, B: Hash]
    ZTrace

    () both[B](Hash[B]): Hash[(A, B)]
    () bothWith[B, C](Hash[B])(C => (A, B)): Hash[C]
    () contramap[B](B => A): Hash[B]
    () either[B](Hash[B]): Hash[Either[A, B]]
    () eitherWith[B, C](Hash[B])(C => Either[A, B]): Hash[C]
    () hash(A): Int
  }
  class PartialOrd~-A~ {
    Chunk[A: PartialOrd]
    Either[A: PartialOrd, B: PartialOrd]
    F[A: PartialOrd]: Derive[_, PartialOrd]
    List[A: PartialOrd]
    Map[A, B: Equal]
    NonEmptyChunk[A: PartialOrd]
    NonEmptyList[A: PartialOrd]
    NonEmptySet[A]
    Option[A: PartialOrd]
    ParMap[A, B: Equal]
    ParSeq[A: PartialOrd]
    ParSet[A]
    PartialOrdering
    Set[A]
    ❨T1: PartialOrd, ..., T22: PartialOrd❩
    Vector[A: PartialOrd]
    ZNonEmptySet[A, B: PartialOrd]
    ZSet[A, B: PartialOrd]

    () both[B](=> PartialOrd[B]): PartialOrd[(A, B)]
    () bothWith[B, C](=> PartialOrd[B])(C => (A, B)): PartialOrd[C]
    () compare(A, A): PartialOrdering
    () contramap[B](B => A): PartialOrd[B]
    () either[B](=> PartialOrd[B]): PartialOrd[Either[A, B]]
    () eitherWith[B, C](=> PartialOrd[B])(C => Either[A, B]): PartialOrd[C]
    () mapPartialOrdering(PartialOrdering => PartialOrdering): PartialOrd[A]

  }
  class Ord~-A~ {
    Boolean
    Byte
    Char
    Chunk[A: Ord]
    Double
    Either[A: Ord, B: Ord]
    F[A: Ord]: Derive[_, Ord]
    Fiber.Id
    Float
    Int
    List[A: Ord]
    Long
    NonEmptyChunk[A: Ord]
    NonEmptyList[A: Ord]
    Nothing
    Option[A: Ord]
    ParSeq[A: Ord]
    Ordering
    Short
    String
    ❨T1: Ord, ..., T22: Ord❩
    Unit
    Vector[A: Ord]

    () both[B](=> Ord[B]): Ord[(A, B)]
    () bothWith[B, C](=> Ord[B])(C => (A, B)): Ord[C]
    () compare(A, A): Ordering
    () contramap[B](B => A): Ord[B]
    () either[B](=> Ord[B]): Ord[Either[A, B]]
    () eitherWith[B, C](=> Ord[B])(C => Either[A, B]): Ord[C]
    () mapOrdering(Ordering => Ordering): Ord[A]
    () reverse: Ord[A]
    () toScala[A1 <: A]: scala.math.Ordering[A1]
  }
```

---

## Associative

`Associative[A]` describes a way of combining two values of type `A` that is associative.

```scala
trait Associative[A] {
  def combine(left: => A, right: => A): A
}
```

If we import `zio.prelude._` we can use the `<>` operator to combine any two values of type `A` that have an `Associative` instance defined for them.

The `combine` operator must be associative, meaning that if we combine `a` with `b` and then combine the result with `c` we must get the same value as if we combine `b` with `c` and then combine `a` with the result.

```scala
(a <> b) <> c === a <> (b <> c)
```

The `Associative` abstraction allows us to combine values of a data type to build a new value of that data type with richer structure.

A variety of data types can be combined in an associative way:

```scala

val string: String =
  "Hello, " <> "world!"
// string: String = "Hello, world!"

val chunk: Chunk[Int] =
  Chunk(1, 2, 3) <> Chunk(4, 5, 6)
// chunk: Chunk[Int] = IndexedSeq(1, 2, 3, 4, 5, 6)
```

The `Associative` abstraction provides several advantages over using existing operators like the `++` operator on `Chunk` directly.

First, `Associative` allows us to combine more complex data types as long as the data types they are composed of can be combined in an associative way.

```scala
case class Topic(value: String)

case class Votes(value: Int)

object Votes {
  implicit val VotesAssociative: Associative[Votes] =
    new Associative[Votes] {
      def combine(left: => Votes, right: => Votes): Votes =
        Votes(left.value + right.value)
    }
}

case class VoteMap(map: Map[Topic, Votes])

object VoteMap {
  def combine(left: VoteMap, right: VoteMap): VoteMap =
    VoteMap(left.map <> right.map)
}
```

If we didn't have ZIO Prelude we would have to implement the operator to combine two `VoteMap` values ourselves. This would require some relatively low level logic that would look like this.

```scala
def combine(left: VoteMap, right: VoteMap): VoteMap =
  VoteMap(right.map.foldLeft(left.map) { case (map, (k, v)) =>
    map + (k -> map.get(k).fold(v)(v1 => Votes(v.value + v1.value)))
  })
```

This code isn't the worst. It uses operators like `foldLeft` to do this combining of the two maps in a relatively high level way.

But we're still having to implement our own collection operators, taking our attention away from implementing our business logic. And it would be hard to implement this without at least having to take a minute and to check our logic, especially if we are less familiar with immutable collection operators.

ZIO Prelude lets us avoid all of this because it knows how to combine two maps as long as there is a way to combine the map values. And it was quite simple to describe how we combine the map values, we just add them.

Second, the `Associative` abstraction lets us generalize over different ways of combining values if we want to.

For example, we could define an operator for reducing any `NonEmptyChunk` to a summary value like this.

```scala

def reduce[A: Associative](as: NonEmptyChunk[A]): A =
  as.reduce(_ <> _)
```

We can then reduce any `NonEmptyChunk` to a summary value as long as there is a way of combining the elements of the `NonEmptyChunk`, whether those elements are strings or vote maps.

So far we have not described some very basic ways of combining that are associative, such as integer addition.

The reason for this is that some data types support more than one way of combining them that is associative. For example, both integer addition and multiplication are associative.

This creates a potential issue when using the type class pattern because the Scala compiler looks up implicit values based on their type, so if there are two different implicit values of a given type the Scala compiler does not know which one to use.

For example, here is what happens if we try to define `Associative` instances for `Int` for both addition and multiplication.

```scala
implicit val IntSumAssociative: Associative[Int] =
  new Associative[Int] {
    def combine(left: => Int, right: => Int): Int =
      left + right
  }

implicit val IntProdAssociative: Associative[Int] =
  new Associative[Int] {
    def combine(left: => Int, right: => Int): Int =
      left * right
  }

2 <> 3
// error: ambiguous implicit values:
//  both value IntSumAssociative in object MdocApp of type zio.prelude.Associative[Int]
//  and value IntProdAssociative in object MdocApp of type zio.prelude.Associative[Int]
//  match expected type zio.prelude.Associative[Int]
// 2 <> 3
// ^^^^^^
```

This makes sense because there are indeed two different instances of `Associative[Int]` and there is no basis for choosing between them.

We could just define one of these instances as "primary" like some other functional programming libraries do and relegate the other to second class status but that would be arbitrary and reflect a lack of compositionality.

Instead we solve the problem in a way that fits with Scala's implicit resolution mechanism. If the Scala compiler looks up implicit values based on their types, then if we want two values we need two types.

We can easily do this with ZIO Prelude's new type functionality.

New types allow us to define new types that "wrap" existing types in a way that has no overhead at runtime. We can also define these new types so that the Scala compiler actually knows that the new type is a subtype of the underlying type.

Using this technique, we can define new types `Sum` and `Prod` that can be combined using addition and multiplication, respectively. ZIO Prelude provides these and similar new types such as `And` and `Or` for logical conjunction and disjunction in the `zio.prelude.newtypes` package.

We can wrap any existing type in a new type using the `apply` or `wrap` operators on the new type object.

```scala

val sumInt: Sum[Int] =
  Sum(1)
// sumInt: Sum[Int] = 1

val prodInt: Prod[Int] =
  Prod.wrap(2)
// prodInt: Prod[Int] = 2
```

We can unwrap any new type to get back the original type using the `unwrap` operator on the new type object.

```scala
val int: Int =
  Sum.unwrap(sumInt)
// int: Int = 1
```

However, we will typically not need to do that for new types like `Sum` and `Prod` because they are subtypes of `Int`.

```scala
val int: Int =
  prodInt
// int: Int = 2
```

Let's use these types to solve our problem from above.

```scala
val sum: Int =
  Sum(2) <> Sum(3)
// sum: Int = 5

val product: Int =
  Prod(2) <> Prod(3)
// product: Int = 6
```

These variants don't just work for `Int`, they work for any numeric data type in the Scala standard library.

```scala
val sum: Double =
  Sum(2.0) <> Sum(3.0)
// sum: Double = 5.0

val product: Double =
  Prod(2.0) <> Prod(3.0)
// product: Double = 6.0
```

Note that the associativity of addition and multiplication for `Double` values is subject to floating point rounding errors. ZIO Prelude assumes that if we are working with "lossy" data types like this we are aware of these issues and provides these instances for us, unlike some other functional programming libraries.

ZIO Prelude provides several other new types that you can use to define how you want to combine various data types.

The `And` and `Or` new types mentioned above let us define `Boolean` values that can be combined using logical conjunction and disjunction.

```scala
val and: Boolean =
  And(true) <> And(false)
// and: Boolean = false

val or: Boolean =
  Or(true) <> Or(false)
// or: Boolean = true
```

Another way of combining two values that is associative is taking the first or the last of two values.

```scala
val first: String =
  First("Hello") <> First("World")
// first: String = "Hello"

val last: String =
  Last("Hello") <> Last("World")
// last: String = "World"
```

The minimum and maximum of two values for which an ordering is defined also constitutes an associative `combine` operator.

```scala
val min: Int =
  Min(1) <> Min(2)
// min: Int = 1

val max: Int =
  Max(1) <> Max(2)
// max: Int = 2
```

These new types are particularly useful when dealing with more complex data types to specify how we want to combine part of them.

For example, let's go to our example with the `VoteMap` but say that now we are not going to define the additional type for `Topic` and `Votes`.

```scala
case class VoteMap(map: Map[String, Int])
```

We need to tell ZIO Prelude how we want to combine `Int` values. In this case we want to use the sum.

We do this by using the `wrapAll` operator. Just like `wrap` wraps a single value in a new type, `wrapAll` wraps a whole collection of values in a new type.

It does this without traversing the collection because ZIO Prelude knows internally that the new type and the underlying type are the same. This way we avoid any runtime overhead for using these abstractions.

```scala
object VoteMap {
  implicit val VoteMapAssociative: Associative[VoteMap] =
    new Associative[VoteMap] {
      def combine(left: => VoteMap, right: => VoteMap): VoteMap =
        VoteMap(Sum.wrapAll(left.map) <> Sum.wrapAll(right.map))
    }
}
```

You can see the documentation on new types for additional information about new types in general. But the material here should give you what you need to combine values of even complex data types and define your own instances of the `Associative` abstraction for your own data types.

If you are interested in combining values of collections it is also worth checking out the `ForEach` functional abstraction, which describes ways to iterate over collection types. The `ForEach` abstraction comes with a variety of built in operators for combining collection types using an associative operator.

For example, using `ForEach`, `Associative`, and new types we could count the total number of words in a collection of lines like this:

```scala

def wordCount(lines: NonEmptyChunk[String]): Int =
  lines.reduceMap(line => Sum(line.split(" ").length))
```

This maps each element in a collection to a new data type for which an `Associative` instance is defined and then reduces all of those values to a single summary value with the `combine` operator.

We could instead count the number of occurrences of each word like this:

```scala
def wordCount(lines: NonEmptyChunk[String]): Map[String, Int] =
  lines.reduceMap { line =>
    Sum.wrapAll(line.split(" ").groupBy(identity).view.mapValues(_.length).toMap)
  }
```

This version is exactly the same except we mapped the elements to a different value. ZIO Prelude automatically applied the appropriate `combine` operator.

This is a great example of how these abstractions can make it easy to combine values of different data types, cutting the boilerplate out of our code and reducing opportunities for bugs.

---

## Commutative

`Commutative[A]` describes a data type that has a `combine` operator that is commutative and associative.

Its signature is:

```scala
trait Associative[A] {
  def combine(left: => A, right: => A): A
}

trait Commutative[A] extends Associative[A]
```

The `Commutative` abstraction does not define any additional operators beyond the `combine` operator defined by `Associative`. However, it does provide additional structure to the `combine` operator.

Specifically, it says that the `combine` operator is commutative in addition to being associative. This means that not only does the order of operations not matter, but the order in which we combine the values does not matter.

```scala
a <> b === b <> a
```

Only some ways of combining that are associative are commutative.

For example, integer addition is both associative and commutative. `2 + 3` is the same as `3 + 2`.

On the other hand, string concatenation is associative but not commutative. `"a" + "b"` is not the same as `"b" + "a"`.

We can use the `Commutative` abstraction to specify at the type level whether one of our data types can be combined in a way that is associative and commutative or only associative. We can also test for it using the laws testing functionality provided by ZIO Prelude.

For example, here is how we could define and test a `Commutative` instance for a version of the `RunningAverage` data type that we discussed in the introduction to functional abstractions:

```scala

case class RunningAverage(sum: Long, count: Int)

object RunningAverage {
  implicit val RunningAverageCommutative: Commutative[RunningAverage] =
    new Commutative[RunningAverage] {
      def combine(left: => RunningAverage, right: => RunningAverage): RunningAverage =
        RunningAverage(left.sum + right.sum, left.count + right.count)
    }
  implicit val RunningAverageEqual: Equal[RunningAverage] =
    Equal.default
}

object RunningAverageSpec extends ZIOSpecDefault {

  def spec = suite("RunningAverageSpec") {
    test("commutative") {
      val runningAverageGen = Gen.long.zipWith(Gen.int)(RunningAverage(_, _))
      checkAllLaws(CommutativeLaws)(runningAverageGen)
    }
  }
}
```

The main advantage of having a `Commutative` instance is that we don't have to worry about the order in which we combine values because we will get the same result no matter what.

To take a simple example, consider combining the values in a set.

```scala
val fruits: Set[String] =
  Set("apple", "orange", "banana")
// fruits: Set[String] = Set("apple", "orange", "banana")

val ints: Set[Int] =
  Set(1, 2, 3)
// ints: Set[Int] = Set(1, 2, 3)

val notCommutative: String =
  fruits.foldLeft("")(_ + _)
// notCommutative: String = "appleorangebanana"

val commutative: Int =
  ints.foldLeft(0)(_ + _)
// commutative: Int = 6
```

The order of values in a set is arbitrary. When we iterate over the values of a set we must do it in some order, and that order may be the same for a particular implementation of a set, but that is not something we can rely on.

As a result, the value of `notCommutative` is not well defined. It could be `"applebananaorange"`, `"orangebananaapple"`, `"bananaorangeapple"`, among others.

We have to be very careful in dealing with `notCommutative`. Either we must not use it at all or we must use it in very limited ways that will return the same result regardless of the order in which it was constructed (e.g. testing whether it contains the substring `"apple"`).

In contrast, the value of `commutative` is extremely well defined. It will always be `6` no matter what.

Regardless of the order in which we iterate over the elements in the set it will always be `6` because addition is a commutative operation. Even if we tried to get a different value we couldn't come up with an order of iterating over the set elements that gave a different result.

Thus, we can safely work with `commutative` and do anything we want with it because it will always have the same value.

This is a simple example but this property of commutativity can come up in much more complex domains such as concurrent or distributed programming. There, the order in which we receive values from other fibers or nodes is often arbitrary.

In that case having a way of combining values that is commutative can be very useful because it means we don't need coordination mechanisms to ensure that we combine results in a certain order.

For an example of this we can look to the `mergeAllPar` operator on `ZIO`.

```scala

object ZIO {
  def mergeAllPar[R, E, A, B](
    in: Iterable[ZIO[R, E, A]]
  )(zero: B)(f: (B, A) => B): ZIO[R, E, B] =
    ???
}
```

This runs all of the `ZIO` workflows in the `Iterable` in parallel and combines their results with the function `f`, starting from `zero`. Its implementation creates a `Ref` with the `zero` value and then each fiber updates the `Ref` with its result as it completes.

The order in which the fibers complete is not determinate so the function `f` should be associative and commutative, as the documentation for `mergeAllPar` helpfully informs us.

We do not want to force functional abstractions on users so in ZIO itself we state that the function must satisfy these properties and leave it at that. But with ZIO Prelude we can define a version of this operator that expresses at the type level that the `combine` operator must be associative and commutative.

```scala
def mergeAllParCommutative[R, E, A, B: Commutative](
  in: Iterable[ZIO[R, E, A]]
)(zero: B)(f: A => B): ZIO[R, E, B] =
  ZIO.mergeAllPar(in)(zero)((b, a) => b <> f(a))
```

Now we run all the `ZIO` workflows in parallel, mapping their results to a type for which a commutative `combine` operator is defined. This will ensure that our way of combining values really is commutative.

With the `Commutative` abstraction we can describe the properties of our data types precisely, helping us understand what guarantees they provide and what our responsibilities are for working with them. This idea of commutative operators is also a helpful one to keep in mind when we are designing our own data types as we saw in the example above.

---

## Debug

`Debug[A]` describes the ability to render a value of type `A` to a human readable format for debugging purposes.

Its signature is:

```scala
trait Debug[-A] {
  def debug(a: A): Debug.Repr
}

object Debug {

  sealed trait Renderer

  sealed trait Repr {
    def render: String
    def render(renderer: Renderer): String
  }
}
```

`Repr` here is a data structure that captures the information needed to render a data type in a structured format so that it can be displayed in different ways. A `Renderer` is a data type that knows how to render a `Repr` to a particular string representation, for example using short names for readability or fully qualified names for using the rendered output as valid code.

If we import `zio.prelude._` then we can use the `debug` operator on any data type with a `Debug` instance defined for it. We can also just use the `render` operator on any data type with a `Debug` instance defined for it to render it to a simple string representation.

The `Debug` abstraction is the functional equivalent of the `toString` operator but it has several advantages over just using `toString`.

First, as with other functional abstractions we can define when it makes sense to render a data type at all.

We can call `toString` on anything including classes whose string representation only points to their memory address. Fortunately this is more often the cause of annoyance than real bugs but it is certainly nice to know when there isn't much point in trying to render something.

Related to this, we can define our own way of rendering for data types that are not under our control. For example, if we are debugging code involving arrays we can frequently be frustrated when the string representation of the array consists of its memory location rather than the elements actually in the array.

With ZIO Prelude, we can easily define a `Debug` instance for `Array`. We can even handle nested arrays.

The final advantage that the `Debug` abstraction has over the Scala standard library and older functional programming libraries is that it captures rendering information in a structured data format instead of just a `String`.

When we render a data type we may want to do it in various ways. We may want to do it using short names for a readable description in the context of a discussion like this one, rendering it as something like `Validation[String, Int]`.

In another context like being able to copy and paste our rendering into a REPL, IDE, or worksheet and having it compile we may want to use fully qualified names, such as `zio.prelude.Validation[scala.String, scala.Int]`. Or we may decide that including the fully qualified name for standard library types is a little too much, and want to render this as `zio.prelude.Validation[String, Int]`.

All of these are valid choices, but just rendering values as a `String` forces us to choose one. A `String` does not have enough structure to preserve all the information we need for rendering different ways so we are forced to choose one arbitrarily.

This also allows us to define meaningful laws for the `Debug` abstraction. In other functional programming libraries interfaces that provide similar functionality are essentially lawless since there is nothing describing what this string representation should look like.

In contrast, in ZIO Prelude `Debug` follows a well defined law that the Scala rendering of any data type should itself be valid Scala code.

## Structured Rendering

The `Repr` data type preserves all the information we need to render a data type to a `String` in various ways. There is nothing magic about this data type, it is just an algebraic data type with a variety of cases representing the different possible pieces of information we could need for rendering.

If you just want to render data types from ZIO or the Scala standard library that have meaningful string representations then you can just use the `debug` operator but if you want to define `Debug` instances for your own data type it is helpful to understand usage of `Repr`.

Most of the cases of `Repr` are "simple" cases that just describe existing primitive data types.

```scala
sealed trait Repr

object Repr {
  case class Int(value: scala.Int)           extends Repr
  case class Double(value: scala.Double)     extends Repr
  case class Float(value: scala.Long)        extends Repr
  case class Long(value: scala.Long)         extends Repr
  case class Byte(value: scala.Byte)         extends Repr
  case class Char(value: scala.Char)         extends Repr
  case class Boolean(value: scala.Boolean)   extends Repr
  case class Short(value: scala.Short)       extends Repr
  case class String(value: java.lang.String) extends Repr
}
```

As you can see these cases are indeed quite straightforward other than the minor complexity of avoiding name collisions with the underlying data types.

With these representations we can describe the structure of primitive values in a way that supports structured rendering. For example, the `Repr` of `42` would be `Repr.Int(42)` and we could match on it to render it in various ways such as `42` or `Int: 42`.

The next case, `Object`, lets us define our own primitive types.

```scala
object Repr {
  case class Object(namespace: List[java.lang.String], name: java.lang.String) extends Repr
}
```

As its name implies, this lets us define representations for data types like objects that aren't existing primitive types but also do not depend on other data types for their rendering. The information needed to render an object is just the name of the object and its namespace.

For example, the `Repr` of `None` would be `Repr.Object(List("scala"), "None")`. Note how the namespace gives us the information we need to render the `Repr` as valid Scala code and gives us the ability to decide at the time of rendering whether we want to include some or all of this or just the name itself.

The final cases of `Repr` let us build more complex data types such as sum types, product types, and collections from simpler ones.

```scala

object Repr {
  case class VConstructor(
    namespace: List[java.lang.String],
    name: java.lang.String,
    reprs: List[Repr]
  ) extends Repr
  case class Constructor(
    namespace: List[java.lang.String],
    name: java.lang.String,
    reprs: ListMap[java.lang.String, Repr]
  ) extends Repr
  case class KeyValue(
    key: Repr,
    value: Repr
  ) extends Repr
}
```

The `VConstructor` case mirrors the constructor arguments of a class or case class. Just like the `Object` constructor it contains a namespace and name but now it also has a list of `Repr` values describing how each of the constructor parameters can be rendered.

For example, the `Repr` of `Some(42)` would be `Repr.VConstructor(List("scala"), "Some", List(Repr.Int(42)))`. Again, this gives us everything we need to render the value in different ways.

The `Constructor` case is like `VConstructor` but mirrors a data type like a case class with named fields. The list of constructor arguments now included the name of each constructor argument along with its representation.

For example, the `Repr` of `Person("John", 42)` would be `Repr.Constructor(List.empty, "Person", Map("name" -> Repr.String("John"), "age" -> Repr.Int(42)))`. The names of the constructor arguments may or may not be shown depending on the renderer used.

The final case is `KeyValue`, which covers situations where the constructor arguments are key value pairs, like in a `Map`. We capture this separately so we can render them appropriately, for example using `->` to ensure that we can render the key value pairs as valid code.

Most of the time you won't have to use most of these constructors yourself but it is helpful to know what they are since you will often have to use one or two in defining your own `Debug` instances, typically the `Object`, `VConstructor`, and `Constructor` cases.

## Defining Debug Instances

ZIO Prelude comes with `Debug` instances defined for all data types with meaningful string representations in ZIO and the Scala standard library, as well as data types composed of those types.

To define a `Debug` instance for your own data type you can use the `make` operator which requires you to specify how to create a representation of your own data type. Typically you can do this just by using one of the `Repr` constructors such as `Constructor`, `Object`, or `VConstructor` and using the `debug` operator on any values that are inside your data type.

For example, here is how we could define a `Debug` instance for a `Person` data type:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val PersonDebug: Debug[Person] =
    Debug.make { case Person(name, age) =>
      Debug.Repr.Constructor(
        List.empty,
        "Person",
        "name" -> name.debug,
        "age"  -> age.debug
      )
    }
}
```

Notice how we used the `debug` operator on `name` and `age`. In this simple case we could have used `Repr.String(name)` and `Repr.Int(age)` but this would not have worked if the constructor values were more complex and would have forced us to spend more time working with `Repr` directly.

We can use the same strategy to define `Debug` instances for polymorphic data types. We just need to add a constraint that there is a `Debug` instance for the different types that we want to render.

For example, here is how we could define a `Debug` instance for a simplified version of the `Validation` data type from ZIO Prelude.

```scala

sealed trait Validation[+E, +A]

object Validation {

  case class Success[+A](a: A) extends Validation[Nothing, A]
  case class Failure[+E](es: NonEmptyChunk[E]) extends Validation[E, Nothing]

  implicit def ValidationDebug[E: Debug, A: Debug]: Debug[Validation[E, A]] =
    Debug.make {
      case Success(a)  => Debug.Repr.VConstructor(List("zio", "prelude"), "Validation.Success", List(a.debug))
      case Failure(es) => Debug.Repr.VConstructor(List("zio", "prelude"), "Validation.Failure", List(es.debug))
    }
}
```

Here we used context bounds of `Debug` for the `E` and `A` type parameters to require that `Debug` instances for them exist. This makes sense because there is no way we can create a meaningful string representation of a value if we can't create meaningful string representations of its components.

## Rendering

The `Repr` data type returned by the `debug` operator is just a representation of the data type in a way that supports meaningful string rendering. To actually get a string we still need to render it with its `render` operator.

The easiest way to do that is to call the `render` operator directly, which uses a default renderer that renders data in a simple way that is being read by humans but is not necessarily valid Scala code. If we want to, we can call `render` with an argument and provide a `Renderer` to render the representation in a different way.

ZIO Prelude comes with two other renderers.

The `Scala` renderer renders data as valid Scala code that you can copy and paste into an IDE, REPL, or worksheet. The `Full` renderer renders data with as much detail as possible, including information like field names where available, and can be helpful for debugging purposes.

A `Renderer` is just a function that takes a `Repr` and returns a `String`, so you can also define your own `Renderer` if you want to. ZIO Prelude's `Debug` abstraction preserves all the information so you can render your data types the way you want to.

---

## Equal

`Equal[A]` describes the ability to compare two values of type `A` for equality.

Its signature is:

```scala
trait Equal[-A] {
  def equal(left: A, right: A): Boolean
}
```

If we import `zio.prelude._` we can use the `===` and `!==` operators to compare any two values of type `A` for equality as long as there is an `Equal` instance defined for them.

This has several advantages over using the `==` and `!=` defined in the Scala standard library.

First, using `Equal` prevents bugs because it stops nonsensical code that attempts to compare unrelated types from compiling.

Using `==` we can compare any two values for equality, even if they are unrelated types. For example, this code has a bug.

```scala
final case class SequenceNumber(value: Int)

def isInitial(sequenceNumber: SequenceNumber): Boolean =
  sequenceNumber == 0
// warning: comparing values of types repl.MdocSession.MdocApp.SequenceNumber and Int using `==` will always yield false
//   sequenceNumber == 0
//   ^^^^^^^^^^^^^^^^^^^
```

We're trying to determine if this is the initial sequence number , but we're comparing the `sequenceNumber`, which is a `SequenceNumber`, with `0`, which is an `Int`. So this comparison will always be false even if the `sequenceNumber` is `SequenceNumber(0)` which is what we are trying to test for.

This snippet may generate a warning but it is still valid Scala code.

In contrast, ZIO Prelude's `Equal` abstraction catches the bug and prevents this code from even compiling.

```scala

final case class SequenceNumber(value: Int)

object SequenceNumber {
  implicit val SequenceNumberEqual: Equal[SequenceNumber] =
    Equal.default
}

def isInitial(sequenceNumber: SequenceNumber): Boolean =
  sequenceNumber === 0
// error: No implicit Equal defined for Any.
//   sequenceNumber === 0
//   ^^^^^^^^^^^^^^^^^^^^
```

To get the code to compile we have to fix the bug.

```scala

case class SequenceNumber(value: Int)

object SequenceNumber {

  val initial: SequenceNumber =
    SequenceNumber(0)

  implicit val SequenceNumberEqual: Equal[SequenceNumber] =
    Equal.default
}

def isFirst(sequenceNumber: SequenceNumber): Boolean =
  sequenceNumber === SequenceNumber.initial
```

A second advantage is catching bugs when we try to compare two values of a type that doesn't have a meaningful notion of equality. For example, it doesn't make sense to compare two Scala functions for equality, but we can still do it with `==`.

In contrast, comparing two functions with `===` will result in a compilation error, preventing us from making this mistake.

A third advantage of the `Equal` abstraction is that it lets us use our own notion of equality for data types that are not under our control. We may be working with data types from other libraries that do not implement a reasonable notion of equality, for example using reference equality instead of value equality.

If we are using `==` we can't do anything about that because `==` is always based on how the data type implements `equals`. But if we are using `===` from ZIO Prelude we can define our own `Equal` instance that implements a more sensible notion of equality.

## Defining Equal Instances

Equal instances are automatically available for data types from the Scala library and ZIO with a meaningful notion of equality, as well as data types made up of those types.

We can always check if an instance of a type class exists for a given data type by using the `apply` operator to tell the Scala compiler to find the appropriate instance for us. This will result in a compilation error if the instance cannot be found.

```scala
Equal[List[Either[String, Int]]]
// res3: Equal[List[Either[String, Int]]] = zio.prelude.Equal$$anonfun$make$2@320f2e42
```

```scala
Equal[String => Int]
```

Defining `Equal` instances for your own data types is generally quite easy.

If you have already defined a sensible `equals` method for your data type then you can just use the `default` operator to construct an `Equal` instance as we saw in the example above involving the `SequenceNumber`.

Since the `SequenceNumber` is a case class it already has a well defined `equals` method as long as each value in the case class also has a well defined `equals` method. Equality for `Int` values is definitely well behaved so there is nothing we have to do here other than use the `default` operator.

If we don't want to use the default definition of equality we can use the `make` operator to define a new `Equal` instance based on our own definition of equality.

This is particularly useful if we are defining an `Equal` instance for a parameterized type and want to compare the types it is parameterized on for equality using their own `Equal` instances.

For example, say we have a class class to represent a pair of values.

```scala
case class Pair[A, B](first: A, second: B)
```

The `equals` method on a case class delegates to the `equals` method on each of its fields and we don't know if `A` or `B` have sensible definitions of `equals` since they are generic. So we want to instead define our own `Equal` instance directly like this:

```scala
object Pair {
  implicit def PairEqual[A: Equal, B: Equal]: Equal[Pair[A, B]] =
    Equal.make { (left, right) =>
      left.first === right.first && left.second === right.second
    }
}
```

Notice how we had to require that `Equal` instances existed for `A` and `B`. We can't just compare any `Pair` for equality, we can only do it if each field of the pair can also be compared for equality.

Including `Equal` after the definition of the `A` and `B` type parameters here is called a _context_ bound_ and is another way of saying that an implicit instance of `Equal[A]` and `Equal[B]` must exist. It can make our code a little cleaner when we don't need to do anything with the instance other than require that it exists, in this case for the `===` operator.

If we can convert our data type to another data type for which an `Equal` instance is already defined we can make this even simpler using the `contramap` operator.

In the example above, we can convert any `Pair` into a tuple without losing any information. And ZIO Prelude already defines an `Equal` instance for any tuple of data types for which `Equal` instances are defined.

So we can rewrite our implementation of `PairEqual` like this:

```scala
object Pair {
  implicit def PairEqual[A: Equal, B: Equal]: Equal[Pair[A, B]] =
    Equal[(A, B)].contramap(pair => (pair.first, pair.second))
}
```

The `contramap` operator essentially says if we can compare two tuples for equality, and we can turn a pair into a tuple, then we can always compare two pairs for equality by turning them both into tuples and comparing the tuples for equality.

The `contramap` operator can be useful to create `Equal` instances from existing ones but the way we "work backwards" from the type we know how to compare for equality to the `Equal` instance we are trying to make can be confusing initially. So it is fine to just use the `make` operator as discussed above, which essentially does the same thing.

With the instances already defined for all types in ZIO and the Scala standard library with meaningful definitions of equality and the ability to define instances for your own data types with `make` and `contramap` it is easy to take advantage of the additional type safety that using `===` provides.

---

## Hash

`Hash[A]` describes the ability to hash a value of type `A`.

The signature is:

```scala
trait Equal[-A] {
  def equal(left: A, right: A): Boolean
}

trait Hash[-A] extends Equal[A] {
  def equal(left: A, right: A): Boolean
  def hash(a: A): Int
}
```

`Hash` builds on `Equal` in a different way than `Ord` by defining a way to `Hash` two values of type `A` in addition to a way to compare them for equality.

If we import `zio.prelude._` We can hash any data type that has a `Hash` instance defined for it using the `hash` operator or its symbolic alias `##`.

In addition to hashing being generally useful, hashing allows us to determine whether a value already exists in a collection of values without individually testing for equality with each value. This allows us to implement some operators that would not otherwise be practical if all we could do was compare values for equality pairwise.

The `Hash` abstraction provides similar functionality as the `hashCode` operator defined on any object in Scala with a few advantages.

First, it allows us to specify whether a data type has a meaningful definition of hash code.

For example, we can call `hashCode` on a Scala function but there is not a meaningful way of comparing Scala functions for equality so the definition of `hashCode` is just based on reference equality.

While this could potentially be useful in some very low level parts of our code this is generally a potential source of bugs if we are not specifically trying to use reference equality. ZIO Prelude just doesn't define a `Hash` instance for functions so this can never happen.

Second, we can use the `Hash` abstraction to define strategies for hashing data types that are not in our control. An existing data type may implement `hashCode` in a way that doesn't make sense or is not what we want for whatever reason (e.g. creates too high a rate of hash collisions, is vulnerable to attacks, etc...).

With just the `hashCode` defined in the Scala standard library we don't have any way of fixing this other than creating a new wrapper for the data type, which has potential performance implications and may require boilerplate in wrapping and unwrapping the new type or reimplementing existing methods. In contrast, with ZIO Prelude defining a different hashing strategy is as simple as implementing a new instance.

Third, the `Hash` abstraction guarantees that our definition of hashing is consistent with our definition of equality. That is, if two values are equal they should always have the same hash code.

This is another common source of errors whenever we implement our own definition of `equals` and `hashCode` and ZIO Prelude can prevent it by automatically checking that all instances of `Hash` define a consistent notion of equality and hashing.

For example, here is how we could test that ZIO Prelude's notions of hashing and equality for strings are consistent.

```scala

object HashSpec extends ZIOSpecDefault {

  def spec = suite("HashSpec") {
    test("StringHash") {
      val stringGen = Gen.string
      checkAllLaws(HashLaws)(stringGen)
    }
  }
}
```

This will automatically generate a large number of string values and test that the definition of hashing is consistent with equality for all of them, reporting any failures.

## Defining Hash Instances

ZIO Prelude automatically includes instances of `Hash` for all data types in ZIO and the Scala standard library that support a meaningful definition of hashing. This also includes types like collections, tuples, and sum types like `Option` and `Either`.

If we are defining our own data types we can use the `default` operator if we want to use the definition of `equals` and `hashCode` already defined on the type. This can be particularly useful if we are defining case classes that are made up of 

```scala
case class Person(name: String, age: Int)

object Person {
  implicit val PersonHash: Hash[Person] =
    Hash.default
}
```

If we are defining a `Hash` instance for our own data type we can use the `make` operator, which allows you to provide your own definition of hashing and equality.

However, hashing can involve some low level logic so for the `Hash` abstraction it can be particularly nice to use the `contramap` operator.

```scala
trait Hash[-A] {
  def contramap[B](f: B => A): Hash[B]
}
```

The `contramap` operator says if we know how to turn a `B` value into an `A` value and we know how to hash an `A` value then we can hash a `B` value simply by turning it into an `A` value and hashing that.

This turns out to be quite useful because we can convert almost any date type we define into some data type in ZIO or the Scala library that there is already a `Hash` instance defined for. The function `f` should be information preserving so if we have two `A` values that are not equal they should map to two `B` values that are not equal and if we have two `A` values that are equal they should map to two `B` values that are equal.

For example, here is how we can use `contramap` to easily define a `Hash` instance for a custom data type to keep track of different topics and the number of votes for each of them.

```scala

case class Topic(value: String)

case class Votes(value: Int)

object Votes {
  implicit val VotesHash: Hash[Votes] =
    Hash.default
}

case class VoteMap(map: Map[Topic, Votes])

object VoteMap {
  implicit val VoteMapHash: Hash[VoteMap] =
    Hash[Map[Topic, Votes]].contramap(_.map)
}
```

ZIO Prelude knows how to hash values of type `Map[A, B]` as long as there is a way of hashing the values in the map. And we know how to hash the values in the map because we defined a `Hash` instance for `Votes`.

So all we have to do is tell ZIO Prelude how to convert our `VoteMap` into a `Map`, which is quite easy because a `VoteMap` just wraps a map!

This strategy turns out to be quite general because there are `Hash` instances defined on so many data types such as collection types, product types like tuples, sum types like `Either` and `Option`, and primitive types. We can almost always convert our data type to some combination of those types.

By doing this we can easily define which data types it makes sense to hash, control the hashing strategies we use, and integrate hashing with the other functional abstractions in ZIO Prelude.

---

## Identity

`Identity[A]` describes a data type with an associative combining operation that also has an identity element.

Its signature is:

```scala
trait Associative[A] {
  def combine(left: => A, right: => A): A
}

trait Identity[A] extends Associative[A] {
  def identity: A
}
```

The identity element is a neutral value that does not have any structure itself and so combining it with any other value just leaves the structure of the other value unchanged. For instance, an empty `Chunk` is an identity element with respect to the `++` operator.

It is important to note here that while `Identity` describes a data type with an identity element, that identity element only has meaning in relation to the binary operator. Specifically, combining any value of type `A` with the identity element with the `combine` operator must return the value unchanged.

```scala
a <> identity === a
identity <> a === a
```

For example, zero is an identity element with respect to addition, but not with respect to multiplication. Similarly, `true` is an identity element with respect to logical conjunction, but `false` is an identity element with respect to logical disjunction.

Many associative combinining operations also have an identity element, but some do not.

For example the minimum or maximum of two values on which a total ordering is defined does not have an identity element. Neither does the first or last of two values or the concatenation of collections that cannot be empty.

If we import `zio.prelude._` we can use the same `<>` operator as we did to combine two values for which an `Associative` instance was defined.

Since the `identity` value is a single value and not an operator we access it slightly different. Typically we will use the `apply` method on the `Identity` companion object.

```scala

val zero: Int =
  Identity[Sum[Int]].identity
// zero: Int = 0
```

The `apply` operator on the `Identity` companion object "looks up" the instance of the type class for the specified type, failing with a compilation error if the instance cannot be found. This operator is available on every functional abstraction in ZIO Prelude and is sometimes referred to as its "summoner".

Notice also here that we are using the `Sum` new type to specify that we are interested in the identity element with respect to addition rather than another operator such as multiplication. See the section on new types in the documentation for the `Associative` abstraction for a discussion of the use of new types to disambiguate type class instances.

The main value the identity element adds over just having the associative combining operation is the ability to handle cases where we do not have any value of the type at all.

With the `Associative` abstraction we know how to combine any two values of type `A`. So if we have one or more values of type `A` we can always combine them all together by repeatedly applying the `combine` operator.

However, we need at least one `A` value to get the process started. If we don't have any `A` value initially we have no way to create one with the `Associative` abstraction alone.

To see this, consider the example of computing the minimum value in a collection.

If the collection has exactly one element we can just return it. And if it has more than one element we can combine them using the `min` operator until we only have a single value.

But what do we do if the collection has no values? We can't do anything so we return an `Option` that is a `Some` with the minimum if there is at least one element and `None` otherwise.

```scala

def min[A: Ord](as: List[A]): Option[A] =
  Max.wrapAll(as) match {
    case h :: t => Some(t.foldLeft(h)(_ <> _))
    case Nil    => None
  }
```

Compare this to computing the sum of a collection. Now If the collection is empty we can just return the identity value.

```scala
def sum[A](as: List[A])(implicit identity: Identity[Sum[A]]): A =
  Sum.wrapAll(as).foldLeft(identity.identity)(identity.combine(_, _))
```

The identity element handles the case where the collection is empty for us, allowing us to always return a summary value.

If we don't have an identity value we can only guarantee that we can return a summary value if the collection is known not to be empty, such as a `NonEmptyChunk` or a `NonEmptyList`. This is why these data types can be valuable.

If the collection could be empty and we only have a `combine` operator we may not be able to return a summary value and the best we can do is an `Option`.

One way to think of this is that the `None` case of the `Option` handles the possibility of failure. Another is that the `None` case of the `Option` provides the identity element for the `combine` operator, which we can express like this:

```scala
implicit def OptionIdentity[A: Associative]: Identity[Option[A]] =
  new Identity[Option[A]] {
    def combine(left: => Option[A], right: => Option[A]): Option[A] =
      (left, right) match {
        case (Some(a1), Some(a2)) => Some(a1 <> a2)
        case (_, None)            => left
        case (None, _)            => right
        case (_, _)               => None
      }
    def identity: Option[A] = None
  }
```

This says that we can define an `Identity` instance for any `Option[A]` as long as there is an `Associative` instance defined for `A`. The `Option` provides the "free" structure of an identity element to go from just having an associative `combine` operator to having an `identity` element.

This can be useful to keep in mind if you are dealing with a collection that could be empty and need an `Identity` instance but only have an `Associative` instance. You can always map your collection type to an `Option` and then you will have an `Identity` instance.

Data types with an associative `combine` operator and an `identity` element are very common. With an `Identity` instance defined for them and the other tools in ZIO Prelude you are in a good position to handle working with even complex data types.

---

## Introduction(Concrete-types)

ZIO Prelude features a set of fundamental functional abstractions for working with concrete data types.

These abstractions fall into two categories.

## Properties Of Concrete Types

The first set of abstractions describe different properties that values of a data type may have. For example, we may be able to compare them for equality, hash them, or order them.

All of these properties can be thought of as functions `A => ???` or `(A, A) => ???`. They give us a way of taking one or more values of a data type and converting them into some other data type, whether that is a `Boolean` indicating whether two values are equal or an `Int` that is the result of hashing a single value.

These properties are often already described in the Scala standard library in some way, but the abstractions in ZIO Prelude give us the tools to define them in ways that are more type safe, extensible, and compositional.

For example, the `Equal` type class prevents us from doing equality checks on unrelated types or types that do not have a meaningful definition of equality, which can be a source of bugs when using the `==` operator from the Scala standard library. We can also define these properties for data types that we do not control, for example providing a custom way of hashing a data type implemented by someone else.

### Equal

The `Equal` abstraction describes a type that can be compared for equality. Its fundamental representation is as a function `(A, A) => Boolean`.

The `Equal` abstraction allows us to define what it means for two instances of a type to be equal, avoiding pitfalls in Scala such as reference equality for arrays. It also lets us avoid bugs caused by accidentally comparing two unrelated types or checking equality for types such as functions that do not have well-defined notions of equality.

### Ord

The `Ord` abstraction builds on the `Equal` abstraction and describes a type that has a total ordering. Its fundamental representation is as a function `(A, A) => Ordering`, where `Ordering` can either be `LessThan`, `EqualTo` or `GreaterThan`.

The `Ord` abstraction serves much the same purpose as the `Ordering` type class from the Scala standard library, but it has improved type inference due to its use of variance. It also integrates with the other functional abstractions in ZIO Prelude, so for example defining an ordering for a type also defines a way of comparing instances of that type for equality that is by definition consistent with the ordering.

### Hash

The `Hash` abstraction builds on the `Equal` abstraction in a different way, describing a type that can be hashed. Its fundamental representation is as a function `A => Int`.

The `Hash` abstraction allows us to define our own way of hashing data types instead of being forced to use the implementation of `hashCode` for existing data types. It also ensures that the definition of the hash is consistent with the definition of equality, another common source of bugs.

### Debug

The `Debug` abstraction describes a type that can be rendered for debugging purposes. Its fundamental representation is as a function `A => Repr`, where `A` is a structured representation of the value that supports rendering in various formats.

The `Debug` abstraction allows us to define how a data type should be rendered, for example rendering an `Array` to display its values rather than just its memory location. The more structured representation also allows rendering in different formats, for example including fully qualified names in the rendering of the data type so the rendering is itself valid Scala code.

## Combining Concrete Types

The second set of abstractions describe different ways of combining two values of a given type. These can all be thought of as functions `(A, A) => A`.

Combining is quite fundamental because it is how we build more complex structures from simpler ones.

Generally unary operators are not very interesting because they don't add additional structure, they just modify existing structure.

In contrast, given the ability to combine two values of a given type we can create a new value of that type with richer structure. And if we can do that once we can do it again and again after that, creating values with arbitrarily complex structure.

The abstractions in ZIO Prelude for describing combining values are each based on fundamental algebraic properties that the combining operation can have.

### Associative

The most basic such abstraction is `Associative`, which describes a combining operation that is associative.

```scala
(a <> b) <> c === a <> (b <> c)
```

This essentially means that if we are combining three values then the order of operations doesn't matter. We can combine `a` and `b` and then combine the result with `c` or we can combine `b` and `c` and then combine `a` with the result.

This is quite an important property for being able to build more complex structures from simpler ones. Otherwise the result of sequentially building more complex structures from simpler ones is not well defined because it depends on the order of operations and there are an exponentially increasing number of ways we could order these operations as we combine more and more values.

There are also an extremely large number of data types that can be combined using associative operations.

For example integers can be combined associatively using addition and multiplication, among other operations. String concatenation is also associative, as is concatenation of lists of any concrete type.

For these reasons the `Associative` abstraction forms the root of the set of abstractions for describing ways to combine concrete types in ZIO Prelude.

### Commutative

The `Commutative` abstraction builds on the `Associative` abstraction by describing a combining operation that is not only associative but also commutative.

```scala
a <> b === b <> a
```

This means that not only does the order of operations not matter, but the order in which we combine the values also does not matter.

Commutativity is a powerful property that gives us additional ability to reason about our code. For instance, if the values we are combining are in an arbitrary order, say because they are being produced asynchronously by multiple concurrent processes, then commutativity guarantees that we will get the same result no matter what order we combine them in.

Commutativity is a stronger property than associativity but that also means there are fewer ways of combining values that are commutative than there are ways of combining that are associative. For example, integer addition and multiplication are commutative but string concatenation is not.

### Identity

The `Identity` abstraction builds on the `Associative` abstraction in a different way. It describes a combining operation that is not only associative but also has an identity element.

```scala
a <> identity === a
identity <> a === a
```

This identity element is a "neutral value" that can be combined with any other value an unlimited number of times with the combining function and does not change the other value.

For example `0` is an identity element with respect to addition, `1` is an identity element with respect to multiplication, the empty string is an identity element with respect to string concatenation, and the empty list is an identity element with respect to list concatenation.

An identity element gives us a "starting point" for combining values that we know will never change the value. It also tells us when we can safely eliminate a value from the combining operation because we know the value will never change the result.

### Inverse

The `Inverse` abstraction further builds on the `Identity` abstraction with the concept of an inverse to the combining operation.

```scala
inverse(a, a) === identity
```

Whereas the combining operation adds structure the inverse operation takes it away, so that combining any value with itself using the inverse operation removes all structure and just returns the identity element.

Defining the inverse as a binary operator rather than a unary operator like this allows us to define an inverse operation even for types that do not have inverse values. 

For example subtraction would be an inverse operation with addition being the combining operation and zero being the identity element. And we can define it on the natural numbers as well as the integers.

---

## Inverse

`Inverse[A]` describes a type that has a `combine` operator and also has an `inverse` operator that is the inverse of the `combine` operator.

Its signature is:

```scala
trait Associative[A] {
  def combine(left: => A, right: => A): A
}

trait Identity[A] extends Associative[A] {
  def identity: A
}

trait Inverse[A] extends Identity[A] {
  def inverse(left: => A, right: => A): A
}
```

For example, subtraction is an `inverse` operator with addition being the `combine` operator and `0` being the `identity` value.

While the `combine` operator adds structure, the `inverse` operator takes it away, undoing the structure that the `combine` operator added. As a result, applying the `inverse` operator to a value and itself takes away all the structure, returning the identity element.

```scala
inverse(a, a) === identity
```

It is important to note that `inverse` is a binary operator rather than a unary operator.

When we think of an inverse with respect to integer addition it is tempting to think of the inverse as the unary operator of negation. This does indeed have the attractive property that the sum of any value and its inverse is zero.

However, this definition turns out to be quite limiting because it does not allow us to describe an inverse for types that do not contain negative values.

To see this, consider the case of natural numbers, represented by the `Natural` new type in ZIO Prelude. By definition there are no negative natural numbers and thus we cannot define a unary inverse operator for `Natural`.

However, natural numbers have a well defined notion of subtraction so it feels like we are missing something here. The solution is to view `inverse` as a binary operator.

We can then define an `Inverse` instance for `Natural` like this:

```scala

implicit val NaturalInverse: Inverse[Natural] =
  new Inverse[Natural] {
    def combine(left: => Natural, right: => Natural): Natural =
      Natural.plus(left, right)
    val identity: Natural =
      Natural.zero
    def inverse(left: => Natural, right: => Natural): Natural =
      Natural.minus(left, right)
  }
// NaturalInverse: Inverse[Natural] = repl.MdocSession$MdocApp0$$anon$1@347e848c
```

This also lets us define `Inverse` instances for other data types that we would not otherwise be able to. For example, what is the inverse of `Set(1, 2, 3)`?

Logically it is the set of all integers other than `1`, `2`, and `3` but we have no way of efficiently representing that because a `Set` is a collection of concrete values. So we would not be able to define an `Inverse` instance for `Set` this way.

On the other hand, by defining `Inverse` as a binary operator we can quite easily do so.

```scala
implicit def SetInverse[A]: Inverse[Set[A]] =
  new Inverse[Set[A]] {
    def combine(left: => Set[A], right: => Set[A]): Set[A] =
      left | right
    val identity: Set[A] =
      Set.empty
    def inverse(left: => Set[A], right: => Set[A]): Set[A] =
      left &~ right
  }
```

The `inverse` operator is set difference, with the `combine` operator being set union and the empty set being the `identity` value.

The `inverse` abstraction tends to be used less frequently than the other abstractions for combining values. In writing programs, we typically want to build up more complex values from simpler ones, even if it is just tearing down one data type to build another.

The `Inverse` abstraction is most useful when we want to generalize over the notion of "subtraction" for different types. For example, if we are working with maps we might be interested in defining an operation that lets us return the difference between one map and another.

```scala
def diff[A, B: Equal : Inverse](left: Map[A, B], right: Map[A, B]): Map[A, B] =
  right.foldLeft(left) { case (map, (a, b)) =>
    val b1 = Inverse[B].inverse(map.getOrElse(a, Identity[B].identity), b)
    if (b1 === Identity[B].identity) map - a else map + (a -> b1)
  }
```

We look up the value for every key in the `right` map in the `left` map, returning the `identity` value if it does not exist. We then compute the difference using the `inverse` operator.

If the difference is equal to the `identity` value we remove the key from the map. Otherwise we add it.

Let's see it in action.

```scala
val peopleWhoOweMe: Map[String, Sum[Int]] =
  Map("Alice" -> Sum(1000), "Bob" -> Sum(1000))
// peopleWhoOweMe: Map[String, Sum[Int]] = Map("Alice" -> 1000, "Bob" -> 1000)

val peopleIOwe: Map[String, Sum[Int]] =
  Map("Alice" -> Sum(1000), "Carol" -> Sum(1000))
// peopleIOwe: Map[String, Sum[Int]] = Map("Alice" -> 1000, "Carol" -> 1000)

val myNetFinancialPosition: Map[String, Int] =
  diff(peopleWhoOweMe, peopleIOwe)
// myNetFinancialPosition: Map[String, Int] = Map(
//   "Bob" -> 1000,
//   "Carol" -> -1000
// )

val friends: Map[String, Set[String]] =
  Map("Alice" -> Set("Bob", "Carol"), "Bob" -> Set("Alice", "Carol"))
// friends: Map[String, Set[String]] = Map(
//   "Alice" -> Set("Bob", "Carol"),
//   "Bob" -> Set("Alice", "Carol")
// )

val unfriends: Map[String, Set[String]] =
  Map("Alice" -> Set("Bob", "Carol"), "Bob" -> Set("Alice"))
// unfriends: Map[String, Set[String]] = Map(
//   "Alice" -> Set("Bob", "Carol"),
//   "Bob" -> Set("Alice")
// )

val updatedFriends: Map[String, Set[String]] =
  diff(friends, unfriends)
// updatedFriends: Map[String, Set[String]] = Map("Bob" -> Set("Carol"))
```

In the first example the two maps represent assets and liabilities. The difference between them is our net financial position with each of our counterparties.

In the second example the first map represents the friends of users of a social networking site and the second represents friend removals for each user that need to be processed. The difference between them is the updated friends for each user we should return.

Often you won't need to use the `Inverse` abstraction directly. For example, functionality similar to the above is provided by the `ZSet` data type in ZIO Prelude.

However, when you do need to work some notion of subtraction `Inverse` is here and will let you describe that shared structure in a composable way.

---

## Ord

`Ord[A]` describes a total ordering on values of type `A`.

Its signature is:

```scala
trait Equal[-A] {
  def equal(left: A, right: A): Boolean
}

trait Ord[-A] {
  def compare(left: A, right: A): Ordering
  final def equal(left: A, right: A): Boolean =
    compare(left, right) == Ordering.Equals
}

sealed trait Ordering
object Ordering {
  case object LessThan extends Ordering
  case object GreaterThan extends Ordering
  case object Equals extends Ordering
}
```

That is, we can compare any two values of type `A` and either the `left` value is less than the `right`, the `left` value is greater than the `right`, or the two values are equal.

Furthermore, the results of these comparisons must satisfy the properties that we expect of a total ordering. For example, if `a` is greater than `b` and `b` is greater than `c` then `a` must be greater than `c` and if `a` is greater than `b` then `b` must be less than `a`.

The `Ord` abstraction builds on the `Equal` abstraction because being able to define a total ordering on values of a type implies being able to compare values of that type for equality, as shown by our implementation of the `equal` operator in terms of `compare` above. This means that if we define an instance of the `Ord` abstraction for a data type we automatically get an instance of the `Equal` abstraction for free and our definition of equality is guaranteed to be consistent with the ordering.

The `Ord` abstraction is similar to the `Ordering` type class in the Scala standard library and in fact we can convert any `scala.math.Ordering` to an `Ord` using the `fromScala` operator and any `Ord` to a `scala.math.Ordering` using the `toScala` operator.

```scala

val toScala: scala.math.Ordering[Int] =
  Ord[Int].toScala
// toScala: Ordering[Int] = zio.prelude.Ord$$anonfun$toScala$2@3e095343

val fromScala: Ord[Int] =
  Ord.fromScala(scala.math.Ordering[Int])
// fromScala: Ord[Int] = zio.prelude.Ord$$anonfun$fromScala$2@5c921216
```

The main advantages that the `Ord` abstraction provides over the `Ordering` in the Scala standard library is improved type inference due to use of variance and integration with the other functional abstractions in ZIO Prelude.

The `Ordering` type class in the Scala standard library is invariant, meaning that from the perspective of the Scala compiler an `Ordering[A]` and an `Ordering[B]` are completely unrelated, even if `A` is a subtype of `B`. If we look at the signature of the `compare` operator on `Ord` we can see that this does not reflect the natural variance of this type.

An `Ord[A]` is a consumer of `A` values, it takes two `A` values and returns a result describing whether one is less than, greater than, or equal to the other. If we know how to order values of type `A` and `B` is a subtype of `A` then we also know how to order values of type `B`, because every `B` is an `A`!

The version of `Ordering` in the Scala compiler "forgets" this information because it does not use variance, which can lead to type inference problems.

To see this, consider a situation like the following. We have a domain model where customer accounts may be either business accounts or consumer accounts, each of which have a unique identifier.

```scala
sealed trait CustomerAccount

case class BusinessAccount(id: String) extends CustomerAccount
case class ConsumerAccount(id: String) extends CustomerAccount
```

We might define a total ordering for all accounts like this:

```scala

implicit val CustomerAccountOrd: Ord[CustomerAccount] =
  Ord.make {
    case (BusinessAccount(id1), BusinessAccount(id2)) => id1 =?= id2
    case (BusinessAccount(_), ConsumerAccount(_))     => Ordering.LessThan
    case (ConsumerAccount(_), BusinessAccount(_))     => Ordering.GreaterThan
    case (ConsumerAccount(id1), ConsumerAccount(id2)) => id1 =?= id2
  }
// CustomerAccountOrd: Ord[CustomerAccount] = zio.prelude.Ord$$anon$4@1f53457b
```

Basically this says that we are ordering all business accounts before all consumer accounts and then within each account type we are ordering by the identifier. Note that we are using the `=?=` operator which is available on any type for which an `Ord` is defined to return the result of comparing two values of that type.

We can use this ordering to compare two customer accounts

```scala
val customerAccount1: CustomerAccount = BusinessAccount("abc")
// customerAccount1: CustomerAccount = BusinessAccount(id = "abc")
val customerAccount2: CustomerAccount = ConsumerAccount("def")
// customerAccount2: CustomerAccount = ConsumerAccount(id = "def")

val compareCustomerAccounts: Boolean =
  customerAccount1 < customerAccount2
// compareCustomerAccounts: Boolean = true
```

We can also use it to compare two business accounts or two consumer accounts.

```scala
val consumerAccount1: ConsumerAccount = ConsumerAccount("abc")
// consumerAccount1: ConsumerAccount = ConsumerAccount(id = "abc")
val consumerAccount2: ConsumerAccount = ConsumerAccount("def")
// consumerAccount2: ConsumerAccount = ConsumerAccount(id = "def")

val compareConsumerAccounts: Boolean =
  consumerAccount1 < consumerAccount2
// compareConsumerAccounts: Boolean = true
```

This just works because a consumer account is a customer account and we know how to order customer accounts.

Let's see what happens when we try to do the same thing with the `Ordering` type class defined in the Scala standard library.

```scala

sealed trait CustomerAccount

case class BusinessAccount(id: String) extends CustomerAccount
case class ConsumerAccount(id: String) extends CustomerAccount

implicit val CustomerAccountScalaOrdering: scala.math.Ordering[CustomerAccount] =
  new scala.math.Ordering[CustomerAccount] {
    def compare(left: CustomerAccount, right: CustomerAccount): Int =
      (left, right) match {
        case (BusinessAccount(id1), BusinessAccount(id2)) => id1 compare id2
        case (BusinessAccount(_), ConsumerAccount(_))     => -1
        case (ConsumerAccount(_), BusinessAccount(_))     => 1
        case (ConsumerAccount(id1), ConsumerAccount(id2)) => id1 compare id2
      }
}
// CustomerAccountScalaOrdering: Ordering[CustomerAccount] = repl.MdocSession$MdocApp2$$anon$1@7f831d97
```

Besides the lack of specific types in modeling the result of the compare operator this works fine for comparing two customer accounts.

```scala
val customerAccount1: CustomerAccount = BusinessAccount("abc")
// customerAccount1: CustomerAccount = BusinessAccount(id = "abc")
val customerAccount2: CustomerAccount = ConsumerAccount("def")
// customerAccount2: CustomerAccount = ConsumerAccount(id = "def")

val compareCustomerAccounts: Boolean =
  customerAccount1 < customerAccount2
// compareCustomerAccounts: Boolean = true
```

But look what happens when we try to compare two consumer accounts.

```scala
val consumerAccount1: ConsumerAccount = ConsumerAccount("abc")
val consumerAccount2: ConsumerAccount = ConsumerAccount("def")

val compareConsumerAccounts: Boolean =
  consumerAccount1 < consumerAccount2
// error: value < is not a member of MdocApp2.this.ConsumerAccount
//   consumerAccount1 < consumerAccount2
//   ^^^^^^^^^^^^^^^^^^
```

This code doesn't compile! The Scala compiler is unable to compare the consumer accounts because of the lack of variance on `scala.math.Ordering`.

We could widen the type of `ConsumerAccount` to `CustomerAccount` but that is unergonomic. And it wouldn't even work if, for example, we were trying to sort the list and needed to maintain the type information that these were consumer accounts rather than just customer accounts.

The `Ord` abstraction helps us avoid problems like this and integrates seamlessly with the rest of ZIO Prelude.

## Defining Ord Instances

`Ord` instances are already defined for all the types in `ZIO` and the Scala standard library that have a total ordering.

ZIO Prelude is also very good at automatically deriving `Ord` instances for more complex data types like tuples. For example if we have a pair of `Double` values representing `x` and `y` coordinates an ordering is already automatically defined for it.

```scala

Ord[(Double, Double)]
// res5: Ord[(Double, Double)] = zio.prelude.Ord$$anon$5@949fc49
```

The default instance for tuples will order values by the first field and then if the first field is the same by the second field and so on.

If we want to define an ordering for our own data type and potentially include our own ordering logic we can use the `make` operator.

```scala
case class Point(x: Double, y: Double)

object Point {
  implicit val PointOrd: Ord[Point] =
    Ord.make { (left, right) =>
      (left.y =?= right.y) <> (left.x =?= right.x)
    }
}
```

The `<>` operator combines two `Ordering` values, using the result of the first one if it is `GreaterThan` or `LessThan` and otherwise using the result of the second one. You can think of this like sorting by last name and then first name.

So now we are sorting by the `y` coordinate and then the `x` coordinate.

Just like the `Equal` abstraction, the `Ord` abstraction supports a `contramap` operator, so we can define a total ordering on a data type as long as we can transform it into another data type that has a total ordering in a way that preserves information.

This is particularly useful if we want to define an `Ord` instance for our case classes because we can just derive it from the instances for tuples.

```scala
object Point {
  implicit val PointOrd: Ord[Point] =
    Ord[(Double, Double)].contramap(point => (point.y, point.x))
}
```

ZIO Prelude makes it as easy as possible to use and define instances of `Ord` so we can abstract over the property of having a total ordering and integrate it with other parts of ZIO Prelude.

---

## Introduction(Functional-abstractions)

ZIO Prelude features a next generation approach to functional abstractions. This approach is based on the following ideas:

1. **Algebraic** - Abstractions should describe fundamental algebraic properties.
2. **Compositional** - These abstractions should describe properties that are orthogonal to each other, allowing definition of higher level abstractions as the composition of more basic ones.
3. **Lawful** - Abstractions should be defined in terms of laws.

The functional abstractions in ZIO Prelude can be broadly divided into two categories.

- **[Abstractions For Concrete Types](concrete-types/index.md)** - These abstractions define properties of concrete types, such as `Int` and `String`, as well as ways of combining those values.
- **[Abstractions For Parameterized Types](parameterized-types/index.md)** - These abstractions define properties of parameterized types such as `List` and `ZIO` and ways of combining them.

As we will see, there is a deep symmetry between the abstractions defined on concrete and parameterized types, such as concepts of _associative operations_, _commutative operations_, and _identity_. This reflects the fundamental nature of these algebraic properties and their ability to unify what were previously separate concepts.

## Abstractions

An _abstraction_ describes some common structure that different data types share. In Scala, we can encode this using a trait that describes that common structure in terms of a set of operators as well as laws that those operators must follow.

For example, we can think of many data types that share the structure of having an associative combining operation. Integer addition is associative, as is string concatenation and list concatenation, among others.

We can describe this common structure using the `Associative` trait.

```scala
trait Associative[A] {
  def combine(left: => A, right: => A): A
}
```

We can then define various concrete values that extend this trait to describe how different data types share this common structure.

```scala
val IntAssociative: Associative[Int] =
  new Associative[Int] {
    def combine(left: => Int, right: => Int): Int =
      left + right
  }
// IntAssociative: Associative[Int] = repl.MdocSession$MdocApp$$anon$1@28717b33
```

Note however that the signature of the trait is not sufficient to define the abstraction.

The signature merely says that we must take two `A` values and return an `A` value. It doesn't say anything about what this combining operation is supposed to do with the value.

With just that signature we could do anything we want in the implementation of `combine` such as subtracting one integer from the other, which is definitely not associative.

```scala
val IntNotAssociative: Associative[Int] =
  new Associative[Int] {
    def combine(left: => Int, right: => Int): Int =
      left - right // don't do this
  }
// IntNotAssociative: Associative[Int] = repl.MdocSession$MdocApp$$anon$2@1993d53d
```

This shows that abstractions are not meaningful without laws. Abstractions describe some common structure that is shared between different data types but without laws we don't know what this structure is supposed to be.

In this case the law is that the combining operation must be associative, which we can write in pseudocode as.

```scala
(a <> b) <> c === a <> (b <> c)
```

Here `<>` represents the combining operation and `a`, `b`, and `c` represent any possible combination of values of the given type.

Every abstraction in ZIO Prelude is described by a trait like the one above and is defined in terms of a set of laws.

ZIO Prelude provides **instances for these abstractions** for a variety of types from ZIO and the Scala standard library. ZIO Prelude also provides **tools for testing** that instances of an abstraction satisfy the appropriate laws.

## Using Abstractions

There are several ways you can use the abstractions described in this library.

The first and most direct, which actually does not require depending on ZIO Prelude at all, is to use these abstractions as inspiration for defining operators on your own data type.

The common structure described by these abstractions exists independent of any library. You don't need ZIO Prelude to define an associative combining operation on your own data type.

However, thinking about whether an associative combining operation exists for your data type, and what it would look like, can help you write better code.

As a simple example, say you want to compute the average of values from some large data set, and you would like to split the work up between different concurrent processes or possibly even different nodes in a distributed network.

Your first stab at the accumulator for the running average might look like this:

```scala
case class RunningAverage(value: Double)
```

However, if you think about it for a minute you will realize that this data type does not support an associative combining operation for combining two averages. This is going to be a serious problem because it means the result is not going to be well-defined if you combine averages from different processes or nodes.

Thinking about the abstractions in ZIO Prelude you might come up with a representation like this:

```scala
case class RunningAverage(sum: Double, count: Int) { self =>
  def average: Double =
    sum / count
  def combine(that: RunningAverage): RunningAverage =
    RunningAverage(self.sum + that.sum, self.count + that.count)
}

object RunningAverage {
  val empty: RunningAverage =
    RunningAverage(0.0, 0)
}
```

Now this data type does have an associative combining operation. In fact the combining operation is both associative and commutative and has an identity element.

This will make it much easier for you to solve your problem because now the different processes or nodes can compute the averages for their partitions independently, and you can combine them in any order.

And you didn't need to use any code from ZIO Prelude to do this. ZIO Prelude was hopefully just a good source of ideas of different algebraic properties that can exist and how they can be important.

This is a great way to get started with functional abstractions. Your colleagues don't have to learn anything new, you just get to write better code because you are taking advantage of these algebraic properties.

This is also the approach taken by ZIO ecosystem libraries.

ZIO ecosystem libraries generally do not directly expose any functional abstractions but still expose a highly compositional interface because their design is based on algebraic properties like this. Users don't have to learn about these abstractions unless they want to, they just get to benefit from better library design.

## Using Type Classes

The second way you can use the abstractions in ZIO Prelude is by leveraging the _type classes_ defined in the library to take the boilerplate out of your own code.

Type classes are a way of encoding functional abstractions in Scala and other programming languages. In the type class pattern, we take the same code as above but define the instances of the type class as `implicit`.

```scala

trait Associative[A] {
  def combine(left: => A, right: => A): A
}

object Associative {

  implicit val IntAssociative: Associative[Int] =
    new Associative[Int] {
      def combine(left: => Int, right: => Int): Int =
        left + right
    }

  implicit def ListAssociative[A]: Associative[List[A]] =
    new Associative[List[A]] {
      def combine(left: => List[A], right: => List[A]): List[A] =
        left ::: right
    }
}
```

If the instance of the type class depends on other parameters, like the `A` in `ListAssociative` we define it as an `implicit def`. Otherwise, we define it as an `implicit val`.

We can think of the `implicit` keyword as associating the type `Int` with the value `IntAssociative`. So now if we ask the Scala compiler for the `Associative` instance for `Int` it will be able to find it.

In the type class pattern we also typically define extension methods that will be available on any data type for which an instance of the type class is defined.

```scala
implicit final class AssociativeSyntax[A](private val self: A) {
  def <>(that: => A)(implicit associative: Associative[A]): A =
    associative.combine(self, that)
}
```

This machinery allows us to use the `<>` operator to combine values of any type as long as an `Associative` instance is defined for it.

```scala
val int: Int =
  1 <> 2
// int: Int = 3

val list: List[Int] =
  List(1, 2, 3) <> List(4, 5, 6)
// list: List[Int] = List(1, 2, 3, 4, 5, 6)
```

Of course, we didn't really need all of this machinery to add two numbers or concatenate two lists, but where this pattern gets powerful is when we can use it to combine more complex data types in a principled way.

For example, say we have an application where users can vote on content they are interested in learning more about. We might have a data structure to keep track of the number of votes for different topics like this.

```scala
final case class Topic(value: String)
final case class Votes(value: Int)
final case class VoteMap(map: Map[Topic, Votes])
```

A common thing we might want to do is combine two `VoteMap` values, for example if the user has a local copy of the `VoteMap` and we want to update it with a new batch of votes from the server.

We could do that manually like this.

```scala
final case class Topic(value: String)

final case class Votes(value: Int) { self =>
  def combine(that: Votes): Votes =
    Votes(self.value + that.value)
}

final case class VoteMap(map: Map[Topic, Votes]) { self =>
  def combine(that: VoteMap): VoteMap =
    VoteMap(that.map.foldLeft(self.map) { case (map, (topic, votes)) =>
      map + (topic -> map.getOrElse(topic, Votes(0)).combine(votes))
    })
}
```

This isn't the worst, but it isn't really the kind of code we want to be writing. We want to be thinking about the logic of our application rather than how to combine maps.

This is where ZIO Prelude can help.

The way we're combining these maps actually follows a pattern. If a key is in a single map we include it in the combined map with its associated key and if a key is in both maps we include it in the combined map with the result of combining the values associated with that key.

We might see that ourselves, but it would be hard to generalize that logic in a way that is worth factoring out. How often are we going to combine maps like this and what exactly does it mean to combine the keys?

Let's look at how ZIO Prelude can help us clean this up.

```scala

case class Topic(value: String)

case class Votes(value: Int)

object Votes {
  implicit val VotesAssociative: Associative[Votes] =
    new Associative[Votes] {
      def combine(left: => Votes, right: => Votes): Votes =
        Votes(left.value + right.value)
    }
}

case class VoteMap(map: Map[Topic, Votes])

object VoteMap {
  implicit val VoteMapAssociative: Associative[VoteMap] =
    new Associative[VoteMap] {
      def combine(left: => VoteMap, right: => VoteMap): VoteMap =
        VoteMap(left.map <> right.map)
    }
}
```

All of that logic of combining the two maps just goes away!

ZIO Prelude knows that we can define an associative combining operation for any two values of type `Map[A, B]` as long as there is an associative combining operation for the `B` values. All we have to do is tell ZIO Prelude how to combine the `B` values, which in this case is quite simple, and it can do the rest.

This is a great example of the practical value that ZIO Prelude can bring. It took this low level logic of how to combine these two maps and just handled it for us.

## Using Generic Programming

The third way you can use the abstractions in ZIO Prelude is by leveraging type classes to do generic programming at the level of these abstractions.

For example, you might find yourself doing a lot of "map reduce" type operations on collections like this.

```scala
def wordCount(lines: List[String]): Int =
  lines.map(_.split(" ").length).sum
```

You might like this way of working with collections and wonder how you can generalize it.

A first step could be to recognize that `sum` is just a particular combining operation that has an identity element. You could then use the `Identity` abstraction in ZIO Prelude to generalize over data types that support this kind of combining operation with an identity element.

```scala
def mapReduce[A, B](as: List[A])(f: A => B)(implicit identity: Identity[B]): B =
  ???
```

You might want to go even further though and generalize over the collection type. You can clearly implement a similar operator for a `Vector` or another collection type so how do you generalize over that?

You could do that with ZIO Prelude's `ForEach` abstraction, which describes parameterized data types with some structure where the elements in the structure can be replaced while preserving the structure itself.

Using this, you could rewrite your operator like so.

```scala
def mapReduce[F[+_]: ForEach, A, B: Identity](as: F[A])(f: A => B): B =
  ???
```

This example illustrates some of the benefits as well as the pitfalls of generic programming using type classes.

If you want to do generic programming in terms of type classes, ZIO Prelude can go as far as you want to go.

However, there is a definite trade off in these three snippets.

The first one is overly specific but is understandable to any Scala programmer. The last one is beautiful and elegant if you understand the necessary concepts but incomprehensible otherwise.

In addition, there is a danger in using type classes to do generic programming that we reinvent the wheel. The generalized "map reduce" operator we developed is just the existing `foldMap` operator from ZIO Prelude!

```scala
def mapReduce[F[+_]: ForEach, A, B: Identity](as: F[A])(f: A => B): B =
  as.foldMap(f)
```

This illustrates the risk that sufficiently general abstractions or operators are likely to already be defined by a functional programming library.

None of this is meant to argue against this style of generic programming but merely to point out that it is one of several ways to use these functional abstractions and none of them are necessarily "better" than others. The right approach to using these abstractions is the one that works for you and your team.

---

## AssociativeBoth

`AssociativeBoth[F]` describes a way of combining two values `F[A]` and `F[B]` into a value `F[(A, B)]` that is associative.

Its signature is:

```scala
trait AssociativeBoth[F[_]] {
  def both[A, B](fa: => F[A], fb: => F[B]): F[(A, B)]
}
```

If we import `zio.prelude._` we can use the `zip` operator or its symbolic alias `<*>` to combine any two values of a parameterized type `F` that have an `AssociativeBoth` instance defined for them.

The `both` operator must be associative, so if we combine `fa` and `fb` with `both` and then combine the result with `fc` we must get the same value as if we combine `fb` and `fc` and then combine `fa` with the result. That is, after reassociating nested tuples:

```scala
(fa <*> fb) <*> fc === fa <*> (fb <*> fc)
```

Notice that this is the same as the associative law for concrete types described by the `Associative` abstraction, just lifted into the context of parameterized types.

The `both` operator corresponds to running the left value and then running the right value.

We can see this because to produce a tuple `(A, B)` we need to run both the left value and the right value and to do it in a way that is associative we need to do it in a consistent order. Technically running the right value and then running the left value would also be associative but since we can always flip the order of the arguments it is helpful to adopt the convention that the first value goes on the left and the second value goes on the right.

For example, in the context of `ZIO` the `zip` operator returns a new `ZIO` workflow that describes running the workflow on the left, then running the workflow on the right, and then returning a tuple of their results.

```scala

val helloZIO: ZIO[Console, IOException, Unit] =
  Console.printLine("Hello") <*> Console.printLine("ZIO")
// helloZIO: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.helloZIO(associativeboth.md:26)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.helloZIO(associativeboth.md:26)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@4dae1e15
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@7f0719d6
// )
```

This will print `Hello` to the console on the first line and then `ZIO` on the second line.

If the first workflow fails then the second workflow will never be run.

```scala
val failZIO: ZIO[Console, IOException, Unit] =
  ZIO.fail(new IOException("Fail")) <*> Console.printLine("ZIO")
// failZIO: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.failZIO(associativeboth.md:33)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.failZIO(associativeboth.md:33)",
//     onState = zio.ZIO$$$Lambda$17057/0x0000000804972840@15f43912
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@1bbbed2c
// )
```

Now `ZIO` will never be printed to the console.

What it means to run the left value and run the right value will vary between data types.

For example, for `Either` and `Option` it corresponds to running the left value and then running the right value if the left value is a success.

```scala
def both[E, A, B](fa: => Either[E, A], fb: => Either[E, B]): Either[E, (A, B)] =
  fa match {
    case Left(e)  => Left(e)
    case Right(a) => fb match {
      case Left(e)  => Left(e)
      case Right(b) => Right((a, b))
    }
  }

def both[E, A, B](fa: => Option[A], fb: => Option[B]): Option[(A, B)] =
  fa match {
    case None    => None
    case Some(a) =>
      fb match {
        case None    => None
        case Some(b) => Some((a, b))
      }
  }
```

For collection types it corresponds to the Cartesian product of the left and right values.

```scala
def both[A, B](fa: => List[A], fb: => List[B]): List[(A, B)] =
  for {
    a <- fa
    b <- fb
  } yield (a, b)
```

If we think of a collection as representing a set of possible states then we can see this as the set of all possible states from taking each element in the left collection and then combining it with each element in the right collection.

For a parser it corresponds to running the left parser and then running the right parser with the updated state if the first parser succeeds.

So far we have been using covariant types that produced values in our example, but there is no reason we have to limit ourselves to that. The `AssociativeBoth` functional abstraction can describe associative ways of combining contravariant and invariant parameterized types as well.

This is a significant advantage over other functional programming libraries, which conflate abstractions for describing ways of combining paramaterized types with abstractions for describing the variance of parameterized types. This requires defining many additional interfaces to try to accomplish the same thing and obscures the fundamental symmetry of the underlying structure here.

To see this, let's start with a `Predicate`.

```scala
trait Predicate[-A] {
  def run(a: A): Boolean
}
```

A `Predicate` is naturally contravariant in the `A` type parameter because `A` appears as an input to `run` but never as an output. We can prove that to ourselves by implementing a `contramap` operator for `Predicate` like this:

```scala
trait Predicate[-A] { self =>
  def run(a: A): Boolean
  def contramap[B](f: B => A): Predicate[B] =
    new Predicate[B] {
      def run(b: B): Boolean =
        self.run(f(b))
    }
}
```

We would like to be able to combine predicates so that if we have predicates about types `A` and `B` we can combine them into a predicate about the tuple `(A, B)`. Especially in combination with the `contramap` operator, which we could use to transform a predicate about a tuple to a predicate about a class class, we could then define predicates about user defined data types in terms of predicates about primitive types.

We can do that with `AssociativeBoth` like this:

```scala

object Predicate {
  implicit val PredicateAssociativeBoth: AssociativeBoth[Predicate] =
    new AssociativeBoth[Predicate] {
      def both[A, B](left: => Predicate[A], right: => Predicate[B]): Predicate[(A, B)] =
        new Predicate[(A, B)] {
          def run(tuple: (A, B)): Boolean =
            left.run(tuple._1) && right.run(tuple._2)
        }
    }

  implicit val PredicateContravariant: Contravariant[Predicate] =
    new Contravariant[Predicate] {
      def contramap[A, B](f: B => A): Predicate[A] => Predicate[B] =
        _.contramap(f)
    }
}
```

Here what the `both` operator means is to run the left predicate and then if it passes to run the right predicate.

When the parameterized type is covariant there are some additional useful operators we can define for it.

```scala
def zipWith[F[+_]: AssociativeBoth : Covariant, A, B, C](
  fa: => F[A],
  fb: => F[B]
)(f: (A, B) => C): F[C] =
  fa.zip(fb).map(f.tupled)

def zipLeft[F[+_]: AssociativeBoth : Covariant, A, B, C](
  fa: => F[A],
  fb: => F[B]
): F[A] =
  zipWith(fa, fb)((a, _) => a)
  
def zipRight[F[+_]: AssociativeBoth : Covariant, A, B, C](
  fa: => F[A],
  fb: => F[B]
): F[B] =
  zipWith(fa, fb)((_, b) => b)
```

The `zipWith` operator uses `zip` to combine two parameterized types and then `map` to combine the outputs with a function. For example, we can generate two random numbers with `ZIO` and then combine their results by adding them.

The `zipLeft` and `zipRight` operators and their symbolic aliases `<*` and `*>` are specialized variants of this that throw away the left or right results. These are useful when we want to include a value of a parameterized type in the larger data type we are building but don't care about its result.

For example, we could avoid creating unnecessary nested tuples when we combine `ZIO` workflows like this:

```scala
val helloFromAssociativeBoth: ZIO[Console, IOException, Unit] =
  Console.printLine("Hello") *>
    Console.printLine("From") *>
    Console.printLine("AssociativeBoth")
// helloFromAssociativeBoth: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.<local MdocApp0>.helloFromAssociativeBoth(associativeboth.md:145)",
//   first = FlatMap(
//     trace = "repl.MdocSession.MdocApp0.<local MdocApp0>.helloFromAssociativeBoth(associativeboth.md:144)",
//     first = Stateful(
//       trace = "repl.MdocSession.MdocApp0.<local MdocApp0>.helloFromAssociativeBoth(associativeboth.md:144)",
//       onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@7410af1b
//     ),
//     successK = zio.ZIO$$Lambda$17058/0x0000000804980840@39030bab
//   ),
//   successK = zio.ZIO$$Lambda$17058/0x0000000804980840@4ea8c4d
// )
```

There is also a variant of `zipWith` for contravariant types called `bothWith` that combines `zip` and `contramap`.

```scala
def bothWith[F[-_]: AssociativeBoth : Contravariant, A, B, C](
  fa: => F[A],
  fb: => F[B]
)(f: C => (A, B)): F[C] =
  fa.zip(fb).contramap(f)
```

Notice again the symmetry here between the `zipWith` and `bothWith` operators.

With this we can combine contravariant data types such as `Predicate` the same way we can with covariant data types.

```scala
case class Email(lines: List[String], sender: String)

val noSuspiciousWords: Predicate[List[String]] =
  new Predicate[List[String]] {
    def run(words: List[String]): Boolean =
      !words.exists(_.contains("get rich"))
  }
// noSuspiciousWords: Predicate[List[String]] = repl.MdocSession$MdocApp0$$anon$5@1fe3c4ca

val noSuspiciousSender: Predicate[String] =
  new Predicate[String] {
    def run(sender: String): Boolean =
      !sender.contains("unclaimedfunds")
  }
// noSuspiciousSender: Predicate[String] = repl.MdocSession$MdocApp0$$anon$6@2e9cc4b5

val spamFilter: Predicate[Email] =
  noSuspiciousWords.bothWith(noSuspiciousSender) { 
    case Email(lines, sender) => (lines, sender)
  }
// spamFilter: Predicate[Email] = repl.MdocSession$MdocApp0$Predicate$1$$anon$1@23096abf
```

The composed predicate will extract the lines and the sender, sending the lines to the `nonSuspiciousWords` predicate and then if it passes sending the sender to the `nonSuspiciousSender` predicate.

We can use the `bothWith` operator with other contravariant types as well. For example, we could combine two sinks to produce a new sink that sends all inputs first to one sink and then to the other.

The `AssociativeBoth` functional abstraction describes a very basic pattern of running one value and then running another value. In addition, the operators that can be defined in terms of it such as `zip` and `zipWith` are very common and practically useful.

If you are working with data types from ZIO or the Scala standard library the value you get from the `AssociativeBoth` abstraction itself may be limited because most of these data types already define these operators directly since they are so useful.

However, there are some cases where these operators are missing such as the `zip` operator on `Either`. In these cases if you bring in ZIO Prelude you will get these missing operators automatically, which can help clean up your code.

If you are defining your own parameterized data type then asking if you can define an `AssociativeBoth` instance for it can be a very helpful exercise.

Associative ways of combining are fundamental to building up more complex structures from simpler ones, so if your data type does not support an associative combining operation it can be worth asking why that is and if there is a way it could be refactored to support such an operation. If it does, defining an `AssociativeBoth` instance will get you a lot of useful operators, especially in combination with the `Covariant` or `Contravariant` and `ForEach` abstractions from ZIO Prelude.

Finally, if you are doing generic programming with the abstractions in ZIO Prelude then `AssociativeBoth` will again be an important operator to describe running one value and then another in a generic way. The orthagonal way that abstractions are described in ZIO Prelude can be particularly valuable here to help you avoid confusing unrelated concepts and describe your shared structure in as general a way as possbile.

---

## AssociativeEither

`AssociativeEither` describes a way of combining two values `F[A]` and `F[B]` into a value `F[Either[A, B]]` that is associative.

Its signature is:

```scala
trait AssociativeEither[F[_]] {
  def either[A, B](fa: => F[A], fb: => F[B]): F[Either[A, B]]
}
```

If we import `zio.prelude._` we can use the `orElseEither` operator or its symbolic alias `<+>` to combine any two values of a parameterized type `F` that have an `AssociativeEither` instance defined for them.

The `either` operator must be associative, so if we combine two values `fa` and `fb` and then combine the result with `fc`, that must be the same as combining `fb` with `fc` and then combining `fa` with the result. That is, after reassociating `Either` values the following property must hold:

```scala
(fa <+> fb) <+> fc === fa <+> (fb <+> fc)
```

This is the same associative law we saw for concrete types described by the `Associative` abstraction, but lifted to the context of parameterized types.

The `either` operator corresponds to running the left value and if that fails running the right value.

To see this, we can observe that since the `either` operator must return an `Either[A, B]` we must choose on some basis whether to return the left value or the right value. Furthermore, we must choose on some consistent basis so that the order of operations does not matter.

There are a couple of trivial ways we could do this, like always running the left value or always running the right value. However, the way we can do this that preserves information is to run the right value and then if it fails in some way to run the right value.

What it means to run the left value and then if it fails run the right value depends on the parameterized type.

For `ZIO` the meaning is quite straightforward. The `orElseEither` operator runs the left `ZIO` workflow and returns its result if it succeeds, otherwise it runs the right `ZIO` workflow and returns its result.

```scala

def orElseEither[R, E, A, B](left: => ZIO[R, E, A], right: => ZIO[R, E, B]): ZIO[R, E, Either[A, B]] =
  left.foldZIO(_ => right.map(b => Right(b)), a => ZIO.succeed(Left(a)))
```

Here we run the `left` workflow and if it is successful we just wrap it up in a `Left`. If the `left` workflow fails we recover from its failure and run the `Right` effect, packaging its result up in a `Right`.

We can see that this is associative because no matter how many `ZIO` workflows we combine with `orElseEither` the result will always be the first one from left to right to successfully complete execution.

Notice here that if the `left` workflow succeeds we never run the right workflow at all or even need to construct it. The fact that `either` and the other binary operators in ZIO Prelude are by name gives us the freedom to model that instead of having to introduce additional interfaces.

Other data types that model failure have similar implementations of the `orElseEither` operator. For example, here are the implementations of `orElseEither` for `Either` and `Option`.

```scala
def orElseEither[E, A, B](left: => Either[E, A], => right: Either[E, B])]): Either[E, Either[A, B]] =
  left match {
    case Left(e) =>
      that match {
        case Left(e)  => Left(e)
        case Right(b) => Right(Right(b))
      }
    case Right(a) => Right(Left(a))
  }

def orElseEither[A, B](left: => Option[A], => right: Option[B])]): Option[Either[A, B]] =
  left match {
    case None =>
      that match {
        case None     => None
        case Right(b) => Some(Right(b))
      }
    case Some(a) => Some(Left(a))
  }
```

In both cases if the `left` value is a success we return its result in a `Left`. Otherwise if the `right` value is a success we return its result in a `Right` and if it is a failure we fail with that error.

Notice in both cases we did not need to evaluate the `right` value if the `left` value was a success. We could have just made the `right` parameter lazy but we could flip any of these binary operators so for correctness it is important that both arguments be lazy.

Another interpretation of `orElseEither` comes from collections. Consider the following implementation of the `orElseEither` operator for `Chunk`.

```scala

implicit val ChunkAssociativeEither: AssociativeEither[Chunk] =
  new AssociativeEither[Chunk] {
    def either[A, B](as: => Chunk[A], bs: => Chunk[B]): Chunk[Either[A, B]] =
      as.map(Left(_)) ++ bs.map(Right(_))
  }
// ChunkAssociativeEither: AssociativeEither[Chunk] = repl.MdocSession$MdocApp0$$anon$1@481dea0f
```

Here we are concatenating the two `Chunk` values, putting the elements from the left `Chunk` in a `Left` and the elements from the right `Chunk` in a `Right`. We can think of this as running the left `Chunk` until it fails by running out of elements and then running the right `Chunk`.

We can see a similar interpretation in the implementation of the `orElseEither` operator for the `Schedule` data type from `ZIO`. It runs the left schedule while it wants to continue and when it stops runs the right schedule, emitting either a `Left` with the output of the left schedule or a `Right` with the output of the right schedule each time.

The `AssociativeEither` abstraction isn't limited to covariant data types.

Let's see how we can use the `orElseEither` operator to combine values of a contravariant type.

The `Predicate` type knows how to evaluate a value of type `A` to return a `Boolean`. This `Boolean` could describe whether the value of type `A` is valid data for example, or whether we should take some further action.

```scala
trait Predicate[-A] {
  def run(a: A): Boolean
}
```

We could implement an instance of the `AssociativeEither` abstraction for `Predicate` like this:

```scala
object Predicate {
  implicit val PredicateAssociativeEither: AssociativeEither[Predicate] =
    new AssociativeEither[Predicate] {
      def either[A, B](left: => Predicate[A], right: => Predicate[B]): Predicate[Either[A, B]] =
        new Predicate[Either[A, B]] {
          def run(either: Either[A, B]): Boolean =
            either match {
              case Left(a) => left.run(a)
              case Right(b) => right.run(b)
            }
        }
    }
}
```

The interpretation is slightly different here. Now failing means not being able to handle a value at all.

The left `Predicate` knows how to determine whether `A` values satisfy the condition and the right `Predicate` knows how to determine whether `B` values satisfy the condition. So when we get an `Either[A, B]` we have to match on it to determine whether it is an `A` that the left `Predicate` can handle at all.

If so, we send it to the left `Predicate` and return its result. Otherwise we send it to the right `Predicate` and return its result.

Just like the `AssociativeBoth` abstraction, if a data type with an `AssociativeEither` instance is covariant there are additional operators we can define on it.

```scala
def orElse[F[+_]: AssociativeEither : Covariant, A](fa: => F[A], fb: => F[A]): F[A] =
  fa.orElseEither(fb).map(_.merge)
```

The `orElse` operator just combines two `F[A]` values with `orElseEither` and then maps the result to merge the left and right sides of the `Either`. This is probably the version of the `orElse` operator we are most familiar with.

We can also define additional operators if a data type that has a `AssociativeEither` instance defined for it is contravariant.

```scala
def eitherWith[F[-_]: AssociativeEither : Contravariant, A, B, C](fa: F[A], fb: F[B])(
  f: C => Either[A, B]
): F[C] =
  fa.orElseEither(fb).contramap(f)
```

The `eitherWith` operator first converts input of type `C` into `Either[A, B]` values with `contramap` and the function `f`, then runs the `fa` and `fb` values using that input and the `orElseEither` operator.

For example, if our data type was a `Predicate` the `eitherWith` operator could split incoming payloads into one of two types, sending them to the appropriate predicate for evaluation and then returning the results. Of course, if we can handle a payload with two types in this way we can also handle payloads with any number of cases by repeatedly applying the `eitherWith` operator.

The `AssociativeEither` functional abstraction represents the second fundamental way of combining two values of a parameterized type. Whereas `AssociativeBoth` combines the `A` and `B` values into the sum type represented by `(A, B)`, `AssociativeEither` combines them into the product type represented by `Either[A, B]`.

For existing data types the `orElseEither` operator and its variants tend to already be implemented, often with more domain specific names, so there isn't necessarily an immediate benefit if you are working with existing data types from ZIO or the Scala standard library.

However, if you are defining your own parameterized types it can be helpful to think about what it would mean for running a value to fail and what it would mean to run another value instead. There are also operators in ZIO Prelude that are defined on types with an `AssociativeEither` instance, so implementing an instance for your own data type can let you take advantage of that functionality and test your implementation with the laws in ZIO Prelude.

Finally, the `AssociativeEither` abstraction can be quite helpful in some cases for writing generic code in terms of the abstractions in ZIO Prelude.

The combination of the ability to express running the left value and then running the right value with `AssociativeBoth` and running the left value and if it fails running the right value with `AssociativeEither` can be quite powerful. For example, we can define many parsers in this way.

---

## AssociativeFlatten

`AssociativeFlatten[F]` describes a way of combining two layers of a value of type `F[F[A]]` into a `F[A]` in a way that is associative.

Its signature is:

```scala
trait AssociativeFlatten[F[+_]] {
  def flatten[A](ffa: F[F[A]]): F[A]
}
```

If we import `zio.prelude._` we can use the `flatten` operator on any data type for which an `AssociativeFlatten` instance is defined to flatten a nested value `F[F[A]]` down to a single value.

This may appear to be a unary operator but it is actually a binary operator because there are two layers of `F` values here, the outer `F` value and the inner `F` value. The `flatten` operator combines these two layers of `F` values into a single value.

The `flatten` operator must be associative, so if we have three layers of nested values flattening the outer and the middle layers and then flattening the resulting layer with the inner layer must be the same as flattening the middle and the inner layers and flattening the outer layer with that.

```scala
fffa.flatten.flatten == fffa.map(_.flatten).flatten
```

This looks slightly different than the signatures of the other associative laws we have seen because of our need to map over the value to flatten it but it is conceptually the same as the associative laws for the `Associative`, `AssociativeBoth`, and `AssociativeEither` functional abstractions.

The `flatten` operator corresponds to running the outer value to produce a new value and then running that value.

We can see this by recognizing that at this point we know that `F` is covariant so it produces values of type `A`. We start with a value `F[F[A]]`, which is essentially a producer of producers of values of type `A`.

So to get a final value `F[A]` that just produces `A` values we are going to need to run the outer `F` value to get those producers of `A` values, then run those producers of `A` values to get actual `A` values, and finally put the whole thing back together in the structure of `F`.

This turns out to be extremely powerful.

To see what this looks like let's start as we have before with `ZIO`.

A common mistake we might make when getting started with `ZIO` is accidentally creating a nested `ZIO` workflow like this:

```scala

val greet: ZIO[Random, Nothing, ZIO[Console, IOException, Unit]] =
  Random.nextIntBounded(100).map { n =>
    Console.printLine(s"The number is $n")
  }
// greet: ZIO[Random, Nothing, ZIO[Console, IOException, Unit]] = FlatMap(
//   trace = "repl.MdocSession.MdocApp.greet(associativeflatten.md:23)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp.greet(associativeflatten.md:23)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@5b6966c1
//   ),
//   successK = zio.ZIO$$Lambda$17101/0x00000008049d5040@40e884fc
// )
```

This is very understandable.

We want to perform a second `ZIO` workflow, in this case printing to the console, based on the result of the first workflow. So we use the `map` operator, which is described by the `Covariant` abstraction, to get the success value of the first workflow and transform it with a function.

It is worth pointing out here that `Covariant` is the only abstraction in ZIO Prelude that describes the ability to transform the output of a covariant type with a function. So based on the structure described by the other functional abstractions in ZIO Prelude there is nothing we could do other than use `map` to make progress towards implementing this.

However, we are now faced with a problem. We have a `ZIO` workflow that produces another `ZIO` workflow instead of producing a value. This is normally not what we want.

This workflow is going to be difficult to compose with other workflows. If we want to get the result of the second workflow we are now going to have to use the `map` operator twice just to access the inner value.

In addition, running this workflow will probably not do what we intend, because it will just generate a random number and then return another workflow. We will not see anything printed to the console.

Conceptually what we would like to do is run this workflow to produce its result, which is another workflow that will actually print something to the console, and then run that workflow. However, we don't want to have to run these nested workflows ourselves and we want to be able to use this workflow in an ergonomic way without going through these multiple layers of nesting.

This is exactly what the `flatten` operator does for us. It takes a `ZIO` workflow that produces another workflow and flattens it down to a single workflow that conceptually runs the first workflow and then runs the resulting workflow.

Let's see how we can use the `flatten` operator to fix our code from above.

```scala
val greet: ZIO[Random with Console, IOException, Unit] =
  Random.nextIntBounded(100).map { n =>
    Console.printLine(s"The number is $n")
  }.flatten
// greet: ZIO[Random with Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp.<local MdocApp>.greet(associativeflatten.md:35)",
//   first = FlatMap(
//     trace = "repl.MdocSession.MdocApp.<local MdocApp>.greet(associativeflatten.md:33)",
//     first = Stateful(
//       trace = "repl.MdocSession.MdocApp.<local MdocApp>.greet(associativeflatten.md:33)",
//       onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@175ec8
//     ),
//     successK = zio.ZIO$$Lambda$17101/0x00000008049d5040@50a19db9
//   ),
//   successK = <function1>
// )
```

Now we have eliminated the nested workflows. This workflow will do exactly what we want, first generating a random number and then printing it to the console.

Of course if we are working with `ZIO` we quickly learn to use the `flatMap` operator, which just combines `map` and `flatten`.

```scala
val greet: ZIO[Random with Console, IOException, Unit] =
  Random.nextIntBounded(100).flatMap { n =>
    Console.printLine(s"The number is $n")
  }
// greet: ZIO[Random with Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp.<local MdocApp>.greet(associativeflatten.md:43)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp.<local MdocApp>.greet(associativeflatten.md:43)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@604c9635
//   ),
//   successK = <function1>
// )
```

The `flatMap` operator is extremely convenient, especially in combination with Scala's syntax for for comprehensions. However, we are now in a position to see that there isn't really anything special about the `flatMap` operator itself.

The `flatMap` operator is just a combination of `map` and `flatten` and `map` is a very basic operator defined on all covariant data types. The real work here is in the `flatten` operator.

The `flatten` operator has to run one value to produce its output then run that output to produce a result.

In the case of `ZIO` this is at least conceptually relatively simple because a `ZIO` workflow always succeeds with exactly one value. So the interpretation of `flatten` is just to run the first workflow to produce its result and then if it is successful run that workflow.

In other cases the interpretation can be more complex.

For example, consider `ZStream`, which can succeed with zero or more values. In that case the interpretation of flattening a stream of streams is to concatenate the inner streams into a single stream.

Many other data types also support `flatten` operators, all of which have a similar interpretation of running the outer value to produce an inner value and then running the inner value.

Data types that model failure, such as `Either` and `Option`, are one example of this. Flattening a nested `Either` or `Option` corresponds to running the outer value and if it is a success running the inner value but otherwise failing immediately.

```scala
def flatten[E, A](eea: Either[E, Either[E, A]]): Either[E, A] =
  eea match {
    case Left(e)   => Left(e)
    case Right(ea) => ea
  }
```

```scala
def flatten[A](ooa: Option[Option[A]]): Option[A] =
  ooa match {
    case None     => None
    case Some(oa) => oa
  }
```

Collection types also have a `flatten` operator with a natural interpretation. If we think of running a collection as producing a set of values then flattening a collection corresponds to running a collection of collections and then running each of those collections to produce its values.

```scala
def flatten[A](aas: Chunk[Chunk[A]]): Chunk[A] =
  aas.foldLeft[Chunk[A]](Chunk.empty)(_ ++ _)
```

Notice the similarity here to the interpretation of `flatten` for `ZStream`, although `ZStream` can be potentially effectual and resourceful.

The `flatten` operator is so powerful because it allows the next value to be produced by running the first value.

To see this, consider the following two `ZIO` workflows.

```scala
val helloAssociativeBoth: ZIO[Console, IOException, Unit] =
  console.putStrLn("What's your name?") *>
    console.readLine *>
    console.putStrLn(s"Hello! Welcome to ZIO!")

val helloAssociativeFlatten: ZIO[Console, IOException, Unit] =
  for {
    _    <- console.putStrLn("What's your name?") *>
    name <- console.readLine *>
    _    <- console.putStrLn(s"Hello $name! Welcome to ZIO!")
  } yield ()
```

The first workflow is written using only the structure described by the `Covariant` and `AssociativeBoth` abstractions.

We can use these abstractions to describe running one workflow and then running another workflow. So the first workflow will ask us for our name, then read our name, and then say hello.

However, notice that in the first workflow we can't greet the user by name. Although we can describe printing to the console after reading from the console, we can't actually use the value we read from the console in printing to the console.

Workflows described solely in terms of the `Covariant` and `AssociativeBoth` abstractions are static like this. We can compose them together but later workflows can't use information produced by prior ones.

This can have significant advantages in terms of opportunities for optimization. The first workflow can potentially be analyzed before being run, allowing us to determine what kinds of things it will do or a more efficient way of running it.

The second workflow is written using the structure described by the `AssociativeFlatten` abstraction.

Now we can describe a more powerful notion of running one workflow that produces a workflow and then running that workflow. Now we are able not only to print a greeting after reading from the console but to use the value we read from the console in the greeting.

Workflows written with the structure of `AssociativeFlatten` are much more flexible and allow building workflows of arbitrary complexity. This is why the `flatMap` operator is so useful when working with `ZIO`.

At the same time, this power limits our ability to introspect on workflows written using the `flatten` operator.

As we saw from our discussion at the beginning, we have to run the outer value to produce the inner value. So we don't know what the next workflow will be until we actually run the current workflow.

There are some ways we can increase the flexibility of workflows that don't use the structure provided by `AssociativeFlatten`. 

For example the `AssociativeEither` abstraction allows us to describe a concept of failure.

So we could express some limited degree of conditionality, for example reading from the console again if reading the first time failed. However that would still not allow us to actually use the result from the previous workflow in the current one.

In some cases this more limited functionality can be acceptable. For example in parsing it can get us quite far, and as discussed above there are benefits to this approach.

But for general purposes programming the flexibility provided by the `flatten` operator is necessary to support the variety of things we may want to describe with our workflows.

As this discussion has hopefully made clear, `AssociativeFlatten` is a very important functional abstraction that gives us the ability to run a value that produces another value and then run that value, allowing us to describe workflows in a way that is extremely flexible.

If you are working with existing data types in ZIO or the Scala standard library they probably already define `flatMap` and `flatten`.

There are some operators defined on the `ForEach` and `NonEmptyForEach` abstractions that work with data types that have instances of `AssociativeFlatten` or `IdentityFlatten`. These can be helpful to do things like fold over a collection with the possibility of failure.

When defining your own parameterized data types the `flatMap` operator is so useful that you should be thinking about what an implementation of this abstraction would look like for your own data type. Or you may intentionally not want to expose it to allow for more static analysis, but that is definitely something worth thinking about.

For all the reasons discussed above the `AssociativeFlatten` and `IdentityFlatten` abstractions are also very important in writing generic code to express values that depend on other values in a parameterized context.

---

## CommutativeBoth

`CommutativeBoth[F]` describes a way of combining two values `F[A]` and `F[B]` into a value `F[(A, B)]` that is commutative.

Its signature is:

```scala
trait AssociativeBoth[F[_]] {
  def both[A, B](fa: => F[A], fb: => F[B]): F[(A, B)]
}

trait CommutativeBoth[F[_]] extends AssociativeBoth[F]
```

If we import `zio.prelude._` we can use the `zipPar` operator or its symbolic alias `<&>` to combine any two values of a parameterized type for which a `CommutativeBoth` instance exists in a commutative way.

The `both` operator must be commutative, so if we combine `fa` and `fb` that must be the same as combining `fb` and `fa`. After rearranging the order of tuples:

```scala
fa <&> fb === fb <&> fa
```

This is the same law as defined for the `Commutative` functional abstraction for concrete types, just lifted into the context of parameterized types.

As the name of the `zipPar` operator implies, the `both` operator of the `CommutativeBoth` abstraction corresponds to running the first value and running the second value in parallel.

We can see this because to produce a tuple `(A, B)` we need to run both the left value and the right value and to do it in a way that order doesn't matter we need to run the left value and the right value in parallel. This doesn't necessarily mean we are actually forking separate fibers to run the left and right values but it does mean that we run the left and right values in a way that the result does not depend on the order in which they are run.

To get a better sense of what it means for a combining operation on parameterized types to be commutative, let's start with looking at why the `zip` operator on `ZIO` is not commutative.

```scala

val helloZIO: ZIO[Console, IOException, Unit] =
  Console.printLine("Hello") <*> Console.printLine("ZIO")
// helloZIO: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.helloZIO(commutativeboth.md:29)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.helloZIO(commutativeboth.md:29)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@650a72a5
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@2efa6013
// )

val zioHello: ZIO[Console, IOException, Unit] =
  Console.printLine("ZIO") <*> Console.printLine("Hello")
// zioHello: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.zioHello(commutativeboth.md:33)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.zioHello(commutativeboth.md:33)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@49299a49
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@34ae8187
// )
```

If the `zip` operator was commutative these two `ZIO` workflows would be the same. But they obviously are not.

The first will print `Hello` to the console on one line followed by `ZIO` on the next whereas the second will print `ZIO` to the console followed by `Hello`. We can easily observe this by running the program ourselves or using a test implementation of the `Console` service like `TestConsole` from ZIO Test.

We can also see this in the context of failures.

```scala
val failZIO: ZIO[Console, IOException, Unit] =
  ZIO.fail(new IOException("Fail")) <*> Console.printLine("ZIO")
// failZIO: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.failZIO(commutativeboth.md:40)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.failZIO(commutativeboth.md:40)",
//     onState = zio.ZIO$$$Lambda$17057/0x0000000804972840@7eb0fd9d
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@3ed2f975
// )

val zioFail: ZIO[Console, IOException, (Unit, Unit)] =
  Console.printLine("ZIO") <*> ZIO.fail(new IOException("Fail"))
// zioFail: ZIO[Console, IOException, (Unit, Unit)] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.zioFail(commutativeboth.md:44)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.zioFail(commutativeboth.md:44)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@459d3c86
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@6c2d7c11
// )
```

Again if the `zip` operator was commutative these two `ZIO` workflows should be the same but they are not. The first program will fail immediately and never print `ZIO` to the console, whereas the second will print `ZIO` to the console and then fail.

What would a commutative version of this operator be? It would have to run both the left and right values at the same time so they both had the same opportunity to run.

The `zipPar` operator on `ZIO` does just this.

```scala
val helloZIOPar: ZIO[Console, IOException, Unit] =
  Console.printLine("Hello") <&> Console.printLine("ZIO")
// helloZIOPar: ZIO[Console, IOException, Unit] = DynamicNoBox(
//   trace = "repl.MdocSession.MdocApp0.helloZIOPar(commutativeboth.md:51)",
//   update = 1L,
//   f = zio.ZIO$$Lambda$17117/0x00000008049eb440@a4bbe1c
// )
```

Now `Hello` and `World` will be printed to the console at the same time. For any given execution one will appear before the other but they each have the same opportunity to appear first and both this and the version with the order reversed describe the same `ZIO` workflow.

This interpretation of `zipPar` as running the left and right values on separate fibers is the most natural one but not the only one. The left and right values can be run in parallel as long as the result of running one does not depend on the result of running the other and information from both has equal opportunity to be included in the result.

To see this, let's look at a slightly simplified version of the `Validation` data type from ZIO Prelude and how we can define the `zipPar` operator for it.

```scala

sealed trait Validation[+E, +A] { self =>

  def zipPar[E1 >: E, B](that: Validation[E1, B]): Validation[E1, (A, B)] =
    (self, that) match {
      case (Success(a), Success(b))    => Success((a, b))
      case (Success(_), Failure(e1s))  => Failure(e1s)
      case (Failure(es), Success(_))   => Failure(es)
      case (Failure(es), Failure(e1s)) => Failure(es ++ e1s)
    }
}

object Validation {
  case class Success[+A](value: A) extends Validation[Nothing, A]
  case class Failure[+E](errors: NonEmptyChunk[E]) extends Validation[E, Nothing]
}
```

We are not forking any fibers here, but notice how in our implementation of `zipPar` we are matching on the left and right sides at the same time and preserving the information from both errors in the case where both fail. As long as we implement equality in a way that does not depend on the order of errors, combining two `Validation` values with `zipPar` will always give the same result regardless of the order in which we combine them.

Another interpretation of what it means to run two values in parallel comes from collections. We can combine values from two collections in parallel by zipping them together pairwise.

```scala

implicit val ListCommutativeBoth: CommutativeBoth[List] =
  new CommutativeBoth[List] {
    def both[A, B](as: => List[A], bs: => List[B]): List[(A, B)] =
      as.zip(bs)
  }
// ListCommutativeBoth: CommutativeBoth[List] = repl.MdocSession$MdocApp0$$anon$1@4244c06c
```

Again we are not forking any fibers here but values from the left list and the right list have an equal opportunity to be included in the zipped list.

We can see this if we zip together two lists of different lengths.

```scala
val left: List[Int] =
  List(1, 2, 3)
// left: List[Int] = List(1, 2, 3)

val right: List[Int] =
  List(4, 5, 6, 7, 8)
// right: List[Int] = List(4, 5, 6, 7, 8)

val leftRight: List[(Int, Int)] =
  left.zip(right)
// leftRight: List[(Int, Int)] = List((1, 4), (2, 5), (3, 6))

val rightLeft: List[(Int, Int)] =
  right.zip(left)
// rightLeft: List[(Int, Int)] = List((4, 1), (5, 2), (6, 3))
```

The zipped lists are identical after rearranging the order of tuples.

Some instances of `CommutativeBoth` actually combine these interpretations. For example the `CommutativeBoth` instance for `ZStream` is implemented in terms of the `zip` operator on `ZStream`, which creates a new stream that pulls values from the left and the right streams pairwise, evaluating effects in the left and right streams on separate fibers.

```scala

def both[R, E, A, B](left: => ZStream[R, E, A], right: ZStream[R, E, B]): ZStream[R, E, (A, B)] =
  left.zip(right)
```

Just like the `AssociativeBoth` abstraction, `CommutativeBoth` is not limited to covariant types. For example we could send inputs to multiple sinks in parallel, having each sink run on a separate fiber.

When the parameterized type is covariant the `zipWithPar` operator is also defined on it.

```scala
def zipWithPar[F[+_]: CommutativeBoth : Covariant, A, B, C](
  fa: => F[A],
  fb: => F[B]
)(f: (A, B) => C): F[C] =
  fa.zipPar(fb).map(f.tupled)
```

This is just like the `zipWith` operator defined on data types that have an `AssociativeBoth` instance except it runs the left and right values in parallel instead of running the left value and then the right value. It is very useful when you want to do two things in parallel and then combine the results.

A version of this is available for contravariant data types called `bothWithPar`.

```scala
def bothWithPar[F[-_]: CommutativeBoth : Contravariant, A, B, C](
  fa: => F[A],
  fb: => F[B]
)(f: C => (A, B)): F[C] =
  fa.zipPar(fb).contramap(f)
```

The `bothWithPar` operator splits a data type `C` up into two parts `A` and `B`, running the left value with `A` and the right value with `B` in parallel.

The `CommutativeBoth` functional abstraction, like the `AssociativeBoth` abstraction, describes a basic way of combining two data types, this time by running them in parallel instead of running the left one and then the right one. The `zipPar` operator, or more domain specific names for it for certain basic types, is also very practically useful and we saw that it corresponded to common operators on these data types that we were probably already familiar with.

For this reason, however, the `CommutativeBoth` abstraction may be less useful if you are working with data types from ZIO or the Scala standard library. They generally already implement this functionality directly, for example in the `zipPar` operator on `ZIO` and the `zip` operator on `ZStream`.

This abstraction is more likely to be useful when you are defining your own data type. If your data type supports a way of combining that is commutative, such as the ones we have discussed here, then defining a `CommutativeBoth` instance will let you express that and work with other operators that are defined on data types with `CommutativeBoth` instances.

Finally, this abstraction is likely to be valuable for those writing generic code to describe running values in parallel. As with the `AssociativeBoth` abstraction the ability to generalize over covariant, contravariant, and invariant data types is likely to be particularly valuable here.

---

## CommutativeEither

`CommutativeEither[F]` describes a way of combining two parameterized types `F[A]` and `F[B]` that is commutative.

Its signature is:

```scala
trait AssociativeEither[F[_]] {
  def either(fa: => F[A], fb: => F[B]): F[Either[A, B]]
}

trait CommutativeEither[F[_]] extends AssociativeEither[F]
```

If we import `zio.prelude._` we can use the `orElseEitherPar` operator or its symbolic alias `<|>` to combine two values of a parameterized type that has a `CommutativeEither` instance defined for it in a commutative way.

The `either` operator must be commutative. So after rearranging the order of `Either` values the following property must hold:

```scala
fa <|> fb === fb <|> fa
```

This is the same as the law defined by the `Commutative` abstraction for concrete types, but lifted into the context of parameterized types.

The `either` operator runs the left value and the right value in parallel, returning the result from either the left value or the right value.

We can see this by noting that the requirement that the `either` operator is commutative places significant limitations on our implementation. We must choose on some basis between the left and right values but we can't run one and then if it fails run the other because that would in general not be commutative.

`ZIO` provides a good example of why running the left value and then running the right value if it fails is not commutative and what a commutative operator would look like.

```scala

val helloZIO: ZIO[Console, IOException, Either[Unit, Unit]] =
  Console.printLine("Hello").orElseEither(Console.printLine("ZIO"))
// helloZIO: ZIO[Console, IOException, Either[Unit, Unit]] = FoldZIO(
//   trace = "repl.MdocSession.MdocApp.helloZIO(commutativeeither.md:15)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp.helloZIO(commutativeeither.md:15)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@7a648899
//   ),
//   successK = zio.ZIO$$$Lambda$17041/0x0000000804965040@49c1fa09,
//   failureK = zio.ZIO$$Lambda$17130/0x0000000804a15040@51fa8201
// )

val zioHello: ZIO[Console, IOException, Either[Unit, Unit]] =
  Console.printLine("ZIO").orElseEither(Console.printLine("Hello"))
// zioHello: ZIO[Console, IOException, Either[Unit, Unit]] = FoldZIO(
//   trace = "repl.MdocSession.MdocApp.zioHello(commutativeeither.md:19)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp.zioHello(commutativeeither.md:19)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@42a2c962
//   ),
//   successK = zio.ZIO$$$Lambda$17041/0x0000000804965040@49c1fa09,
//   failureK = zio.ZIO$$Lambda$17130/0x0000000804a15040@47cda1bd
// )
```

If the `orElseEither` operator was commutative then these two programs should be the same. But they obviously are not.

The first program will print `Hello` to the console on one line but never print `ZIO`. Since the left `ZIO` workflow was successful the right `ZIO` workflow will never be run.

Similarly the second program will print `ZIO` to the console on one line but will never print `Hello`.

We have to return either of the values but the order can't matter. The `raceEither` operator on `ZIO` does just that.

It runs both effects concurrently, returning the first one to complete successfully and safely interrupting the other. With it we can make our example above satisfy the commutative law.

```scala

val helloZIO: ZIO[Console, IOException, Either[Unit, Unit]] =
  Console.printLine("Hello").raceEither(Console.printLine("ZIO"))
// helloZIO: ZIO[Console, IOException, Either[Unit, Unit]] = Stateful(
//   trace = "repl.MdocSession.MdocApp.<local MdocApp>.helloZIO(commutativeeither.md:33)",
//   onState = zio.ZIO$$Lambda$17135/0x0000000804a28040@7f183625
// )

val zioHello: ZIO[Console, IOException, Either[Unit, Unit]] =
  Console.printLine("ZIO").raceEither(Console.printLine("Hello"))
// zioHello: ZIO[Console, IOException, Either[Unit, Unit]] = Stateful(
//   trace = "repl.MdocSession.MdocApp.<local MdocApp>.zioHello(commutativeeither.md:37)",
//   onState = zio.ZIO$$Lambda$17135/0x0000000804a28040@47b0e781
// )
```

Now both `ZIO` workflows will begin execution at the same time.

For an individual execution of this workflow either `Hello` or `ZIO` may be printed to the console, or possibly both if the loser is not interrupted before it runs. But both the left and right workflows have an equal opportunity to be run and we can say that both `helloZIO` and `zioHello` describe the same program.

We can also see this interpretation of `orElseEitherPar` in terms of racing in its implementation for `ZStream`.

```scala

def orElseEitherPar[R, E, A, B](left: => ZStream[R, E, A], right: => ZStream[R, E, B]): ZStream[R, E, Either[A, B]] =
  left.mergeEither(right)
```

We still run both streams concurrently but now instead of cancelling the loser we just emit elements from either stream whenever they are available. This is commutative because whatever order we merge the streams they will still be evaluated concurrently.

The difference in the implementations here reflects the fact that a `ZIO` always succeeds with exactly one value, whereas a `ZStream` can succeed with zero or more values.

So in the case of `ZIO` if one fiber completes execution successfully first we have nothing to do with the value from the other and so just interrupt it. In contrast, a `ZStream` can succeed with multiple values so we can just emit the values from either stream as they are available.

In many cases the implementation of the `orElseEitherPar` will involve actual concurrency but it does not need to.

To see this, consider the following instance of the `CommutativeEither` abstraction for `Set`.

```scala

implicit val SetCommutativeEither: CommutativeEither[Set] =
  new CommutativeEither[Set] {
    def either[A, B](left: => Set[A], right: => Set[B]): Set[Either[A, B]] =
      left.map(Left(_)) ++ right.map(Right(_))
  }
// SetCommutativeEither: CommutativeEither[Set] = repl.MdocSession$MdocApp$$anon$1@36d200f
```

In this case there is no actual concurrency going on but we are combining the left `Set` and right `Set`, wrapping their values in `Left` and `Right` respectively.

This is like the interpretation of `orElseEither` as concatenating two collections except normally concatenation would not be commutative since order matters. In the case of a `Set` the order of elements doesn't matter.

We can think of this as running each `Set` to get its elements until it fails by being empty.

As with the other operators for describing ways of combining parameterized types, we can also define `CommutativeEither` instances for data types that are contravariant or invariant. These also often do not require any actual concurrency because the logical parallelism is handled by running either the left value or the right value depending on the input.

For example, here is the instance of `CommutativeEither` for the `Predicate` data type.

```scala
trait Predicate[-A] {
  def run(a: A): Boolean
}

object Predicate {
  implicit val PredicateCommutativeEither: CommutativeEither[Predicate] =
    new CommutativeEither[Predicate] {
      def either[A, B](left: => Predicate[A], right: => Predicate[B]): Predicate[Either[A, B]] =
        new Predicate[Either[A, B]] {
          def run(either: Either[A, B]): Boolean =
            either match {
              case Left(a) => left.run(a)
              case Right(b) => right.run(b)
            }
        }
    }
}
```

It is commutative because no matter what order we combine the two predices, we will always send `A` values to the predicate that can handle `A` values and `B` values to the predicate that can handle `B` values. If we did something different with the `A` and `B` values other than just sending them to the predicate based on whether they were a `Left` or a `Right` then we would be at risk of violating the laws.

When a data type for which a `CommutativeEither` instance is defined is also covariant we can use the `orElsePar` operator as well.

```scala
def orElsePar[F[+_]: CommutativeEither : Covariant, A](left: => F[A], right: => F[A]): F[A] =
  left.orElseEitherPar(right).map(_.merge)
```

The `orElsePar` operator just maps over the output of the `orElseEitherPar` operator and merges the `Either` values when both the `left` and `right` values are parameterized on the same types.

This is quite useful and typically has more domain specific names. For example this is the `race` operator on `ZIO` and the `merge` operator on `ZStream`. We can think of this as just like `orElseEither` but eliminating the `Either` when the types that the left and right values are parameterized on are the same.

There is also an additional operator we can use for contravariant types called `eitherWithPar`.

```scala
def orElseEitherWithPar[F[-_]: CommutativeEither : Contravariant, A, B, C](
  left: => F[A],
  right: => F[B]
)(f: C => Either[A, B]): F[C] =
  left.orElseEitherPar(right).contramap(f)
```

This is like the `orElseEitherWith` defined on data types that have an `AssociativeEither` and `Contravariant` instance except it guarantees that the `F[A]` and `F[B]` values will be combined in a way that is commutative in addition to being associative.

The `CommutativeEither` functional abstraction, along with the `CommutativeBoth` abstraction, describe the two fundamental ways of combining values in parallel. We can either run both values and return both their results, as in the `zipPar` operator on `ZIO`, or we can run both values and return either of their results, as in the `raceEither` operator on `ZIO`.

As with some of the other functional abstractions for parameterized types, `CommutativeEither` tends to be most useful when you are defining your own parameterized types or writing generic code. If you are working with existing data types like `ZIO` or `ZStream` the operators described by this abstraction tend to already be available to you with more domain specific names.

If you are defining your own data type then exploring whether there is a way to run both values in parallel to produce either result can be a useful exercise.

If your data type describes some kind of effect then there is often a very obvious interpretation in terms of concurrency. If not it may require more thought.

Defining a `CommutativeEither` instance, if it exists, is a good practice to allow you to use other operators in ZIO Prelude that are defined for data types with `CommutativeEither` instances. It will also allow you to test that your implementation satisfies the expected laws, which can be tricky with some of these more complex types.

Finally, if you are writing generic code the `CommutativeBoth` and `CommutativeEither` abstractions describe the fundamental binary operators for combining values in parallel. So if you want to describe parallel or concurrent operators in a generic way you will definitely want to use these abstractions.

---

## Contravariant

`Contravariant` describes a parameterized type `F[A]` that potentially consumes but never produces `A` values.

Its signature is:

```scala
trait Invariant[F[_]] {
  def invmap[A, B](f: A <=> B): F[A] <=> F[B]
}

trait Contravariant[F[-_]] extends Invariant[F] {
  def contramap[A, B](f: B => A): F[A] => F[B]
  final def invmap[A, B](f: A <=> B): F[A] <=> F[B] =
    Equivalence(contramap(f.from), contramap(f.to))
}

type <=>[A, B] = Equivalence[A, B]

case class Equivalence[A, B](to: A => B, from: B => A)
```

The `contramap` operator says we can lift a function `B => A` to a function `F[A]` to `F[B]`. If we import `zio.prelude._` we can use `contramap` to transform a `F[A]` to a `F[B]` with a function `B => A`.

Notice that the arrows go in the opposite direction here. To transform a `F[A]` into a `F[B]` with the `contramap` operator we provide a function `B => A` rather than a function `A => B` like with the `map` operator. 

This can be a little counterintuitive because we are used to primarily working with data types that produce values and transforming their outputs. We will build a better sense for the `contramap` operator later in this section.

The other thing to notice here is the `-` that appears in brackets in the definition of `Contravariant`. This tells the Scala compiler that `F` is contravariant with respect to the `A` type parameter.

Doing this improves type inference because the Scala compiler knows that if `A` is a subtype of `B` then an `F[B]` is a subtype of an `F[A]`, since an `F[B]` can accept `B` inputs and every `A` is a `B`. It also allows the compiler to check that `A` really does only appear as an input to `F`.

Other functional programming libraries don't take advantage of contravariance here and so have to define a `narrow` operator, which essentially forces users to do this type casting manually.

The law is that the lifting of the function `f` in `contramap` can transform `B` values into `A` values but cannot otherwise change the structure of `F`, so using `contramap` with the identity function is an identity and separately using `contrmap` with two functions is the same as doing it with the composition of those functions.

```scala
fa.contramap(identity) === fa
fa.contramap(f).contramap(g) === fa.contramap(f.compose(g)))
```

Examples of data types that are contravariant include functions with respect to their inputs, `ZIO` with respect to its environment type, and `ZSink` with respect to its input type.

To get a better sense of what it means for a data type to be contravariant let's look at a `JSONCodec`.

```scala
trait JsonCodec[A] {
  def decode(json: String): Either[String, A]
  def encode(a: A): String
}
```

This data type doesn't have either a `+` or a `-` before the `A` type parameter, indicating that it is invariant with respect to the `A` type parameter. If we try to make `JsonCodec` contravariant by adding a `-` before the `A` type parameter we get a compilation error telling us that `A` appears in covariant position in the `decode` operator.

This is accurate because `A` does indeed appear as an output of the `decode` operator whereas it is only supposed to appear as an input to a contravariant type. To fix this we need to break the `JsonCodec` up into separate `JsonDecoder` and `JsonEncoder` types that are covariant and contravariant respectively.

```scala
trait JsonDecoder[+A] {
  def decode(json: String): Either[String, A]
}

trait JsonEncoder[-A] {
  def encode(a: A): String
}

trait JsonCodec[A] extends JsonDecoder[A] with JsonEncoder[A]
```

Now we can define a `Contravariant` instance for the `JsonEncoder`.

```scala
trait JsonEncoder[-A] { self =>
  def encode(a: A): String
  def contramap[B](f: B => A): JsonEncoder[B] =
    new JsonEncoder[B] {
      def encode(b: B): String =
        self.encode(f(b))
    }
}

object JsonEncoder {
  implicit val JsonEncoderContravariant: Contravariant[JsonEncoder] =
    new Contravariant[JsonEncoder] {
      def contramap[A, B](f: B => A): JsonEncoder[A] => JsonEncoder[B] =
        jsonEncoder => jsonEncoder.contramap(f)
    }
}
```

Let's think about what this means.

A `JsonEncoder` is something that knows how to encode values of type `A`. It says we can give it any value of type `A` and it will encode it.

The `contramap` operator says if we have a function `B => A` we can transform a `JsonEncode[A]` into a `JsonEncoder[B]`. We can do that by taking any `B` values and transforming them into `A` values with the function `f` before sending them to the original encoder.

The pattern works the same way for any contravariant type. The `contramap` operator lets us adapt the inputs to the data type with a function, for example transforming a sink that writes bytes to a file to a sink that writes strings to a file by providing a function to transform strings to bytes.

While we may be less familiar with it, the `contramap` operator is quite useful for working with contravariant types and lets us "work backwards" from the input type we need to the input type we have. So it can be useful to implement a `Contravariant` instance for our own data types just for that.

In addition, some operators in ZIO Prelude are only defined on data types that have a `Contravariant` instance along with instances of one or more other functional abstractions. So it is useful to define instances of all functional abstractions that are applicable for your own data types so that you can use these operators when you need them.

Finally, if you are writing your own generic code in terms of the abstractions in ZIO Prelude a `Contravariant` instance can be important to define certain classes of operators.

---

## Covariant

`Covariant[F]` describes a parameterized type `F[A]` that potentially produces but never consumes `A` values.

Its signature is:

```scala
trait Invariant[F[_]] {
  def invmap[A, B](f: A <=> B): F[A] <=> F[B]
}

trait Covariant[F[+_]] extends Invariant[F] {
  def map[A, B](f: A => B): F[A] => F[B]
  final def invmap[A, B](f: A <=> B): F[A] <=> F[B] =
    Equivalence(map(f.to), map(f.from))
}

type <=>[A, B] = Equivalence[A, B]

case class Equivalence[A, B](to: A => B, from: B => A)
```

The `map` operator says we can "lift" a function `A => B` to a function `F[A] => F[B]`. If we import `zio.prelude._` we can use the `map` operator to transform an `F[A]` into an `F[B]` with a function `A => B`.

The law is that the lifting of this function can transform `A` values into `B` values but cannot otherwise change the structure of `F`, so using `map` with the identity function is an identity and separately using `map` with two functions is the same as doing it with the composition of those functions.

```scala
fa.map(identity) === fa
fa.map(f).map(g) === fa.map(f.andThen(g))
```

Data types that are covariant may either contain zero or more existing `A` values, such as a `Chunk`, or potentially produce zero or more `A` values at some point in the future, such as a `ZIO`.

In the definition of `Covariant` above you may notice that a `+` appears before the `_`. This tells the Scala compiler that the parameterized type is covariant with respect to this type parameter.

This improves type inference because it allows the Scala compiler to automatically widen an `F[A]` to an `F[B]` if `A` is a subtype of `B` because a data type that outputs `A` values also outputs `B` values by definition. It also allows the Scala compiler to check for us that types for which we define a `Covariant` instance really are covariant with respect to their type parameters.

Other functional programming libraries do not take advantage of Scala's support for variance here and so have to resort to a `widen` operator that essentially amounts to mapping with the identity function.

Often data types that appear to be invariant are actually versions of more polymorphic data types that are covariant with respect to one or more of their type parameters. Generalizing these data types can lead to improved API design and allow defining additional operators.

Let's see these ideas in action with the `JsonCodec` data type.

```scala
trait JsonCodec[A] {
  def decode(json: String): Either[String, A]
  def encode(a: A): String
}
```

The `JsonCodec` data type is naturally invariant in the `A` type parameter because `A` appears as both an input to `encode` and an output from `decode`. As a result, the `A` type parameter appears in the definition of `JsonCodec` without a `+` or `-`.

Let's try adding a `+` before the `A` type parameter to indicate that `JsonCodec` is covariant with respect to this type parameter.

```scala
trait JsonCodec[+A] {
  def decode(json: String): Either[String, A]
  def encode(a: A): String
}
// error: covariant type A occurs in contravariant position in type A of value a
//   def encode(a: A): String
//              ^^^^
```

This code does not compile!

The Scala compiler helpfully informs us that the covariant type `A` appears in contravariant position in the `encode` operator. Translating slightly, the Scala compiler is telling us that we are not honoring the guarantee of `A` being covariant that `A` is an output but never an input because `A` appears as an input to `encode`.

Whenever we have to make a type invariant because `A` is both an input and an output we should ask ourselves whether it is possible to split that up. We can do that either by using different type parameters for the input and output or by refactoring to create separate interfaces where `A` only appears as an input or output.

Here we will take the second approach and split the `JsonCodec` up into a `JsonDecoder` and a `JsonEncoder`.

```scala
trait JsonDecoder[+A] {
  def decode(json: String): Either[String, A]
}

trait JsonEncoder[-A] {
  def encode(a: A): String
}

trait JsonCodec[A] extends JsonDecoder[A] with JsonEncoder[A]
```

Now `A` only appears as an output of the `JsonEncoder` and only appears as an input of `JsonDecoder` and we can make `A` covariant by using the `+` before the type parameter. We can also make `JsonEncoder` contravariant, but we will defer discussion of that to the section on the `Contravariant` functional abstraction.

With the `JsonDecoder` and `JsonEncoder` separated out, we can now define a `Covariant` instance for the `JsonDecoder`.

```scala
trait JsonDecoder[+A] { self =>
  def decode(json: String): Either[String, A]
  def map[B](f: A => B): JsonDecoder[B] =
    new JsonDecoder[B] {
      def decode(json: String): Either[String, B] =
        self.decode(json).map(f)
    }
}

object JsonDecoder {
  implicit val JsonDecoderCovariant: Covariant[JsonDecoder] =
    new Covariant[JsonDecoder] {
      def map[A, B](f: A => B): JsonDecoder[A] => JsonDecoder[B] =
        jsonDecoder => jsonDecoder.map(f)
    }
}
```

In addition to being able to define a `Covariant` instance, we have improved our API design by recognizing that our definition of `JsonCodec` was actually combining two things that are conceptually distinct. Now users can work with just a `Decoder` if all they are doing is reading data and it is easier to transform a decoder because we only need to provide a function and not an equivalence relationship.

The `map` operator is extremely useful and you are probably familiar with it from a variety of data types in the Scala standard library and ZIO. However, the `map` operator is already defined directly on most data types that support it.

As a result, the `Covariant` abstraction tends to be useful in two ways.

First, defining a `Covariant` instance for your own data type allows it to work with other operators in ZIO Prelude that require a data type to be covariant.

There are only a few operators that are defined for a data type that only has a `Covariant` instance but there are many more that we will learn about later that require a `Covariant` instance as well as some other instance. So defining an instance of `Covariant`, as well as whatever other functional abstractions are defined for your date type, is a good practice so that you can use any of these operators for your data type if you need to.

Second, `Covariant` can be useful if you are writing your own generic code since many generic operators you may want to define will require a `Covariant` instance. In particular, having a `Covariant` instance as well as an instance of one of the abstractions that describe ways to combine parameterized types allow many interesting operators to be defined.

---

## ForEach

`ForEach[F]` describes a parameterized type `F[A]` that contains zero or more values of type `A`.

Its signature is:

```scala
trait Covariant[F[+_]] {
  def map[A, B](f: A => B): F[A] => F[B]
}

trait ForEach[F[+_]] extends Covariant[F] {
  def forEach[G[+_]: IdentityBoth: Covariant, A, B](fa: F[A])(f: A => G[B]): G[F[B]]
  final def map[A, B](f: A => B): F[A] => F[B] =
    ???
}

trait IdentityBoth[F[_]] {
  def any: F[Any]
  def both[A, B](fa: => F[A], b: => F[B]): F[(A, B)]
}
```

The `ForEach` functional abstraction builds on the `Covariant` abstraction to describe a type that contains zero or more `A` values rather than merely potentially producing `A` values at some point in the future.

For example `Chunk` has a `ForEach` instance because it contains zero or more `A` values. In contrast `ZIO` has a `Covariant` instance but not a `ForEach` instance because it is only a description of a workflow that may produce an `A` value and does not output any actual `A` value until it is run.

`ForEach` generalizes over collection types like `List`, `Map`, and `Chunk`. It also describes data types like `Option` and `Either` that contain zero or one value, which can be thought of as a special case of containing zero or more values.

The defining operator of the `ForEach` abstraction is `forEach`. It lets us take a collection of type `F[A]` and run a function `A => G[B]` for each element in the collection, returning a new collection `F[B]` in the context of `G`.

This is somewhat abstract so let's look at the `foreach` operator on `ZIO`, which is a variant of this, to get a sense of what it means.

```scala
trait ZIO[-R, +E, +A]

object ZIO {
  def foreach[R, E, A, B](as: List[A])(f: A => ZIO[R, E, B]): ZIO[R, E, List[B]] =
    ???
}
```

Here we have specialized the collection type `F` from `ForEach` to `List` and the effect type `G` from `ForEach` to `ZIO`. The interpretation of this is now much more clear.

For each element in the list we construct a new `ZIO[R, E, B]` by applying `f` to the element. Then we combine all of those `ZIO` effects into a single effect that will run each of the individual effects and collect their results in a `List`.

This pattern of doing something for each element in the collection and then collecting the results back into the original collection is common to all implementations of `ForEach`. We can get a better sense for it by looking at a simple implementation of the `foreach` operator on `ZIO`.

```scala
trait ZIO[-R, +E, +A] { self =>
  def zipWith[R1 <: R, E1 >: E, B, C](that: ZIO[R1, E1, B])(f: (A, B) => C): ZIO[R, E, C] =
    ???
}

object ZIO {
  def foreach[R, E, A, B](as: List[A])(f: A => ZIO[R, E, B]): ZIO[R, E, List[B]] =
    as.foldRight[ZIO[R, E, List[B]]](ZIO.succeed(List.empty)) { (a, zio) =>
      f(a).zipWith(zio)(_ :: _)
    }
  def succeed[A](a: => A): ZIO[Any, Nothing, A] =
    ???
}
```

In the implementation of `foreach` the `foldRight` operator on `List` tears down the original list into each of its elements, using the function `f` to produce a new `ZIO` value for each element. Then we use the `zipWith` operator on `ZIO` to combine the resulting `ZIO` values into a single `ZIO` value, putting the values back together into a `List` with the `::` constructor.

The fact that the implementation of such a complex operator can be so simple should have us thinking about how we can make this work for collection types other than `List` and types other than `ZIO`. The `ForEach` functional abstraction does just that.

The `ForEach` abstraction is parameterized on the collection type `F`, so in the implementation of `ForEach` for a collection we will know how to tear down that collection and build it back up. For example, in the implementation of `ForEach` for `List` we will know that we can tear down a `List` using `foldRight` and build it back up using `List.empty` and `::`.

The other piece of information we need is how to put a value into the type `G`, described by the `succeed` operator, and how to combine two `G` values into a single `G` value, described by the `zipWith` operator. Fortunately, that is exactly the functionality that the combination of the `Covariant` and `IdentityBoth` abstractions provide.

Recall that the `Covariant` functional abstraction defines a `map` operator that allows us to transform a `F[A]` into an `F[B]` with a function `A => B`.

```scala
trait Covariant[F[+_]] {
  def map[A, B](f: A => B): F[A] => F[B]
}
```

The `IdentityBoth` abstraction defines a `both` operator that allows us to combine an `F[A]` and an `F[B]` into an `F[(A, B)]` as well as an `any` value of type `F[Any]` that is an identity element for that operator.

```scala
trait AssociativeBoth[F[_]] {
  def both[A, B](fa: => F[A], b: => F[B]): F[(A, B)]
}

trait IdentityBoth[F[_]] extends AssociativeBoth[F] {
  def any: F[Any]

}
```

If instances of these two abstractions exist for a data type we can define the `zipWith` and `succeed` operators that we used in our implementation of `foreach` for `ZIO`.

```scala
def succeed[F[+_], A](a: => A)(implicit covariant: Covariant[F], both: IdentityBoth[F]): F[A] =
  covariant.map[Any, A](_ => a)(both.any)

def zipWith[F[+_], A, B, C](
  fa: F[A],
  fb: F[B]
)(f: (A, B) => C)(implicit covariant: Covariant[F], both: AssociativeBoth[F]): F[C] =
  covariant.map(f.tupled)(both.both(fa, fb))
```

We are now in a position to complete our generalization of the `foreach` operator on `ZIO`. Here is what the implementation of the `ForEach` instance for `List` would look like.

```scala
trait ForEach[F[+_]]{
  def forEach[G[+_]: IdentityBoth: Covariant, A, B](fa: F[A])(f: A => G[B]): G[F[B]]
}

implicit val ListForEach: ForEach[List] =
  new ForEach[List] {
    def forEach[G[+_]: IdentityBoth: Covariant, A, B](fa: List[A])(f: A => G[B]): G[List[B]] =
      fa.foldRight(succeed[G, List[B]](List.empty)) { (a, gbs) =>
        zipWith(f(a), gbs)(_ :: _)
      }
  }
// ListForEach: ForEach[List] = repl.MdocSession$MdocApp$$anon$1@2fa46f19
```

Notice how similar our implementation of the `ForEach` abstraction looks to our initial implementation of the `foreach` operator on `ZIO`. Other than using operators defined on `Covariant` and `IdentityBoth` instead of operators on `ZIO` they are identical.

We have already come a long way with our implementation of `ForEach`. We have a much better understanding now of what the `forEach` operator does and how its signature reflects the minimum possible set of constraints necessary to implement it.

We can now implement instances of `ForEach` for all collection types, like `List`, `Chunk`, and `Map`, as well as types that are like collections, such as `Option` and `Either`. In each case we can perform an effect like a `ZIO` for each element in the collection, returning a new `ZIO` that collects the results back into the original collection type.

This is already quite powerful, but the other aspect of the `ForEach` abstraction that is important is our ability to use other parameterized types for the `G` in the signature of `ForEach`. We have focused so far on `ZIO` because it is the prototypical functional effect and provides a very clear intuition of what it means to do something for each element of a collection, but we can use many other data types as well.

For example, we can use the `forEach` operator with a function that returns a `Validation` value from ZIO Prelude.

`Validation` is a data type that can either succeed with a value or fail with one or more errors. Using `forEach` with `Validation` corresponds to validating all the values in a collection, returning either a new collection of fully validated data or an accumulation of all validation errors that occurred.

```scala

case class Person(name: String, age: Int)

def validateName(name: String): Validation[String, String] =
  Validation.fromPredicateWith("Name was empty")(name)(_.nonEmpty)

def validateAge(age: Int): Validation[String, Int] =
  Validation.fromPredicateWith(s"Age $age was less than zero")(age)(_ >= 0)

def validatePerson(name: String, age: Int): Validation[String, Person] =
  Validation.validateWith(validateName(name), validateAge(age))(Person)

val validData: List[(String, Int)] =
  List(("John", 35), ("Jane", 25))
// validData: List[(String, Int)] = List(("John", 35), ("Jane", 25))

val invalidData: List[(String, Int)] =
  List(("", 35), ("John", -1))
// invalidData: List[(String, Int)] = List(("", 35), ("John", -1))

val success: Validation[String, List[Person]] =
  validData.forEach { case (name, age) => validatePerson(name, age) }
// success: Validation[String, List[Person]] = Success(
//   log = IndexedSeq(),
//   value = List(Person(name = "John", age = 35), Person(name = "Jane", age = 25))
// )

val failure: Validation[String, List[Person]] =
  invalidData.forEach { case (name, age) => validatePerson(name, age) }
// failure: Validation[String, List[Person]] = Failure(
//   log = IndexedSeq(),
//   errors = NonEmptyChunk(Name was empty, Age -1 was less than zero)
// )
```

Using `forEach` with the valid data will return a validation success containing the fully validated list of `Person("John", 35)` and `Person("Jane", 25")`. In contrast, using `forEach` with the invalid data will return a validation failure with the failures `"Name was empty"` and `"Age -1 was less than zero"`.

We can return other data types from the function we provide to `ForEach` to obtain other functionality. For example, if we use `Option` or `Either` we will get either a collection of all the successful results or the first failure to occur, rather than the accumulation of all failures as with `Validation`.

There are also a couple of specialized data types we can use in the return type of `f` that turn out to be particularly important for implementing other operators.

One of these you may have noticed from the very beginning of this section was the `Id` type.

`Id` is the parameterized type `F[A]` where `F` doesn't have any structure itself and just contains exactly one `A` value. This seems quite trivial but it is useful to adapt a function `A => B` to the type `A => G[B]` that `ForEach` is expecting.

This allows us to implement the `map` operator in terms of `forEach` and proves that the `ForEach` abstraction is an extension of the `Covariant` abstraction.

```scala
def map[F[+_]: ForEach, A, B](fa: F[A])(f: A => B): F[B] =
  Id.unwrap(fa.forEach(a => Id.wrap(f(a))))
```

Another data type that is important for implementing other operators is `State`.

The `State` data type from ZIO Prelude describes a state transition function `S => (A, S)` that takes an initial state and returns a value and an updated state. Using `ForEach` with `State` corresponds to composing all of those state transition functions into a single state transition function.

ZIO Prelude provides a specialized operator for using `forEach` with `State` called `mapAccum`.

```scala
def mapAccum[F[+_]: ForEach, S, A, B](fa: F[A])(s: S)(f: (S, A) => (S, B)): (S, F[B]) =
  ???
```

This lets us specify an initial state `S` and then in our function `f` instead of just providing a function `A => B` like in `map` we can update the state as well. The implementation of `mapAccum` takes care of putting each of these functions in the `State` data type for us and then runs the final computation for us with the initial state to produce both the final state and the final value.

This is very useful for implementing other operators because it lets us get information out of the collection type `S`, so now we can implement collection operators that don't just transform the collection but reduce it to a summary value.

To see this, let's implement the `foldLeft` operator from the Scala collection library in terms of `mapAccum`.

```scala
def foldLeft[F[+_]: ForEach, A, S](fa: F[A])(s: S)(f: (S, A) => S): S =
  mapAccum(fa)(s)((s, a) => (f(s, a), ()))._1
```

The `foldLeft` operator simply accumulates the fold state and discards the collection.

This is very powerful because it means that we can implement almost every collection operator in terms of `ForEach`.

This includes any collection operator that reduces a collection to some other value, like `count`, `exists`, `find`, `foldLeft`, `foldRight`, `forall`, `groupBy`, `isEmpty`, `product`, `size`, `sum`, and `toList`.

It also includes any operator that modifies the values of a collection while maintaining the "shape" of the collection such as `zipWithIndex`.

The only collection operators we can't implement in terms of `ForEach` are those that change the shape of the collection like appending a new element to the collection. We don't know enough about the structure of the data type to know what that would mean or even if it would be well defined.

For example, the data type might be a tree that does not have a well defined notion of adding an element without specifying where it should be added. Or it might be a data type like `Option` or `Either` that doesn't support appending at all.

In addition to the standard Scala collection operators, `ForEach` allows us to define additional operators that take advantage of the functional abstractions and data structure in ZIO Prelude.

One variant of `forEach` that you may be familiar with is `flip`.

```scala
def flip[F[+_]: ForEach, G[+_]: IdentityBoth : Covariant, S, A, B](fga: F[G[A]]): G[F[A]] =
  fga.forEach(identity)
```

This is the generalized version of the `collectAll` operator on `ZIO`.

```scala

def collectAll[R, E, A](as: List[ZIO[R, E, A]]): ZIO[R, E, List[A]] =
  ZIO.foreach(as)(identity)
```

This operator already takes a list of `ZIO` values so all it has to do is combine them into a single `ZIO` value and put the results back together into a list.

Another particularly useful operator is `foldMap`, which lets us reduce a collection to a summary value by mapping each value to a value for which an `Identity` instance is defined and combining those values with the `combine` operator.

```scala
def foldMap[F[+_]: ForEach, A, B: Identity](as: F[A])(f: A => B): B =
  foldLeft(as)(Identity[B].identity)((b, a) => b <> f(a))
```

This is a very nice operator that lets us express a variety of ways of reducing a collection to a summary value in a very concise way.

For example we could implement `sum` in terms of `foldMap` like this:

```scala

def sum[F[+_]: ForEach, A](as: F[A])(implicit identity: Identity[Sum[A]]): A =
  foldMap(as)(a => Sum[A](a))
```

Here is how we could implement a fold that computes the sum and product of the values of a collection in a single pass:

```scala
def sumProd[F[+_]: ForEach, A](as: F[A])(implicit sum: Identity[Sum[A]], product: Identity[Prod[A]]): (A, A) =
  foldMap(as)(a => (Sum[A](a), Prod[A](a)))
```

As you can see by using `foldMap` with the functional abstractions in ZIO Prelude for describing ways of combining concrete types we can implement folds like this in an extremely high level and compositional way.

One common variant of a fold that can be particularly useful is `concatenate`, which allows us to combine all the elements of a collection into one using an `Identity` instance already defined for the element type. For example, here is how we could use it to combine a list of strings:

```scala
val strings = List("Hello", ", ", "World", "!")

strings.concatenate
```

Note that as with other extension methods in ZIO Prelude we need to do `import zio.prelude._` to bring these extension methods into scope.

In summary, `ForEach` is one of the most useful abstractions in ZIO Prelude in terms of providing a very large number of practically useful operators for implementing a single method. So if you are implementing your own parameterized type that contains zero or more values of the type it is parameterized on then defining a `ForEach` instance is a great "quick win" to immediately add a lot of functionality to your data type.

---

## IdentityBoth

`IdentityBoth[F]` describes an associative way to combine two values `F[A]` and `F[B]` into a value `F[(A, B)]` that also has an identity element of type `F[Any]`.

Its signature is:

```scala
trait AssociativeBoth[F[_]] {
  def both[A, B](fa: => F[A], fb: => F[B]): F[(A, B)]
}

trait IdentityBoth[F[_]] extends AssociativeBoth[F] {
  def any: F[Any]
}
```

The `any` value must be an identity element with respect to the `both` operator, so that after removing unnecessary tuples the following property holds:

```scala
fa <*> identity === fa
identity <*> fa === fa
```

This is the same as the laws for the `Identity` functional abstraction for concrete types except lifted into the context of parameterized types.

To be an identity element, running the `any` value must not do anything so we can always compose it as many times as we want with the `both` operator without changing the result.

As with the other abstractions for parameterized types we have looked at `ZIO` provides a good initial example.

The `any` value for `ZIO` is `unit`, the workflow that does not do anything, always succeeds, and produces no useful information. We can do nothing before or after another `ZIO` workflow as many times as we want and we will always get a workflow that does the same thing.

```scala

val helloUnit: ZIO[Console, IOException, Unit] =
  Console.printLine("Hello") <*> ZIO.unit
// helloUnit: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.helloUnit(identityboth.md:31)",
//   first = Stateful(
//     trace = "repl.MdocSession.MdocApp0.helloUnit(identityboth.md:31)",
//     onState = zio.FiberRef$unsafe$PatchFiber$$Lambda$17053/0x0000000804975840@4b1db8d6
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@8b9e48f
// )

val unitHello: ZIO[Console, IOException, Unit] =
  ZIO.unit <*> Console.printLine("Hello")
// unitHello: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.unitHello(identityboth.md:35)",
//   first = Sync(
//     trace = "",
//     eval = zio.ZIO$$$Lambda$17034/0x0000000804960040@59e5877c
//   ),
//   successK = zio.ZIO$$Lambda$17055/0x0000000804974040@7f984f9e
// )
```

These programs are identical because `ZIO.unit` does not do anything at all and always succeeds.

Similarly, the `any` with respect to data types such as `Either` and `Option` that model failure is a successful value that contains no useful information.

```scala
val anyEither: Either[Nothing, Any] =
  Right(())
// anyEither: Either[Nothing, Any] = Right(value = ())

val anyOption: Option[Any] =
  Some(())
// anyOption: Option[Any] = Some(value = ())
```

Since the `both` operator for these data types corresponds to failing on the first error or returning a success with all the results, the `any` value can never change the result. If `any` is combined with a failed value the result will be that failure, and if it is combined with a successful value the result will be that success.

For collection types the `any` value is a collection with a single value containing no useful information.

```scala
val anyList: List[Any] =
  List(())
// anyList: List[Any] = List(())
```

We might be tempted to think that an empty collection would be the identity value but we can see this is not the case because the `both` operator corresponds to the Cartesian product of two collections and the product of a collection with the empty collection is the empty collection, not the original collection.

If we think of a collection as representing a set of possible states then we can think of this as the state that occurs with certainty. Or if we think of the `both` operator as being the product of two collections the identity element for multiplication is one, not zero.

For a parser the identity element would be a parser that always succeeds with no useful information and does not change the parse state.

We can also define identity values with respect to contravariant types.

For example, consider the `Predicate` data type.

```scala
trait Predicate[-A] {
  def run(a: A): Boolean
}
```

We could define an `IdentityBoth` instance for it like this:

```scala

object Predicate {
  implicit val PredicateIdentityBoth: IdentityBoth[Predicate] =
    new IdentityBoth[Predicate] {
      val any: Predicate[Any] =
        new Predicate[Any] {
          def run(a: Any) =
            true
        }
      def both[A, B](left: => Predicate[A], right: => Predicate[B]): Predicate[(A, B)] =
        new Predicate[(A, B)] {
          def run(tuple: (A, B)): Boolean =
            left.run(tuple._1) && right.run(tuple._2)
        }
    }
}
```

The `both` operator for `Predicate` combines two predicates to return a new predicate that is true if both of the original predicates are true. So we can always combine any predicate with the predicate that is always true without changing the result.

This again shows the value of separating abstractions for defining how parameterized types can be combined from abstractions for describing their variance. Otherwise we would face a proliferation of abstractions to describe the product of these various combinations of properties of the combining operation and the variance of the data type, as other functional programming libraries have experienced.

When a data type has both a `IdentityBoth` and a `Covariant` instance we can define a particularly useful operator for it called `succeed`.

```scala
def succeed[F[+_]: IdentityBoth : Covariant, A](a: => A): F[A] =
  IdentityBoth[F].any.map(_ => a)
```

This says that if a data type is also covariant we can always "lift" any value into the data type by starting with the identity value and using `map` to transform the output type to the specified value.

This is very useful for working with parameterized data types in general because it allows us to take ordinary values and use them in the context of our parameterized type.

For example we can use the `succeed` operator on `ZIO` to wrap any arbitrary block of Scala code in a `ZIO` effect.

```scala
val helloScala: ZIO[Any, Nothing, Unit] =
  ZIO.succeed(println("Hello, Scala!"))
// helloScala: ZIO[Any, Nothing, Unit] = Sync(
//   trace = "repl.MdocSession.MdocApp0.helloScala(identityboth.md:98)",
//   eval = <function0>
// )
```

This is also quite useful in ZIO Prelude in particular because many operators are only defined for data types that have an `IdentityBoth` instance, such as the `forEach` operator on the `ForEach` abstraction.

As with concrete data types, it is quite useful to have an identity element with respect to the combining operator for parameterized types.

Most of the time for working with existing data types from ZIO or the Scala standard library it will be less important directly because operators for that data type will already be defined for you.

However, it can still be helpful to think about what the identity value is with respect to a given operation. For example, you may not have thought of a `Right` with no useful information as being an identity with respect to combining `Either` values.

When defining your own parameterized data types, it can be helpful to think about whether an identity element exists with respect to combining values of your data type. If not, could it be refactored to have such an identity value, or does thinking about that tell you something about why it makes sense that your data type does not have one?

Finally, if you are writing generic code in terms of the functional abstractions in ZIO Prelude this is likely to be another important abstraction. In particular the ability to lift a value into the parameterized type described by the combination of the `CommutativeBoth` and `Covariant` abstractions is often important.

---

## IdentityEither

`IdentityEither[F]` describes an associative way to combine two values `F[A]` and `F[B]` into a value `F[Either[A, B]]` with an identity value `none` of type `F[Nothing]`.

Its signature is:

```scala
trait AssociativeEither[F[_]] {
  def either[A, B](fa: => F[A], fb: => F[B]): F[Either[A, B]]
}

trait IdentityEither[F[_]] extends AssociativeEither[F] {
  def none: F[Nothing]
}
```

The `none` value must be an identity element with respect to the `either` operator so that, after eliminating cases of `Either` that can never contain values, the following property holds:

```scala
fa <+> none === fa
none <+> fa === fa
```

This is the same as the law for the `Identity` abstraction for concrete types but lifted into the context of parameterized types.

To satisfy these properties, rhe `none` value  must must be a value that fails with no useful information.

To see what this means, let's look at the `IdentityEither` instance for `Option`.

```scala

implicit val OptionIdentityEither: IdentityEither[Option] =
  new IdentityEither[Option] {
    def either[A, B](fa: => Option[A], fb: => Option[B]): Option[Either[A, B]] =
      fa match {
        case None =>
          fb match {
            case None    => None
            case Some(b) => Some(Right(b))
          }
        case Some(a) => Some(Left(a))
      }
    val none: Option[Nothing] =
      None
  }
// OptionIdentityEither: IdentityEither[Option] = repl.MdocSession$MdocApp0$$anon$1@6646477d
```

If we think of an `Option[A]` as representing either a success with a value of type `A` or a failure with no useful information, then the implementation of `none` is `None`. Combining any `Option[A]` with `None` with `orElseEither` won't change its result.

Now let's compare that to `Either` to see why we can't define a `IdentityEither` instance for `Either`.

We said that the identity value must be a value that fails with no useful information but `Either` is polymorphic in its error type so we can't define what a value that contains no information would be for an arbitrary type. We don't even know how to construct an value of that type.

```scala
def none[E]: Either[E, Nothing] =
  Left(???)
```

To define an identity value we would have to have the concept of an empty error. For example, if we created a data type that could fail without any error then the empty failure would be the `none` value.

```scala

val noneOption: Either[Option[Nothing], Nothing] =
  Left(None)
// noneOption: Either[Option[Nothing], Nothing] = Left(value = None)

val noneChunk: Either[Chunk[Nothing], Nothing] =
  Left(Chunk.empty)
// noneChunk: Either[Chunk[Nothing], Nothing] = Left(value = IndexedSeq())
```

Having this `none` value can be useful in certain situations. For example, it can help us to define a `filter` operator on a data type in combination with some of the other functional abstractions in ZIO Prelude.

However, supporting a `none` value requires allowing the possibility of that a value can fail without containing any errors, which is unattractive in many cases.

For example, `ZIO` and most effect types such as `ZManaged` and `ZStream` do not have an `IdentityEither` instance defined for them because is they fail they always fail with a failure `E`. Otherwise when we used an error handling operator we would have to deal with the error as well as the possibility that there is a failure but no error at all.

Note that in the case of `ZIO` a workflow can also die with a cause that can potentially be `Cause.empty`, but since the associative operator `orElseEither` does not recover from `Die` failures that doesn't help us with our problem here.

The other data types that do tend to have `IdentityEither` instances are collection types that can be empty. For example, the `IdentityEither` instance for `Chunk` looks like this.

```scala
implicit val ChunkIdentityEither: IdentityEither[Chunk] =
  new IdentityEither[Chunk] {
    def either[A, B](as: => Chunk[A], bs: => Chunk[B]): Chunk[Either[A, B]] =
      as.map(Left(_)) ++ bs.map(Right(_))
    val none: Chunk[Nothing] =
      Chunk.empty
  }
// ChunkIdentityEither: IdentityEither[Chunk] = repl.MdocSession$MdocApp0$$anon$2@576a0071
```

Here the identity element is just the empty chunk. If we think of running a collection as producing a set of values and then failing when the collection is empty, `none` is the collection that doesn't produce any values and fails immediately.

Another more mathematical way to think about it is that if concatenation corresponds to addition then the empty collection corresponds to zero. or this reason, we can also see that collections that cannot be empty, such as `NonEmptyChunk` from ZIO or `NonEmptyList` from ZIO Prelude, cannot have `IdentityEither` instances defined for them.

We can also define `none` values for contravariant types, though these tend to be less interesting.

For contravariant types, the `none` value corresponds to a value that can never be run because it has `Nothing` as its input type and there are no values of type `Nothing`. There the existence of the `none` value and our ability to compose it with the `orElseEither` operator in implies that values can be run independently of each other.

For example, let's look at the `IdentityEither` instance for `Predicate`.

```scala
trait Predicate[-A] {
  def run(a: A): Boolean
}

object Predicate {
  implicit val PredicateIdentityEither: IdentityEither[Predicate] =
    new IdentityEither[Predicate] {
      def either[A, B](left: => Predicate[A], right: => Predicate[B]): Predicate[Either[A, B]] =
        new Predicate[Either[A, B]] {
          def run(either: Either[A, B]): Boolean =
            either match {
              case Left(a) => left.run(a)
              case Right(b) => right.run(b)
            }
        }
      val none: Predicate[Nothing] =
        new Predicate[Nothing] {
          def run(a: Nothing): Boolean =
            a
    }
  }
}
```

The `none` predicate can never be run, but we can still safely compose it with other predicates with the `either` operator because since there can never be a value of type `Nothing`, there can also never be a case where the `either` operator tries to run the `none` predicate.

Overall the `IdentityEither` abstraction is a useful one to generalize over the concept of a value of a parameterized type that always fails with no information, complementing the concept of a value that always succeeds with no information from `IdentityBoth`.

For concrete types that have such a `none` value there tend to be existing operators defined on the data type that already take advantage of this this functionality, so `IdentityEither` tends to be less helpful when working with concrete data types.

For defining your own data types thinking about whether there is a `none` value can be helpful for API design. As discussed above, there is often a trade off between having an identity element, which is generally a good thing, and providing the guarantee that all failures contain some useful information.

Defining an `IdentityEither` instance for your own data type will also let you work with generic code in ZIO Prelude that uses this abstraction.

Finally `IdentityEither` can be useful when you are writing generic code to describe the concept that a data type must have this failure element with no information to provide a "default" failure value.

---

## IdentityFlatten

`IdentityFlatten[F]` describes a way of combining two layers of a value of type `F[F[A]]` into a `F[A]` in a way that is associative and has an identity value `any` of type `F[Any]`.

Its signature is:

```scala
trait AssociativeFlatten[F[+_]] {
  def flatten[A](f: F[F[A]]): F[A]
}

trait IdentityFlatten[F[+_]] extends AssociativeFlatten[F] {
  def any: F[Any]
}
```

The `any` value must be an identity with respect to the `flatten` operator. So if we add a new layer of nesting by mapping a value into the `any` value and then remove that layer with `flatten` we should get the original value back unchanged.

```scala
fa.map(a => any.map(_ => a)).flatten === fa
any.map(_ => fa).flatten === fa

```

This may look slightly different than the normal identity law because we have to map over the value but it is the same as the identity laws for the `Identity`, `IdentityBoth` and `IdentityEither` abstractions. Note how the two laws say whether we add the additional layer on the inside or the outside doesn't matter.

To be an identity element, the `any` value must not do anything when run other than return a value with no information content. This way we can always add a layer and remove it by using `flatten` to get back to the original value.

For example the `any` value for `ZIO` is `unit`. The `unit` workflow doesn't do anything and always succeeds.

So we can always `flatten` a `ZIO` workflow constructed by mapping over the `unit` value to the original workflow back.

```scala

val helloIdentity: ZIO[Console, IOException, Unit] =
  ZIO.unit.map { _ =>
    Console.printLine("Hello from an identity!")
  }.flatten
// helloIdentity: ZIO[Console, IOException, Unit] = FlatMap(
//   trace = "repl.MdocSession.MdocApp0.helloIdentity(identityflatten.md:33)",
//   first = FlatMap(
//     trace = "repl.MdocSession.MdocApp0.helloIdentity(identityflatten.md:31)",
//     first = Sync(
//       trace = "",
//       eval = zio.ZIO$$$Lambda$17034/0x0000000804960040@59e5877c
//     ),
//     successK = zio.ZIO$$Lambda$17101/0x00000008049d5040@27335fdb
//   ),
//   successK = <function1>
// )
```

Recall from our discussion of `AssociativeFlatten` that the `flatten` operator corresponds to running a workflow, which returns a new workflow, and then running the workflow. Here the `unit` workflow doesn't do anything and always succeeds so we can simply eliminate it.

Just like the `any` value that is an identity with respect to the `both` operator of a `IdentityBoth`, the `any` value represents a successful value that contains no information. So other examples of the `any` value would include:

```scala
val anyOption: Option[Any] =
  Some(())
// anyOption: Option[Any] = Some(value = ())

val anyEither: Either[Nothing, Any] =
  Right(())
// anyEither: Either[Nothing, Any] = Right(value = ())

val anyList: List[Any] =
  List(())
// anyList: List[Any] = List(())
```

By combining the structure described by the `IdentityFlatten` abstraction and the `Covariant` abstraction we can also define the `succeed` operator we saw from `IdentityBoth`.

```scala

def succeed[F[+_]: IdentityFlatten : Covariant, A](a: => A): F[A] =
  IdentityFlatten[F].any.map(_ => a)
```

As discussed in the section on `IdentityBoth`, the `succeed` operator is helpful because it lets us "lift" any normal value into the context of the parameterized type.

For example, using the `succeed` operator on `ZIO` we can import an arbitrary block of Scala code into a `ZIO` workflow.

```scala
def helloScala: ZIO[Any, Nothing, Unit] =
  ZIO.succeed(println("Hello Scala!"))
```

For this operator to be well defined we have to be able to construct a value `any` that doesn't do anything and has no information content. If all values of the parameterized type required some additional information, for example, we would not be able to define the `IdentityFlatten` abstraction or `succeed` for it.

In addition to the `succeed` operator being generally useful, there are some other operators that require an `IdentityFlatten` instance so that they can use it in their own implementations.

In particular, the `ForEach` abstraction defines a set of operators that allow us to fold over a collection in the context of a parameterized type. These require that we have an `IdentityFlatten` instance so that if the collection is empty we can simply succeed with an empty collection in the context of the parameterized type.

The `IdentityFlatten` abstraction is quite powerful, combining the flexibility of the `AssociativeFlatten` abstraction to run one value and then use its result to run another value with the ability to lift arbitrary values into the context of the parameterized type.

If you are using data types from ZIO or the standard library there are some operators defined on the `ForEach` abstraction that can be helpful but otherwise the operators defined in terms of `IdentityFlatten` are likely already implemented directly on the data types you are using.

If you are implementing your own data type you should certainly consider whether your data type can implement `flatMap`, or whether you explicitly don't want to. Either way being able to lift a value into the parameterized type with `succeed` using either the `IdentityBoth` or `IdentityFlatten` abstraction is extremely useful so you should either implement one of those abstractions or have very specific reasons why that is not possible.

Finally, for writing generic code the combination of the `IdentityFlatten` and `Covariant` abstractions is also quite powerful, allowing modeling imperative code in a functional context.

---

## Introduction(Parameterized-types)

In addition to abstractions for concrete types, ZIO Prelude provides a set of functional abstractions to describe the common structure of _parameterized types_.

A parameterized type is a type that is parameterized on one or more other types. For example, a `List[A]` is parameterized on the element type `A`.

When we are describing the common structure of parameterized types we are talking about the structure of the parameterized type without knowing about the type it is parameterized on. For example, consider the following two abstractions to describe associative ways of combining two values of a data type.

```scala
trait Associative[A] {
  def combine(left: => A, right: => A): A
}

trait AssociativeBoth[F[_]] {
  def both[A, B](left: => F[A], right: => F[B]): F[(A, B)]
}
```

The `Associative` abstraction describes a way of associatively combining two concrete data types.

The syntax is relatively straightforward here. The `Associative` is parameterized on a type `A` that has an associative `combine` operator.

When we define an instance of `Associative` we know the type of the values we are combining and can use that information in implementing the `combine` operator. For example we know we have two `Int` values and can combine them by adding them together.

```scala
implicit val IntAssociative: Associative[Int] =
  new Associative[Int] {
    def combine(left: => Int, right: => Int): Int =
      left + right
  }
// IntAssociative: Associative[Int] = repl.MdocSession$MdocApp$$anon$1@59ce9e30
```

In contrast, the `AssociativeBoth` abstraction describes a way of associatively combining two parameterized types.

The syntax here may look a little new. Instead of being parameterized on an `A`, `AssociativeBoth` is parameterized on an `F[_]`.

There is nothing special about the `F` here, just like the `A` in `Associative` it is just a placeholder for some type. By convention we start with `A` for concrete types and `F` for parameterized types but we can call these types whatever we want.

What is important is the `[_]` after `F`. This tells the Scala compiler that this is a parameterized type, such as a `List` rather than a `List[Int]`.

The importance of this becomes clear when we define an instance of `AssociativeBoth` because we do not know anything about the types `A` and `B` that the `left` and `right` values are parameterized on. This prevents us from doing anything with the `A` and `B` values that requires knowledge of their types and so essentially forces us to work exclusively at the level of the `F` structure.

To see this, consider how we would implement an instance of `AssociativeBoth` for `List`.

```scala
implicit val ListAssociativeBoth: AssociativeBoth[List] =
  new AssociativeBoth[List] {
    def both[A, B](left: => List[A], right: => List[B]): List[(A, B)] =
      left.flatMap(a => right.map(b => (a, b)))
  }
// ListAssociativeBoth: AssociativeBoth[List] = repl.MdocSession$MdocApp$$anon$2@147352df
```

Notice how the `AssociativeBoth` instance is parameterized on `List` rather than a list of any specific type. This is important because it says the `ListAssociativeBoth` instance knows how to combine any two lists in an associative way, not just two lists of some specific type.

What would the implementation of such a way of combining look like? It would not be able to use any information about the `A` or `B` values, because they could be anything, so it would have to combine them in a way that worked for any `A` and `B` values.

The `AssociativeBoth` abstraction does this by combining the `A` and `B` values into a tuple. We will see later that this is one of two fundamental ways that we can combine two values in a generic way.

In our implementation for `List` we return the Cartesian product of the two lists, which is an associative operation after reassociating nested tuples.

We'll talk more about `AssociativeBoth` in the section on that abstraction. For now the goal of this discussion is not to discuss `AssociativeBoth` in detail but to get a sense of what it means to describe the structure of a parameterized type instead of a concrete one.

Just like for concrete types, the abstractions for parameterized types fall into two categories.

## Properties Of Parameterized Types

The first set of abstractions define properties of single values of a parameterized type `F[A]`. All of them describe the fundamental nature of the `A` parameter with respect to `F`.

### Invariant

The first relationship that `A` can have to `F` is that `F` both produces and consumes `A` values. For example, a `JsonCodec[A]` both produces `A` values by turning JSON into `A` values and consumes `A` values by turning `A` values into JSON.

Parameterized types that both produce and consume `A` values are _invariant_ in the `A` type parameter and we describe them with the `Invariant` abstraction.

An invariant type `F[A]` can be transformed into an `F[B]` with an `Equivalence[A, B]` using the `invmap` operator. Conceptually, if we have a data type that produces and consumes `A` values we can always turn it into a data type that produces and consumes `B` values by transforming all inputs into `A` values and all outputs back to `B` values with the equivalence relationship.

This is useful to "lift" equivalence relationships into the context of a parameterized type. For example, if we have defined an equivalence between new and old versions of our data model we can use `Invariant` to convert the `JsonCodec` for the old data model into a `JsonCodec` for the new data model.

### Covariant

The second relationship that `A` can have to `F` is that `F` produces `A` values but does not consume them. Data types that produce `A` values may either just contain existing `A` values, like a `Chunk`, or potentially produce `A` values at some point in the future, like a `ZIO`.

Parameterized types that produce but do not consume `A` values are _covariant_ in the `A` type parameter and we describe them with the `Covariant` abstraction.

A covariant type `F[A]` can be transformed into an `F[B]` with a function `A => B` using the `map` operator. Conceptually, if we have a data type that produces `A` values then we can create a data type that produces `B` values simply by taking each output and transforming it with the function.

This is useful to allow us to transform the output of a covariant type to build richer data pipelines. For example, if we had a function that decrypted some data and a `ZIO` effect that loaded the encrypted data from a file we could return a new `ZIO` effect that produced the decrypted data, which we could then compose with the rest of our program.

### Contravariant

The third relationship that `A` can have to `F` is that `F` consumes `A` values but does not produce them. Examples of data types that consume values of one or more types include Scala functions with respect to their input, `ZIO` with respect to its environment type, and `ZSink` with respect to stream elements.

Parameterized types that consume but do not produce `A` values are _contravariant_ in the `A` type parameter and we describe them with the `Contravariant` abstraction.

A contravariant type `F[A]` can be transformed into an `F[B]` with a function `B => A` using the `contramap` operator. Conceptually, if we have a data type that consumes `A` values then we can create a data type that consumes `B` values  simply by transforming each `B` value into an `A` value with the function before feeding it to the original data type.

This is useful to allow us to transform the input of a contravariant type to build richer data pipelines. For example, if we have a sink that takes chunks of bytes and writes them to a file we could transform it into a sink that takes strings and writes them to a file by providing a function to transform strings into bytes.

### ForEach

The `ForEach` abstraction builds on the `Covariant` abstraction by describing a data type that not only produces `A` values but actually contains zero or more `A` values as opposed to merely being able to potentially generate them at some point in the future. For example, a `Chunk` has zero or more existing `A` values and so has a `ForEach` instance defined for it whereas a `ZIO` merely may produce an `A` value at some point in the future.

A data type with a `ForEach` instance contains zero or more existing `A` values so we can iterate over it, potentially transforming its values while maintaining its structure using an operator like `ZIO.foreach`. We can also tear it down entirely and reduce it to a summary value with an operator like `foldLeft`.

This ability to get an `A` value out of the data type only exists because `ForEach` describes a data type where the `A` values already exist. If we had a data type like `ZIO` then we would not be able to get an `A` value out of it because a `ZIO` is only a description of a workflow that may produce an `A` value when it is run.

The `ForEach` abstraction is one of the most practically useful functional abstractions in ZIO Prelude. Most collection operators can be defined in terms of `ForEach` and especially in combination with the abstractions for describing ways of combining concrete types it provides very powerful ways of working with collections.

### NonEmptyForEach

The `NonEmptyForEach` abstraction further builds on `ForEach` to describe a parameterized type that contains one or more existing `A` values. For example, a `NonEmptyChunk` is guaranteed to contain at least one `A` value.

Because a data type with a `NonEmptyForEach` instance must always contain at least one existing `A` value we can define certain operators for it that would not be safe to define for a data type that might not contain any `A` values. For example, we can define a `reduce` operator that reduces the collection to a summary value with an associative operator whereas with a `ForEach` instance we can only define a `fold` operator that also requires an identity value to handle the empty case.

## Combining Parameterized Types

The second set of abstractions describe ways of combining parameterized types.

Because of the additional structure of parameterized types there are three separate ways that we can combine values of parameterized types, each of which may satisfy properties of associativity, commutativity, and identity.

The first way of combining an `F[A]` and an `F[B]` is to return both `A` and `B` values. This corresponds to some notion of doing both things, though what exactly this means will depend on the parameterized type and the combining operation.

This way of combining is described by the `AssociativeBoth`, `CommutativeBoth`, and `IdentityBoth` functional abstractions.

The second way of combining an `F[A]` and an `F[B]` is to return either an `A` or a `B` value. This corresponds to some notion of choosing which value to return, though again what exactly this means and how we choose will depend on the parameterized type and the combining operation.

This way of combining is described by the `AssociativeEither`, `CommutativeEither`, and `IdentityEither` functional abstractions.

The third way of combining actually combines two layers of `F` values, converting an `F[F[A]]` into an `F[A]`. This corresponds to some notion of evaluating the outer layer and then evaluating the inner layer.

This way of combining is described by the `AssociativeFlatten` and `IdentityFlatten` functional abstractions.

### AssociativeBoth

The `AssociativeBoth` abstraction describes a parameterized type `F[A]` with an associative operator `both` that can combine a value of type `F[A]` and a value of type `F[B]` into a value of type `F[(A, B)]`. This is described by the `zip` operator on `ZIO` and corresponds to running the left value and then running the right value.

### CommutativeBoth

`CommutativeBoth` describes a way of combining an `F[A]` and an `F[B]` to produce an `F[(A, B)]` that is both associative and commutative.

This is described by the `zipPar` operator on `ZIO` and corresponds to running the left value and the right value in parallel, since this is the only way that the order will not matter as required by the commutative property.

### IdentityBoth

The `IdentityBoth` abstraction describes a way of combining an `F[A]` and an `F[B]` to produce an `F[(A, B)]` that is both associative and has an identity value of type `F[Any]`. In ZIO the identity value is value is `ZIO.unit` since zipping any value with unit is equivalent to returning the original value unchanged

### AssociativeEither

The `AssociativeEither` abstraction describes a parameterized type `F[A]` with an associative operator `either` that can combine a value of type `F[A]` and a value of type `F[B]` into a value of type `F[Either[A, B]]`. This is described by the `orElseEither` operator on `ZIO` and corresponds to running the left value, but if it fails then running the right value.

### CommutativeEither

`CommutativeEither` builds on `AssociativeEither` by describing an operator that combines an `F[A]` and and `F[B]` to produce an `F[Either[A, B]]` in a way that is both associative and commutative. In ZIO this corresponds to the `raceEither` operator which runs both effects concurrently and returns the first to succeed, since running both effects concurrently is the only way that the order will not matter as required by the commutative property.

### IdentityEither

The `IdentityEither` abstraction describes a way of combining an `F[A]` and an `F[B]` to produce an `F[Either[A, B]]` that is both associative and has an identity value of type `F[Nothing]`. `ZIO` does not have such an identity value since it is not possible to fail without any error but conceptually such an identity would correspond to a failure that did not contain any information.

### AssociativeFlatten

The `AssociativeFlatten` abstraction describes the ability to flatten two layers of `F` structure in an associative way. In `ZIO` this corresponds to the `flatten` operator, and in combination with the `Covariant` abstraction the `flatMap` operator, corresponding to running one `ZIO` value and using its result to generate another `ZIO` value and running that value.

### IdentityFlatten

The `IdentityFlatten` abstraction describes an identity value of type `F[Any]` with respect to the associative flattening operation defined by `AssociativeFlatten`. In `ZIO` this corresponds to `ZIO.unit` and is the same as the identity value defined by the `IdentityBoth` instance.

---

## Invariant

`Invariant[F]` describes a parameterized type `F[A]` that potentially both consumes and produces `A` values.

Its signature is:

```scala
trait Invariant[F[_]] {
  def invmap[A, B](f: A <=> B): F[A] <=> F[B]
}

type <=>[A, B] = Equivalence[A, B]

case class Equivalence[A, B](to: A => B, from: B => A)
```

The `invmap` operator says we can "lift" an equivalence relationship between `A` and `B` into an equivalence relationship between `F[A]` and `F[B]`. An equivalence relationship says we can transform back and forth between `A` and `B` values without losing any information and is described by the `Equivalence` data type in ZIO Prelude.

If we import `zio.prelude._` we can use the `invmap` operator in infix form to transform an `F[A]` into an `F[B]` with an equivalence relationship `A <=> B`.

The law is that the lifting of this equivalence relationship can transform `A` and `B` values but cannot otherwise change the structure of `F`, so using `invmap` with the identity equivalence relationship is an identity and separately using `invmap` with two equivalence relationships is the same as doing it with the composition of those equivalence relationships.

```scala
fa.invmap(Equivalence.identity) === fa
fa.invmap(f).invmap(g) === fa.invmap(f.andThen(g))
```

An example of a data type that is invariant is a JSON codec.

```scala
trait JsonCodec[A] {
  def decode(json: String): Either[String, A]
  def encode(a: A): String
}
```

Notice that `A` appears in both the output type of the `decode` operator and the input type of the `encode` operator. This makes `JsonCodec` naturally invariant with respect to the `A` type parameter.

Let's try implementing an `invmap` operator for our `JsonCodec` type.

```scala

trait JsonCodec[A] { self =>

  def decode(json: String): Either[String, A]

  def encode(a: A): String

  def invmap[B](f: A <=> B): JsonCodec[B] =
    new JsonCodec[B] {
      def decode(json: String): Either[String, B] =
        self.decode(json).map(f.to)
      def encode(b: B): String =
        self.encode(f.from(b))
    }
}

object JsonCode {
  implicit val JsonCodecInvariant: Invariant[JsonCodec] =
    new Invariant[JsonCodec] {
      def invmap[A, B](f: A <=> B): JsonCodec[A] <=> JsonCodec[B] =
        Equivalence(_.invmap(f), _.invmap(f.flip))
    }
}
```

The implementation of `invmap` was quite simple and follows a pattern we can always use when implementing `invmap` for our own data types.

For every operator that outputs `A` values, we can output `B` values instead by taking the `A` values and transforming them to `B` values with the equivalence relationship. For example, in our implementation of `decode` we decode a string to produce an `Either[String, A]` with the original codec and then map over the `Either[String, A]` with the equivalence relationship to produce an `Either[String, B]`.

Similarly, for every operator that accepts `A` values as an input, we can accept `B` values instead and then simply transform them into `A` values with the equivalence relationship. So in our implementation of `encode` we accept `B` values to be encoded, transform them into `A` values with the equivalence relationship, and then feed them to the `encode` operator of the original codec to get a JSON string.

The operators may vary and we may need to do slightly different things to transform the inputs or outputs like mapping over the `Either` in the `JsonCodec` example, but this basic pattern applies to implementing the `invmap` operator for all invariant types.

Other data types that are invariant are the `Set` data type in the Scala standard library and the `Ref`, `Queue`, and `Hub` data types in `ZIO`.

The `Invariant` data type itself describes very limited structure. It just says that this parameterized type potentially consumes and produces `A` values, which are conceptually the only things it could do with `A` values.

As a result, the `invmap` operator itself is not very powerful. It only lets us go back and forth between `F[A]` and `F[B]` values when there is an equivalence relationship between `A` and `B`, which basically means that `A` and `B` contain the same information.

However, `Invariant` can be useful when we do have an equivalence relationship between two data types to "lift" that equivalence relationship to the level of the parameterized type.

For example, if we have two different versions of our data model it could be generally useful to define an `Equivalence` between those different versions of our data model so we could translate back and forth between different versions of the data model. With that, we could also use `Invariant` to generate a `JsonCodec` for the new data model given a `JsonCodec` for the old data model, or vice versa.

---

## NonEmptyForEach

`NomEmptyForEach[F]` describes a parameterized type `F[A]` that contains one or more values of type `A`.

Its signature is:

```scala
trait ForEach[F[+_]] {
  def forEach[G[+_]: IdentityBoth: Covariant, A, B](fa: F[A])(f: A => G[B]): G[F[B]]
}

trait NonEmptyForEach[F[+_]] extends ForEach[F] {
  def forEach1[G[+_]: AssociativeBoth : Covariant, A, B](fa: F[A])(f: A => G[B]): G[F[B]]
  final def forEach[G[+_]: IdentityBoth: Covariant, A, B](fa: F[A])(f: A => G[B]): G[F[B]] =
    forEach1(fa)(f)
}

trait Covariant[F[+_]] {
  def map[A, B](f: A => B): F[A] => F[B]
}

trait AssociativeBoth[F[_]] {
  def both[A, B](fa: => F[A], b: => F[B]): F[(A, B)]
}

trait IdentityBoth[F[_]] extends AssociativeBoth[F] {
  def any: F[Any]
}
```

The `NonEmptyForEach` functional abstraction builds on the `ForEach` abstraction to describe a type that contains one or more `A` values rather than zero or more `A` values.

For example `NonEmptyChunk` has a `NonEmptyForEach` instance defined for it because it always contains at least one value. In contrast `Chunk` only has a `ForEach` instance defined for it because it could be empty.

`NonEmptyForEach` generalizes over collection types and types that always contain at least one value, such as `NonEmptyChunk`, `NonEmptyList`, and certain tree data structures.

The defining operator of the `NonEmptyForEach` abstraction is `forEach1`, which has the same signature as the `forEach` operator defined by the `ForEach` abstraction except that it doesn't require an identity element with respect to the combining operator for `G` values.

Recall that if instances of `Covariant` and `AssociativeBoth` exist for a data type we can define the `map` and `zipWith` operators and if an instance of `IdentityBoth` exists as well we can define the `succeed` operator.

```scala
def map[F[+_], A, B](fa: F[A])(f: A => B)(implicit covariant: Covariant[F]): F[B] =
  covariant.map(f)(fa)

def succeed[F[+_], A](a: => A)(implicit covariant: Covariant[F], both: IdentityBoth[F]): F[A] =
  covariant.map[Any, A](_ => a)(both.any)

def zipWith[F[+_], A, B, C](
  fa: F[A],
  fb: F[B]
)(f: (A, B) => C)(implicit covariant: Covariant[F], both: AssociativeBoth[F]): F[C] =
  covariant.map(f.tupled)(both.both(fa, fb))
```

To see why we don't need an identity element for the combining operator when the collection contains at least one value let's compare the implementation of the `ForEach` instance for `List` with the implementation of the `NonEmptyForEach` instance for the `NonEmptyList` data type from ZIO Prelude.

```scala

implicit val ListForEach: ForEach[List] =
  new ForEach[List] {
    def forEach[G[+_]: IdentityBoth: Covariant, A, B](fa: List[A])(f: A => G[B]): G[List[B]] =
      fa.foldRight(succeed[G, List[B]](List.empty)) { (a, gbs) =>
        zipWith(f(a), gbs)(_ :: _)
      }
  }
// ListForEach: ForEach[List] = repl.MdocSession$MdocApp$$anon$1@20557b40

implicit val NonEmptyListNonEmptyForEach: ForEach[NonEmptyList] =
  new NonEmptyForEach[NonEmptyList] {
    def forEach1[G[+_]: AssociativeBoth: Covariant, A, B](fa: NonEmptyList[A])(f: A => G[B]): G[NonEmptyList[B]] =
      fa.reduceMapRight(a => map(f(a))(NonEmptyList.single))((a, gbs) => zipWith(f(a), gbs)(NonEmptyList.cons))
  }
// NonEmptyListNonEmptyForEach: ForEach[NonEmptyList] = repl.MdocSession$MdocApp$$anon$2@4bfe794b
```

In our implementation of `ForEach` for `List` we needed to use the `succeed` operator to handle the case where the collection was empty, because in that case we had to be able to lift an empty collection into the context of `G`. The `succeed` operator requires an `IdentityBoth` instance because we have to be able to construct a neutral value of type `G` that we can then fill with the value we are lifting using the `map` operator.

In contrast, in our implementation of `NonEmptyForEach` we never need to call `succeed` because there is always at least one one element in the collection. So we can just apply the function `f` to each element in the collection and then use the `zipWith` operator to combine the results.

The fact that the collection can never be empty allows us to relax constraints on other operators as well.

For example, we can define an operator called `reduceMapLeft` that is a more powerful version of `foldLeft` that does not require an initial value.

```scala

def reduceMapLeft[F[+_]: ForEach, A, S](as: F[A])(map: A => S)(reduce: (S, S) => S): S =
  as.foldLeft[Option[S]](None) {
    case (Some(s), a) => Some(reduce(s, map(a)))
    case (None, a) => Some(map(a))
  }.get
```

We know it is safe to call `get` here because the collection is guaranteed to have at least one element.

The `reduceMapLeft` operator allows us to define additional operators for reducing a collection to a summary value that would not be safe to call on a collection that might be empty.

In particular, we can define a more powerful version of the `foldMap` operator defined on the `ForEach` abstraction called `reduceMap`.

```scala
def reduceMap[F[+_]: ForEach, A, B: Associative](as: F[A])(f: A => B): B =
  reduceMapLeft(as)(f)(_ <> _)
```

Since we know the collection contains at least one element we do not need an `identity` value with respect to the associative `combine` operator, just like we did not need an `any` value with respect to the `both` operator in `forEach1`.

With this we can easily do something like calculate the sum, product, min, and max of a collection in a single pass.

```scala

def stats[F[+_]: ForEach, A](as: F[A])(
  implicit sum: Associative[Sum[A]],
  prod: Associative[Prod[A]],
  min: Associative[Min[A]],
  max: Associative[Max[A]]
): (A, A, A, A) =
  reduceMap(as)(a => (Sum[A](a), Prod[A](a), Min[A](a), Max[A](a)))
```

This is a very nice way to describe reducing a collection to a summary value and gives us additional flexibility to use ways of combining that are associative but do not have an identity element relative to the `foldMap` operator.

As we can see, the `NonEmptyForEach` functional abstraction builds on the `ForEach` functional abstraction to describe parameterized types that contain one or more values of the type they are parameterized on. This lets us define a wide variety of operators, even more than the ones we could define for the `ForEach` abstraction.

So if you are defining your own data type like a collection and it will never be empty you should definitely define a `NonEmptyForEach` instance for it so you can take advantage of all the nice operators that are defined in terms of it.

---

## Equivalence

An `Equivalence[A, B]` describes an equivalence relationship between two types `A` and `B`.

An equivalence relationship is defined in terms of two functions `to` and `from` that convert a value of type `A` to a value of type `B` and vice versa.

```scala
case class Equivalence[A, B](to: A => B, from: B => A)
```

To be a valid `Equivalence` the functions `to` and `from` must satisfy an identity law, which says that for any value of type `A`, if we transform it to a `B` using `to` and then back using `from` we get the same value. The same property must also apply for the reverse.

Simple examples of an equivalence relationship would be between an `List[A]` and a `Chunk[A]`. We can transform any `List` into a `Chunk` using the `Chunk.fromIterable` operator, and we can transform any `Chunk` into a `List` using the `toList` operator:

```scala

def listChunkEquivalence[A]: Equivalence[List[A], Chunk[A]] =
  Equivalence(Chunk.fromIterable, _.toList)
```

This essentially represents the fact that `List` and `Chunk` contain the same information. They represent it in different ways internally and have different performance characteristics but both of them model zero or more values of some type `A`.

We can see from the example above that the order of describing `List` and `Chunk` in the equivalence relationship was somewhat arbitrary. We could just as well have described this as an equivalence between `Chunk[A]` and `List[A]`.

The `Equivalence` data type lets us express that through the `flip` operator.

```scala
def chunkListEquivalence[A]: Equivalence[Chunk[A], List[A]] =
  listChunkEquivalence.flip
```

In addition to being able to reverse equivalence relationships we can also compose them. If `A` is equivalent to `B` and `B` is equivalent to `C` then `A` is equivalent to `C`.

We can express this using the `andThen` operator on `Equivalence` or its symbolic alias `>>>`.

To demonstrate this let's define another equivalence relationship `Vector[A]` and `List[A]`.

```scala
def vectorListEquivalence[A]: Equivalence[Vector[A], List[A]] =
  Equivalence(_.toList, _.toVector)
```

Given these two equivalence relationships we can then define an equivalence relationship between a `Vector[A]` and a `Chunk[A]`.

```scala
def vectorChunkEquivalence[A]: Equivalence[Vector[A], Chunk[A]] =
  vectorListEquivalence.andThen(listChunkEquivalence)
```

This is not a particularly interesting example because we could have easily converted directly from `Vector` to `Chunk` in a more performant way without going through `List`. But we can imagine that if `A`, `B`, and `C` were equivalent but different representations of some more complex data type, being able to build up these conversions incrementally could be quite useful.

Beyond this there are not a lot of operators on the `Equivalence` data type itself. The main power of this data type is the ability to express the fact that two types are equivalent and capture this as a data type that we can test and reuse.

In particular, if we have what we believe to be an equivalence relationship between two types `A` and `B` ZIO Prelude makes it easy for us to test this.

```scala

object EquivalenceSpec extends ZIOSpecDefault {

  def spec = suite("EquivalenceSpec") {
    test("chunkListEquivalence") {
      implicit val equivalence = listChunkEquivalence[Int]
      val listGen = Gen.listOf(Gen.int)
      val chunkGen = Gen.chunkOf(Gen.int)
      checkAllLaws(EquivalenceLaws)(listGen, chunkGen)
    }
  }
}
```

ZIO Test will generate a large number of `A` and `B` values and check that the identity law holds. This can be helpful to catch corner cases where we think two things are equivalent, but they are really not.

With these equivalence relationships in hand we can then convert between different representations of our data in a straightforward and principled way. This is helpful when we need to implement higher level logic that needs to rely on this equivalence relationship existing between certain data types.

---

## Functional Data Types in ZIO Prelude

ZIO Prelude includes several data types to help us model our domains more accurately and solve common problems.

- **[Equivalence](equivalence.md)** - A description of an equivalence relationship between two data types.
- **[NonEmptyList](nonemptylist.md)** - A list that is guaranteed to be non-empty to more accurately model situations where we know a collection has at least one element.
- **[These](these.md)** - A data type that may either be a `Left` with an `A`, a `Right` with a `B`, or a `Both` with an `A` and a `B`, useful for modeling problems such as merging streams of data.
- **[Validation](validation.md)** - A data type that may be either a success or an accumulation of one or more errors, allowing modeling multiple failures for applications such as data validation.
- **[ZSet](zset.md)** - A generalization of a set that generalizes measures of "how many" of an element exist in a set, supporting multi-sets, "fuzzy" sets, and other data structures.
- **[ZValidation](zvalidation.md)** - A generalization of `Validation` that allows maintaining a log of warnings in addition to accumulating errors.

---

## NonEmptyList

`NonEmptyList` represents a `List` that is guaranteed not to be empty.

This is useful to allow more accurate domain modeling in situations where we know a collection is not empty.

For example, we might be handling a collection of errors. If an error has occurred at all then we know the collection is not empty.

However, if we are working with a collection type from the Scala standard library such as `List`, `Vector`, or `Iterable` then the Scala compiler does not know that. It will force us to handle the case where the collection is empty even though we know that cannot occur.

Or perhaps we are implementing a batched API and need to handle a collection of requests. We know the collection is not empty, or we would not have been called at all, but now we have to handle the case of an empty collection.

Sometimes there are sensible defaults in these cases, but other times we are forced to resort to potentially unsafe operators like `head` or to throw exceptions in branches of logic we know should never occur. This wastes developer time, makes code harder to understand, and encourages bad coding practices.

To avoid these problems we want a data type that is specific enough to describe at the type level what we know to be true, in this case that a collection is not empty.

We can try to represent this using the `::` case of `List` from the Scala standard library but unfortunately `::` itself has extremely poor ergonomics. It is hard to construct `::` values, and it is easy for operators to forget the more specific type. This is where `NonEmptyList` comes in.

A `NonEmptyList` looks like this:

```scala
sealed trait NonEmptyList[+A]

object NonEmptyList {
  case class Single[A](head: A)                      extends NonEmptyList[A]
  case class Cons[A](head: A, tail: NonEmptyList[A]) extends NonEmptyList[A]
}
```

We can see that `NonEmptyList` looks very similar to `List` except that there is no `Nil` case. A `NonEmptyList` is either a `Single` with a single value of type `A` or a `Cons` with a value of type `A` and a `NonEmptyList` of further values of type `A`.

Generally you should just be able to use higher level operators on `NonEmptyList` but if you ever want to implement your own operators on `NonEmptyList` you can do so by pattern matching on a `NonEmptyList` just like you would on a `List`.

## Constructing NonEmptyList Values

The easiest way to construct a `NonEmptyList` is with the `apply` operator.

```scala
trait NonEmptyList[+A]

object NonEmptyList {
  def apply[A](a: A, as: A*): NonEmptyList[A] =
    ???
}
```

If we already have another collection type we can convert it into a `NonEmptyList` using the `fromIterableOption` operator.

```scala
object NonEmptyList {
  def fromIterableOption[A](as: Iterable[A]): Option[NonEmptyList[A]] =
    ???
}
```

Notice that `fromIterableOption` returns an `Option[NonEmptyList[A]]` because the original collection might be empty, and so we might not be able to construct a `NonEmptyList` from it.

If we know the collection is not empty we can use the `fromIterable` operator and provide the first element of the collection to establish that it is not empty.

```scala
object NonEmptyList {
  def fromIterable[A](a: A, as: Iterable[A]): NonEmptyList[A] =
    ???
}
```

We can also construct a `NonEmptyList` from the `::` case of a `List`.

```scala
object NonEmptyList {
  def fromCons[A](as: ::[A]): NonEmptyList[A] =
    ???
}
```

This is useful if we already have a `::` value, for example from a `ZIO` operator like `ZIO.validate` that returns `::[E]` if it fails.

In addition to these operators for constructing `NonEmptyList` values from other values there are a couple of operators that are useful for building up `NonEmptyList` values.

The `single` operator lets us construct a `NonEmptyList` from a single value. The `cons` operator lets us construct a new `NonEmptyList` from an existing `NonEmptyList` and a new head value.

```scala
object NonEmptyList {
  def cons[A](a: A, as: NonEmptyList[A]): NonEmptyList[A] =
    ???
  def single[A](a: A): NonEmptyList[A] =
    ???
}
```

These operators let us build larger `NonEmptyList` values from smaller ones in recursive algorithms much like the `::` operator and `Nil` allow us to do for `List` values.

## Working With NonEmptyList Values

The operators on `NonEmptyList` values are essentially the same as those on `List`. So if you know how to work with a `List` you know how to work with a `NonEmptyList`.

The main difference between a `NonEmptyList` and a `List` is that many operators are safe on `NonEmptyList` that are not safe on `List`.

For example, we would generally not consider it a good practice to call `head` on a `List` because that could throw an exception. However, it is fine to call `head` on a `NonEmptyList` because it is always guaranteed to contain at most one value.

Similarly, we can use `reduceLeft` or `reduceRight` to reduce the elements of a `NonEmptyList` to a summary value even though this would not be safe to do on a `List` because we know there is always at least one element.

The other difference between a `NonEmptyList` and a `List` is that operators on `NonEmptyList` will try to preserve the knowledge that the list is not empty whenever possible.

For example, the `map` operator on `NonEmptyList` looks like this:

```scala
trait NonEmptyList[+A] {
  def map[B](f: A => B): NonEmptyList[B]
}
```

The return type is also a `NonEmptyList` because `map` transforms each element with a function but does not change the number of elements, so if the original list was not empty then we know that the resulting list is not empty.

However, sometimes it is not possible to preserve this information because of the nature of the operator.

```scala
trait NonEmptyList[+A] {
  def dropWhile(f: A => Boolean): List[A]
}
```

The `dropWhile` operator returns a `List[A]` instead of a `NonEmptyList[A]` because it could drop all of the elements of the list, resulting in a list that is empty. So we can no longer return a `NonEmptyList`.

The final difference between `NonEmptyList` and `List` is some additional operators are provided that take advantage of the fact that we know the list is not empty.

The most notable of these are `reduceMapLeft` and `reduceMapRight`, which take the place of `foldLeft` and `foldRight`.

```scala
trait NonEmptyList[+A] {
  def reduceMapLeft[B](map: A => B)(reduce: (B, A) => B): B
  def reduceMapRight[B](map: A => B)(reduce: (A, B) => B): B
}
```

On standard collections `foldLeft` and `foldRight` are fundamental collection operators that allow us to "tear down" a collection to a summary value by providing an initial value to use if the collection is empty and a way to update that initial value for every element in the collection.

Since we know that a `NonEmptyList` contains at least one value, we don't need to provide an initial value for when the list is empty and can use the `reduceLeft` and `reduceRight` operators. However, these operators don't provide us quite the power we need in a lot of situations because they don't allow the type of the summary value to differ from the type of the elements in the collection.

The `reduceMapLeft` and `reduceMapRight` operators allow us to transform the first value in the `NonEmptyList` to the summary value and then combine each other value with that summary value. This is useful because it allows us to preserve the information that the collection is not empty.

For example, here is how we could use `reduceMapLeft` to convert a `NonEmptyList` to a `NonEmptyChunk`.

```scala

def toNonEmptyChunk[A](as: NonEmptyList[A]): NonEmptyChunk[A] =
  as.reduceMapLeft(a => NonEmptyChunk(a))(_ appended _)
```

This would have been difficult to express with `foldLeft` because we would have had to start with an initial value that was an empty chunk and thus would have gotten back a `Chunk` instead of a `NonEmptyChunk`.

Of course, we could have just used the `toNonEmptyChunk` operator on `NonEmptyList`.

```scala
def toNonEmptyChunk[A](as: NonEmptyList[A]): NonEmptyChunk[A] =
  as.toNonEmptyChunk
```

## Interoperability With Scala Collections

One of the great features of the `NonEmptyList` data type is its excellent integration with the Scala standard library collections framework.

A `NonEmptyList` is implicitly convertible into the `::` case of a `List` so anywhere you need a `List` and provide a `NonEmptyList` it will just work.

```scala

def myExistingAPI(as: List[Int]): String =
  "Some fancy stuff"

val nonEmptyList: NonEmptyList[Int] =
  NonEmptyList(1, 2, 3)
// nonEmptyList: NonEmptyList[Int] = Cons(
//   head = 1,
//   tail0 = Cons(head = 2, tail0 = Single(head = 3))
// )

myExistingAPI(nonEmptyList)
// res2: String = "Some fancy stuff"
```

This is also extremely convenient because it means all operators available on `List` are available on `NonEmptyList`.

For example, the `filter` operator is not implemented directly on `NonEmptyList`. But you can still filter a `NonEmptyList` because we can always view a `NonEmptyList` as a `List`.

```scala
val filtered: List[Int] =
  nonEmptyList.filter(_ % 2 == 0)
// filtered: List[Int] = List(2)
```

The return type of `List[Int]` is as specific as possible because the `filter` operator could filter out all the elements of the list and return a list that is empty.

Thus, `NonEmptyList` lets us work in a way that is as close as possible to being as if `NonEmptyList` was a subtype of `List`, which conceptually it is.

---

## These

`These[A, B]` is a data type that models a value that can be either a `Left` with a value of type `A`, a `Right` with a value of type `B`, or a `Both` with both a value of type `A` and a value of type `B`.

```scala
sealed trait These[+A, +B] {
  case class Left[+A](a: A)           extends These[A, Nothing]
  case class Right[+B](b: B)          extends These[Nothing, B]
  case class Both[+A, +B](a: A, b: B) extends These[A, B]
}
```

You can think of `These` as like `Either` except that it has one additional case. Whereas an `Either` contains either an `A` or a `B` but never both, a `These` can contain both an `A` and a `B`.

A simple example of a situation where `These` might arise is in merging two upstreams in a streaming application. We could be signaled when at least one upstream has data for us and could have a situation where either only the left upstream has data for us, only the right upstream has data for us, or both upstreams have data for us.

Another example of using the `These` data type is in implementing a variant of the `zipAll` operator on collections.

When we zip two collections, if the two collections have different sizes the default `zipWith` operator will just drop the "extra" elements from the larger collection.

```scala

val left: Chunk[Int] =
  Chunk(1, 2, 3)
// left: Chunk[Int] = IndexedSeq(1, 2, 3)

val right: Chunk[Int] =
  Chunk(4, 5, 6, 7, 8)
// right: Chunk[Int] = IndexedSeq(4, 5, 6, 7, 8)

val zip: Chunk[Int] =
  left.zipWith(right)(_ + _)
// zip: Chunk[Int] = IndexedSeq(5, 7, 9)
```

We can use the `zipAll` operator to allow the caller to specify how they want to handle the situation where the two collections have different sizes. Normally the signature would look something like this:

```scala
def zipAllWith[A, B, C](
  as: Chunk[A],
  bs: Chunk[B]
)(left: A => C, right: B => C, both: (A, B) => C): Chunk[C] =
  ???
```

Now the `both` function will be called as long as there are elements of both collections to zip together. If the left collection is longer then the `left` function will be called for the extra elements of the left collection, and if the right collection is longer then the `right` operator will be called for the extra elements of the right collection.

This works and is basically the signature of the `zipAllWith` operator on `Chunk`, but there is something a little less than ideal here that now we need to spread out the logic for handling these cases across three different functions. It would be nice if we could describe this as a single function.

With `These` we can do just that. We could rewrite the signature of `zipAllWith` like this:

```scala
def zipAllWith[A, B, C](
  as: Chunk[A],
  bs: Chunk[B]
)(f: These[A, B] => C): Chunk[C] =
  ???
```

Now the function `f` bundles up all the logic that was previously in `left`, `right`, and `both`. The caller gets to provide a single function that handles all three of these cases.

The `These` data type is a relatively modest one, but it captures a situation that can arise where we can have both values in addition to one or the other. Otherwise, we would have to create our own data type for each of these situations, but now we can simply use `These`.

---

## Validation(Functional-data-types)

A `Validation[E, A]` is a data type that is either a `Success` with a value of type `A` or a `Failure` with one or more errors of type `E`.

You can think of a `Validation` as looking something like this:

```scala

sealed trait Validation[+E, +A]

object Validation {
  case class Success[+A](value: A)                 extends Validation[Nothing, A]
  case class Failure[+E](errors: NonEmptyChunk[E]) extends Validation[E, Nothing]
}
```

Thus, a `Validation` looks very much like an `Either` except that it can contain multiple errors instead of just one.

The distinguishing feature of `Validation` is that it can accumulate multiple errors. This makes it extremely convenient for problems like data validation where we want to accumulate all validation errors and report them back instead of failing immediately on the first validation error.

To see this, let's consider a simple example of validating a `Person` data type.

```scala
case class Person(name: String, age: Int)
```

The business requirements are that a valid person must have a name that is not empty and an age that is greater than zero.

We could try to model validating a `Person` using `Either` like this:

```scala
def validateName(name: String): Either[String, String] =
  if (name.isEmpty) Left("Name was empty")
  else Right(name)

def validateAge(age: Int): Either[String, Int] =
  if (age <= 0) Left(s"Age $age was less than zero")
  else Right(age)

def validatePerson(name: String, age: Int): Either[String, Person] =
  for {
    name <- validateName(name)
    age  <- validateAge(age)
  } yield Person(name, age)
```

However, there is a problem with this. If we try to validate a `Person` with both an invalid `name` and an invalid `age` we will just get a single failure indicating that the name was empty.

This is because `Either` embodies "fail fast" semantics where if one part of a computation fails the subsequent parts of the computation will never be performed.

Normally this is what we want because there is no point in doing further work if a computation has already failed. This is similar to how in traditional imperative programs if an exception is thrown on one line then further lines will not be executed.

However, it is not what we want here.

We could imagine that the `name` and `age` were transmitted by a customer. They would probably not be happy if we told them the name was invalid, they corrected that and sent it again, and then we failed again because the age was invalid.

"Why didn't you tell me about all the problems up front?" we can see them saying. `Validation` does just that.

We can fix this example using `Validation` like so:

```scala

case class Person(name: String, age: Int)

def validateName(name: String): Validation[String, String] =
  if (name.isEmpty) Validation.fail("Name was empty")
  else Validation.succeed(name)

def validateAge(age: Int): Validation[String, Int] =
  if (age <= 0) Validation.fail(s"Age $age was less than zero")
  else Validation.succeed(age)

def validatePerson(name: String, age: Int): Validation[String, Person] =
  Validation.validateWith(validateName(name), validateAge(age))(Person)
```

Here we are just using the basic operators on `Validation` of `succeed` and `fail` which construct a `Validation` that succeeds or fails with the specified value. We are then using the `validateWith` operator, which combines the results of multiple `Validation` values, accumulating all errors that occur.

Note that we are using the `apply` method of `Person` to combine the validated name and age into a `Person`. This is often a convenient pattern when we are modeling our data using case classes.

Now if we validate a person where both the `name` and the `age` are invalid we will get a failure containing both errors, giving us exactly what we need for our internal customer, so they can efficiently correct the problem and send us valid data.

## Constructing Validation Values

Now that we understand what `Validation` is, let's look at how to construct `Validation` values.

### From Existing Values

The simplest ways to construct `Validation` values are the `succeed` and `fail` operators we saw above:

```scala
object Validation {
  def fail[E](error: E): Validation[E, Nothing] =
    ???
  def succeed[A](value: A): Validation[Nothing, A] =
    ???
}
```

These just construct validation successes or failures with the specified value. We can use these as the building blocks for creating more complex validation logic as we saw above.

### From Code That May Throw Exceptions

Another useful constructor of `Validation` values is the `apply` method on `Validation`:

```scala
object Validation {
  def apply[A](a: => A): Validation[Throwable, A] =
    ???
}
```

The `apply` operator takes a by name argument and evaluates that argument, converting any non-fatal thrown exceptions into `Validation` failures. This is useful when working with code that might throw exceptions to safely import that code into a `Validation` value.

### From Predicates

The `fromPredicateWith` constructor is useful for constructing `Validation` values from predicates. This is nice for cleaning up code that constructs a validation failure or success based on some condition like the example we wrote above:

```scala
object Validation {
  def fromPredicateWith[E, A](error: => E)(value: A)(f: A => Boolean): Validation[E, A] =
    ???
}
```

The `fromPredicateWith` constructor constructs a `Validation` from a value, a predicate about that value, and an error to provide if the predicate is false.

For example, here is how we could simplify the example we wrote above using the `fromPredicateWith` operator:

```scala
def validateName(name: String): Validation[String, String] =
  Validation.fromPredicateWith("Name was empty")(name)(_.nonEmpty)

def validateAge(age: Int): Validation[String, Int] =
  Validation.fromPredicateWith(s"Age $age was less than zero")(age)(_ >= 0)
```

This is the same as the original example but lets us express our logic at a slightly higher level.

### From Other Data Types

There are also operators for constructing `Validation` values from a variety of other data types in the Scala standard library such as `Either`, `Option`, and `Try`:

```scala

object Validation {
  def fromEither[E, A](value: Either[E, A]): Validation[E, A] =
    ???
  def fromOption[A](value: Option[A]): Validation[Unit, A] =
    ???
  def fromTry[A](value: => Try[A]): Validation[Throwable, A] =
    ???
}
```

Each of these operators just converts another data type that models potential failures into a `Validation`.

In the case of `fromEither` the `Validation` will fail with the same error type as the `Either`.

In the case of `fromTry` the error type is always `Throwable`, since a `Try` can fail with any `Throwable`. The `fromTry` operator will also catch any exception thrown in the construction of the `Try` value and convert it to a `Validation` failure.

A `Validation` constructed using `fromOption` has `Unit` for the error type since an `Option` contains no useful information about why a failure has occurred. We will see shortly when we learn about transforming `Validation` values how we can add additional information here about why a validation failure occurred.

## Transforming Validation Values

Now that we know about creating `Validation` values, the next thing we need to know is how to transform them.

We can transform the value type of the `Validation` using the `map` operator:

```scala
trait Validation[+E, +A] {
  def map[B](f: A => B): Validation[E, B]
}
```

We can transform the error type of the `Validation` using the `mapError` operator:

```scala
trait Validation[+E, +A] {
  def mapError[E2](f: E => E2): Validation[E2, A]
}
```

This can be useful to transform the error type to our own internal data model. It can also be useful to add additional information about why a validation failure occurred, for example adding a descriptive error message if an optional value did not exist.

## Combining Validation Values

We can also combine `Validation` values.

### Accumulating Errors

The most common way to do this is with the `validateWith` operator we saw above:

```scala
object Validation {
  def validateWith[E, A, B, C](
    left: Validation[E, A],
    right: Validation[E, B]
  )(f: (A, B) => C): Validation[E, C] =
    ???
}
```

This will check each of the original `Validation` values and if they are both successes combine their values with the function `f` to produce a new successful `Validation` value. If one or more of the original `Validation` values are failures then the result will be a failure with all of the failures of both original `Validation` values.

There are variants of `validateWith` for combining up to twenty two different `Validation` values so even if you are working with large cases classes with many fields you can still use `validateWith` to combine them.

There is also a `validate` variant that combines multiple `Validation` values but does not take a combining function. In this case the returned `Validation` just contains a tuple with all the original values, and you can combine them yourself later using `map`.

If you have a whole collection of values you want to validate you can use the `validateAll` operator to validate them all at the same time.

### Chaining Validations

The second way we might want to combine `Validation` values is by returning another `Validation` value based on the result of the first validation value. For example, we might first try to parse a string into an integer and then validate that the integer is within a reasonable range.

```scala

def validateInt(s: String): Validation[String, Int] =
  Validation(s.toInt).mapError(_.getMessage)

def validateAge(age: Int): Validation[String, Int] =
  Validation.fromPredicateWith(s"Age $age was less than zero")(age)(_ >= 0)

def validateAgeString(s: String): Validation[String, Int] =
  for {
    n <- validateInt(s)
    _ <- validateAge(n)
  } yield n
```

Here the second validation depended on the first one because if we could not parse the string into an integer then we would not even have an integer to validate. Just like with other data types we can express this with the `flatMap` operator and use a for comprehension to chain multiple validations together.

Note that when we chain validations like this we only do the second validation if the first one is successful so we will never see both errors here. If all we are doing is chaining then we don't actually need `Validation` and could just use `Either`.

However, frequently we want to do some validations with error accumulating semantics and chain others. Having `flatMap` on `Validation` makes it very easy for us to do this.

To see this, let's expand on our example of validating a `Person` data type from above. Now we will validate the person based on two `String` inputs using the logic we implemented above.

```scala
def validatePerson(name: String, age: String): Validation[String, Person] =
  Validation.validateWith(validateName(name), validateAgeString(age))(Person)
```

Now we are combining accumulating errors and chaining validations.

Validating the age will proceed by validating that the age can be parsed into an `Int` and then that the age is not negative. This will return either a validated age or a validation error if either of these steps failed.

We are also validating that the name is not empty. The final result will be either a validated person or a validation error containing both any errors that occurred in validating the name and age.

By combining operators like `validateWith` and `flatMap` you can build complex validation logic that handles any of your problems in this or related domains.

## Extracting Validation Values

The final thing we want to be able to do with `Validation` values is extract them to access the success value or errors and do something with them. This also includes transforming `Validation` values into other data types that we may be interested in.

The most basic way to do this is with the `fold` operator, which lets us provide functions specifying what we want to do with the success or failure.

```scala
trait Validation[+E, +A] {
  def fold[B](failure: E => B, success: A => B): B
}
```

Using `fold`, we can get out the success value of the `Validation` by providing a function that says what we want to do if the `Validation` is a failure. For example, we might provide a default value or refail in the context of some other data type that can model failure such as `ZIO`.

There are a variety of helpful operators implemented in terms of `fold` for transforming `Validation` values into other data types we might be interested in. The most common of these are:

```scala

trait Validation[+E, +A] {
  def toEither: Either[NonEmptyChunk[E], A]
  def toEitherWith[E2](f: NonEmptyChunk[E] => E2): Either[E2, A]
  def toZIO: IO[E, A]
}
```

The `toEither` operator converts the `Validation` into an `Either` where the `Left` side of the other contains a `NonEmptyChunk` of all the errors that occurred if the `Validation` was a failure. The `toEitherWith` operator is similar to this but lets us provide our own function for converting the collection of errors that occurred into our own error type.

The `toZIO` operator is particularly useful because frequently we will do some data validation which we will model with `Validation` but then want to do some further work with the result that needs a `ZIO` effect such as writing it to a file or sending it over the wire.

With `toZIO` we can easily do our validation in the context of `Validation` and then "lift" it into the context of `ZIO` to compose with our other `ZIO` effects. One nice feature of this operator is that it will automatically preserve all validation errors that occurred in ZIO's `Cause` data type.

---

## ZSet

`ZSet` is a version of a set that generalizes the measure of "how many times" each element appears in the set.

Conceptually, we can think of a `ZSet[A, B]` as a `Map[A, B]`, where `A` represents the elements of the set and `B` represents some notion of how many times each element appears.

We can then define a `Set` as a `ZSet` where the measure of how many times each element appears is a `Boolean`.

```scala

type Set[+A] = ZSet[A, Boolean]
```

That is, in a `Set` the measure of how many times an element appears is just yes or no. Either the element is in the set or it is not.

However, we can also define many other variants of sets by choosing different types to measure how many times an element appears.

For example, by using the `Natural` new type from ZIO Prelude, which describes integers greater than or equal to zero, we can define the type of a `MultiSet` or "bag" in some programming languages.

```scala

type MultiSet[+A] = ZSet[A, Natural]
```

This ends up being a quite useful data type and can represent anything where the number of times an element appears matters but the order doesn't matter, such as a shopping cart.

By using other measures of how many times an item appears we can describe other variants of sets such as `Int` for sets where items can appear a negative number of times or `Double` for sets where the number of times an item appears can be fractional.

`ZSet` is focused on using the functional abstractions and new types in ZIO Prelude to describe operators on these generalizations of sets in as practical a way as possible.

## Constructing ZSets

The most basic way to construct a `ZSet` is from a `Map` where the keys in the map represent the elements in the `ZSet` and the values in the map represent the number of times each element appears.

```scala
val shoppingList: ZSet[String, Int] =
  ZSet.fromMap(Map("apples" -> 2, "oranges" -> 3, "bananas" -> 5))
// shoppingList: ZSet[String, Int] = ZSet(apples -> 2, bananas -> 5, oranges -> 3)
```

We can also construct a `ZSet` from existing collection types.

If we have a normal `Set` from the Scala standard library, we can convert it to a `ZSet` using the `fromSet` operator.

```scala
val set: ZSet[Int, Boolean] =
  ZSet.fromSet(Set(1, 2, 3))
// set: ZSet[Int, Boolean] = ZSet(1 -> true, 2 -> true, 3 -> true)
```

If we have an `Iterable` we can convert it into a `ZSet` using the `fromIterable` constructor. The measure of how many times each item appears will just be how many times it appears in the `Iterable`.

```scala
val multiSet: MultiSet[Int] =
 ZSet.fromIterable(List(1, 1, 2))
// multiSet: MultiSet[Int] = ZSet(1 -> 2, 2 -> 1)
```

Finally, we can just construct an empty `ZSet` using the `empty` operator and combine `ZSet` with other `ZSet` values later.

```scala
val empty: ZSet[Nothing, Nothing] =
  ZSet.empty
// empty: ZSet[Nothing, Nothing] = ZSet()
```

## Operators on ZSets

As a generalization of sets, `ZSet` supports many of the same operators that exist on sets.

### Set Membership

We can determine how many times an element appears in the set using the `apply` operator. This requires that the measure type has an `Identity` defined for it.

If the element exists in the set the measure of how many times it appears will be returned. Otherwise the identity element will be returned.

Thus, `shoppingList("apples")` will return `2` and `shoppingList("eggs")` will return `0` while `set(1)` will return `true` and `set(4)` will return `false`.

### Set Union

One simple way of combining sets is the union of two sets.

The union of two sets contains all elements that appear in either set. If an element appears in both sets the measure of how many times it appears in the union will be the maximum of the two measures, as determined by an `Ord` defined on the measure type.

```scala
val left: ZSet[Int, Int] =
  ZSet.fromMap(Map(1 -> 2, 2 -> 3, 3 -> 4))
// left: ZSet[Int, Int] = ZSet(1 -> 2, 2 -> 3, 3 -> 4)

val right: ZSet[Int, Int] =
  ZSet.fromMap(Map(2 -> 2, 3 -> 3, 4 -> 4))
// right: ZSet[Int, Int] = ZSet(2 -> 2, 3 -> 3, 4 -> 4)

val union: ZSet[Int, Int] =
  left.union(right)
// union: ZSet[Int, Int] = ZSet(1 -> 2, 2 -> 3, 3 -> 4, 4 -> 4)
```

Here the result will be `ZSet(1 -> 2, 2 -> 3, 3 -> 4, 4 -> 4)` because all elements in both sets will be included in the union of the two sets.

## Set Intersection

Set intersection is another basic way of combining sets.

The intersection of two sets only contains the elements that appear in both sets. If an element appears in both sets the measure of how many times it appears in the intersection will be the minimum of the two measures, as determined by an `Ord` defined on the measure type.

```scala
val left: ZSet[Int, Int] =
  ZSet.fromMap(Map(1 -> 2, 2 -> 3, 3 -> 4))
// left: ZSet[Int, Int] = ZSet(1 -> 2, 2 -> 3, 3 -> 4)

val right: ZSet[Int, Int] =
  ZSet.fromMap(Map(2 -> 2, 3 -> 3, 4 -> 4))
// right: ZSet[Int, Int] = ZSet(2 -> 2, 3 -> 3, 4 -> 4)

val intersection: ZSet[Int, Int] =
  left.intersect(right)
// intersection: ZSet[Int, Int] = ZSet(2 -> 2, 3 -> 3)
```

The result will be `ZSet(2 -> 2, 3 -> 3)` because the only commonality of the two sets is that they contain these elements.

## Set Difference

We can also compute the difference of two sets.

The difference of two sets is obtained by subtracting the measure of how many times each element appears in the second set from how many times it appears in the first set. This requires an `Inverse` defined on the `Sum` of the measure type to tell us what it means to "subtract" for this type.

```scala
val left: ZSet[Int, Int] =
  ZSet.fromMap(Map(1 -> 2, 2 -> 3, 3 -> 4))
// left: ZSet[Int, Int] = ZSet(1 -> 2, 2 -> 3, 3 -> 4)

val right: ZSet[Int, Int] =
  ZSet.fromMap(Map(2 -> 2, 3 -> 3, 4 -> 4))
// right: ZSet[Int, Int] = ZSet(2 -> 2, 3 -> 3, 4 -> 4)

val diff: ZSet[Int, Int] =
  left.diff(right)
// diff: ZSet[Int, Int] = ZSet(1 -> 2, 2 -> 1, 3 -> 1, 4 -> -4)
```

The result will be `ZSet(1 -> 2, 2 -> 1, 3 -> 1, 4 -> -4)`.

Note that the measure of how many times `4` appears in the resulting set is negative. This is permissible because `Int` allows negative numbers.

If we didn't want this we could define the measure type to be a type like `Natural` that did not permit negative numbers.

```scala
val left: MultiSet[String] =
  ZSet.fromIterable(List("apple", "banana"))
// left: MultiSet[String] = ZSet(banana -> 1, apple -> 1)

val right: MultiSet[String] =
  ZSet.fromIterable(List("banana", "banana", "banana"))
// right: MultiSet[String] = ZSet(banana -> 3)

val diff: MultiSet[String] =
  left.diff(right)
// diff: MultiSet[String] = ZSet(banana -> 0, apple -> 1)
```

Now the measure of how many times each item appears is a natural number so "banana" cannot appear a negative number of times.

## Set Combination

Another helpful way of combining sets that makes more sense for generalized sets is the notion of "adding" two sets. This is less relevant for traditional sets because an element can never appear more than once in a traditional set, so "adding" reduces to set union.

However, for generalized sets this way of combining can be quite intuitive. For example if we think of a multiset as John's shopping list and another multiset as Jane's shopping list, the combination of the two includes everything on both shopping lists, with overlapping items appearing the total number of times they appear on both lists.

This requires a `Commutative` defined on the `Sum` of the measure type to tell us how to do this addition.

```scala
val janeShoppingList: ZSet[String, Int] =
  ZSet.fromMap(Map("apples" -> 4, "bananas" -> 5))
// janeShoppingList: ZSet[String, Int] = ZSet(apples -> 4, bananas -> 5)

val johnShoppingList: ZSet[String, Int] =
  ZSet.fromMap(Map("eggs" -> 12, "bananas" -> 1))
// johnShoppingList: ZSet[String, Int] = ZSet(eggs -> 12, bananas -> 1)

val combinedList: ZSet[String, Int] =
  janeShoppingList.combine(johnShoppingList)
// combinedList: ZSet[String, Int] = ZSet(eggs -> 12, apples -> 4, bananas -> 6)
```

The result is `ZSet("apples" -> 4, "bananas" -> 6, "eggs" -> 12)`, the combination of what each of them wants to buy.

## Set Product

We can also define the product of two sets.

This is the Cartesian product of the two sets, with each combination of elements from the two sets appearing in the product set and the measure of how many times an element appears being the product of the two measures. This requires a `Commutative` defined on the `Prod` of the measure type to tell us how to do this multiplication.

A nice example of this is modeling discrete probabilities. We can define the set of all possible outcomes of a dice roll along with their associated probabilities like this:

```scala
val die: ZSet[Int, Double] =
  ZSet.fromMap(
    Map(
      1 -> 1.0 / 6.0,
      2 -> 1.0 / 6.0,
      3 -> 1.0 / 6.0,
      4 -> 1.0 / 6.0,
      5 -> 1.0 / 6.0,
      6 -> 1.0 / 6.0
    )
  )
// die: ZSet[Int, Double] = ZSet(5 -> 0.16666666666666666, 1 -> 0.16666666666666666, 6 -> 0.16666666666666666, 2 -> 0.16666666666666666, 3 -> 0.16666666666666666, 4 -> 0.16666666666666666)
```

The product of this set with itself represents the set of all possible outcomes of rolling two dice.

```scala
val pair: ZSet[(Int, Int), Double] =
  die.zip(die)
// pair: ZSet[(Int, Int), Double] = ZSet((5,4) -> 0.027777777777777776, (2,6) -> 0.027777777777777776, (4,1) -> 0.027777777777777776, (5,2) -> 0.027777777777777776, (1,1) -> 0.027777777777777776, (1,6) -> 0.027777777777777776, (6,4) -> 0.027777777777777776, (3,2) -> 0.027777777777777776, (3,3) -> 0.027777777777777776, (4,3) -> 0.027777777777777776, (5,6) -> 0.027777777777777776, (2,2) -> 0.027777777777777776, (3,4) -> 0.027777777777777776, (2,1) -> 0.027777777777777776, (6,1) -> 0.027777777777777776, (2,5) -> 0.027777777777777776, (4,2) -> 0.027777777777777776, (4,4) -> 0.027777777777777776, (1,2) -> 0.027777777777777776, (6,5) -> 0.027777777777777776, (6,2) -> 0.027777777777777776, (1,4) -> 0.027777777777777776, (4,5) -> 0.027777777777777776, (4,6) -> 0.027777777777777776, (2,4) -> 0.027777777777777776, (6,3) -> 0.027777777777777776, (1,5) -> 0.027777777777777776, (3,1) -> 0.027777777777777776, (5,1) -> 0.027777777777777776, (3,6) -> 0.027777777777777776, (2,3) -> 0.027777777777777776, (3,5) -> 0.027777777777777776, (1,3) -> 0.027777777777777776, (5,5) -> 0.027777777777777776, (5,3) -> 0.027777777777777776, (6,6) -> 0.027777777777777776)
```

This set has thirty six elements, corresponding to each possible combination of outcomes from the two dice rolls, each with a probability of one in thirty six.

This gets more interesting if we say that matters is not the individual dice rolls but their sum.

We can express this in code with the `zipWith` operator, which allows us to combine the elements from the product of the two sets with a function. If multiple elements from the product are mapped to the same value the measures of how many times each value appears will be added, using a `Commutative` instance defined on the `Sum` of the measure type.

```scala
val sum: ZSet[Int, Double] =
  die.zipWith(die)(_ + _)
// sum: ZSet[Int, Double] = ZSet(5 -> 0.1111111111111111, 10 -> 0.08333333333333333, 6 -> 0.1388888888888889, 9 -> 0.1111111111111111, 2 -> 0.027777777777777776, 12 -> 0.027777777777777776, 7 -> 0.16666666666666669, 3 -> 0.05555555555555555, 11 -> 0.05555555555555555, 8 -> 0.1388888888888889, 4 -> 0.08333333333333333)
```

The resulting set has eleven elements, corresponding to the possible sums from two dice rolls of two to twelve. The measure for each value corresponds to the probability of that sum being rolled.

So for example the probability of seven is about 16.7% whereas the probability of rolling two or twelve is only about 2.8%.

We can take this even further using the `flatMap` operator.

The `zipWith` operator lets us combine two sets when the sets are independent of each other. With the `flatMap` operator we can create a new set based on each element of the first set and then "flatten" the resulting set down.

We could rewrite the example above using `flatMap` and a for comprehension like this:

```scala
val sum: ZSet[Int, Double] =
  for {
    a <- die
    b <- die
  } yield a + b
// sum: ZSet[Int, Double] = ZSet(5 -> 0.1111111111111111, 10 -> 0.08333333333333333, 6 -> 0.1388888888888889, 9 -> 0.1111111111111111, 2 -> 0.027777777777777776, 12 -> 0.027777777777777776, 7 -> 0.16666666666666669, 3 -> 0.05555555555555555, 11 -> 0.05555555555555555, 8 -> 0.1388888888888889, 4 -> 0.08333333333333333)
```

This allows us to express conditional logic, such as the probability distribution of outcomes for a subsequent event depending on the outcome of a previous event.

## Transforming Elements

We can transform set elements using the `map` operator.

For example, we could add one to each value in a set like this:

```scala
val setPlusOne: ZSet[Int, Boolean] =
  set.map(_ + 1)
// setPlusOne: ZSet[Int, Boolean] = ZSet(2 -> true, 3 -> true, 4 -> true)
```

This requires a `Commutative` instance defined for the `Sum` of the measure type so that we can combine the measures if we map two elements to the same value.

## Transforming Measures

We can also transform the measure we use to represent how many times an item appears using the `transform` operator. For example, perhaps we have constructed a set where the measure is an `Int` but we would like to transform it to a `Double` so we can compose it with other `ZSet` values with `Double` as the measure type.

We can do that with `transform` like this:

```scala
def constant(n: Int): ZSet[Int, Double] =
  ZSet(n).transform(_.toDouble)
```

## Extracting ZSet Values

If the number of times a value appears in the set is a natural number then we can use normal collection operators to fold over its values, treating it much like a normal set

```scala
val fruits: MultiSet[String] =
  ZSet.fromIterable(List("apple", "banana", "orange", "apple", "orange"))
// fruits: MultiSet[String] = ZSet(banana -> 1, orange -> 2, apple -> 2)

val fruitCount: Int =
  fruits.foldLeft(0)((n, _) => n + 1)
// fruitCount: Int = 5
```

This will return `5` because there are five elements in the set we are folding over.

We can convert a `ZSet` back to a `Map` from elements to a measure of how many times they appear using the `toMap` operator.

We can also always convert a `ZSet` back to a `Set` using the `toSet` operator. Any elements in the `ZSet` with a measure equal to zero based on an `Identity` defined on the `Sum` of the measure type will be discarded from the resulting set.

---

## ZValidation

Like many other data types in ZIO, the `Validation` data type we learned about before is actually a type alias for a more general data type called `ZValidation`.

A `ZValidation[W, E, A]` is either a `Success` with a value of type `A` or a `Failure` with one or more errors of type `E`, and in either case contains a log of values of type `W`. A `Validation[E, A]` is just a type alias for `ZValidation[Nothing, E, A]` and since the log type was `Nothing` we were able to ignore it.

```scala
type Validation[+E, +A] = ZValidation[Nothing, E, A]
```

This type `W` can be used for two main purposes.

First, it can be used to maintain a "log" of the things we did in validating some data. In some cases it can be important for auditing or debugging purposes to know not just whether data is valid or why not but the steps we took in validating that data.

Second, it can be used to represent a concept of "warnings".

By definition a `Validation` is either a success with a value or a failure with one or more errors. So if an error occurred the data is invalid and there is no value at all.

However, we can imagine some cases where data might be questionable or noteworthy in some way without being invalid.

For example, we might say that a valid `Person` must have an age that is not negative, so a `Person` with an age of `10` would still be valid data. But depending on the application this data might still be unusual, for example if this `Person` represents a credit card customer.

In cases like this it would be useful to have some way of capturing these warnings as data and propagating them through our validation logic so that we can see all the warnings that occurred and take some further action based on them. This could include logging them, triggering a user to confirm that their input is correct, or triggering other processes based on the content of these warnings.

To model this we can represent a `ZValidation` as follows:

```scala

trait ZValidation[+W, +E, +A]

object ZValidation {
  case class Success[W, A](log: Chunk[W], value: A)                 extends ZValidation[W, Nothing, A]
  case class Failure[W, E](log: Chunk[W], errors: NonEmptyChunk[E]) extends ZValidation[W, E, Nothing]
}
```

Now both validation successes and failures have a log of values of type `W` so we can keep track of values in the log regardless of whether the validation is a success or a failure.

Since a `Validation` is a `ZValidation` all of the operators we learned about before for `Validation` still work. So we will focus here on operators that are specific to working with the `W` type.

## Writing Values To The Log

To write values to the log we can just use the `log` operator on the `ZValidation` trait.

```scala
trait ZValidation[+W, +E, +A] {
  def log[W1 >: W](w: W1): ZValidation[W1, E, A]
}
```

This will just append the specified value to the log without changing the success or failure value of the `ZValidation`.

For example, we could add a warning that the `Person` is a child like this:

```scala

def validateAge(age: Int): ZValidation[String, String, Int] =
  if (age < 0) ZValidation.fail(s"Age $age was less than zero")
  else if (age < 18) ZValidation.succeed(age).log(s"Age $age is less than 18")
  else ZValidation.succeed(age)
```

Notice that we used exactly the same constructors as when we were working with `Validation` except we now use the `log` operator to append to the log.

## Transforming The Log

We can transform the value of the log entries with the `mapLog` operator.

```scala
trait ZValidation[+W, +E, +A] {
  def mapLog[W2](f: W => W2): ZValidation[W2, E, A]
}
```

This can be useful to transform the log values into some common data model, for example mapping `String` logs into some more structured data format.

## Reading Values From The Log

We can get values from the log by using the `getLog` operator, which just returns the current contents of the log.

```scala

trait ZValidation[+W, +E, +A] {
  def getLog: Chunk[W]
}
```

Since a `ZValidation` is just a value we can use a combination of the `getLog` operator and `fold` to extract the log and the errors or success value.

---

## Introduction to ZIO Prelude

[ZIO Prelude](https://github.com/zio/zio-prelude) is a lightweight, distinctly Scala take on **functional abstractions**, with tight ZIO integration.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-prelude/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-prelude_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-prelude_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-prelude_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-prelude_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-prelude-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-prelude-docs_2.13) [![ZIO Prelude](https://img.shields.io/github/stars/zio/zio-prelude?style=social)](https://github.com/zio/zio-prelude)

## Introduction

ZIO Prelude is a small library that brings common, useful algebraic abstractions and data types to scala developers. It is an alternative to libraries like _Scalaz_ and _Cats_ based on radical ideas that embrace **modularity** and **subtyping** in Scala and offer **new levels of power and ergonomics**. It throws out the classic functor hierarchy in favor of a modular algebraic approach that is smaller, easier to understand and teach, and more expressive.

ZIO Prelude has three key areas of focus:

1. **Data structures, and type classes for traversing them.** ZIO Prelude embraces the collections in the Scala standard library, and extends them with new instances and new useful additions.
2. **Patterns of composition for types.** ZIO Prelude provides a small catalog of patterns for binary operators, which combine two values into another value of the same type. These patterns are named after the algebraic laws they satisfy: associativity, commutativity, and identity.
3. **Patterns of composition for type constructors.** ZIO Prelude provides a catalog of patterns for binary operators on type constructors (things like `Future`, `Option`, ZIO `Task`). These patterns are named after the algebraic laws they satisfy (associativity, commutativity, and identity) and the structure they produce, whether a tuple or an either.

Design principles behind ZIO Prelude:

1. **Radical** — So basically it ignores all dogma, and it is completely written with a new mindset.
2. **Orthogonality** — The goal for ZIO Prelude is to have no overlap. Type classes should do one thing and fit it well. So there is not any duplication to describe type classes.
3. **Principled** — All type classes in ZIO Prelude include a set of laws that instances must obey.
4. **Pragmatic** — If we have data types that don't satisfy laws but that are still useful to use in most cases, we can go ahead and provide instances for them.
5. **Scala-First** - It embraces subtyping and benefit from object-oriented features of Scala.

ZIO Prelude gives us:

- **[Functional Data Types](functional-data-types/index.md)**— Additional data types to supplement the ones in the Scala standard library such as `Validation` and `NonEmptyList` to enable more accurate domain modeling and handle common problems like data validation. For example:
  - `NonEmptyList`, `NonEmptySet`
  - `ZSet`, `ZNonEmptySet`
  - `Validation`, `ZValidation`
- **[Functional Abstractions](functional-abstractions/index.md)**— Functional abstractions to describe different ways of combining data, making it easy for us to combine complex data types in a principled way.
- **[New Types](newtypes/index.md)**— that allow to _increase type safety_ in domain modeling. Wrapping existing type adding no runtime overhead. These refined newtypes allow us to increase the type safety of our code base with zero overhead and minimal boilerplate.
- **[ZPure](zpure/index.md)**— A description of a computation that supports logging, context, state, and errors, providing all the functionality traditionally offered by monad transformers with dramatically better performance and ergonomics.

The library has a small research-stage package (`zio.prelude.fx`) that provides abstraction over expressive effect types like ZIO and `ZPure`.

ZIO Prelude is a library focused on providing a core set of functional data types and abstractions that can help you solve a variety of day to day problems. The tools provided by ZIO Prelude fall into the following main categories:

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-prelude" % "1.0.0-RC44"
```

## Example

In this example, we are going to create a simple voting application. We will use two features of ZIO Prelude:
1. To become more type safety we are going to use _New Types_ and introducing `Topic` and `Votes` data types.
2. Providing instance of `Associative` type class for `Votes` data type which helps us to combine `Votes` values.

```scala

object VotingExample extends scala.App {

  object Votes extends Subtype[Int] {
    implicit val associativeVotes: Associative[Votes] =
      new Associative[Votes] {
        override def combine(l: => Votes, r: => Votes): Votes =
          Votes(l + r)
      }
  }
  type Votes = Votes.Type

  object Topic extends Subtype[String]
  type Topic = Topic.Type

  final case class VoteState(map: Map[Topic, Votes]) { self =>
    def combine(that: VoteState): VoteState =
      VoteState(self.map combine that.map)
  }

  val zioHttp    = Topic("zio-http")
  val uziHttp    = Topic("uzi-http")
  val zioTlsHttp = Topic("zio-tls-http")

  val leftVotes  = VoteState(Map(zioHttp -> Votes(4), uziHttp -> Votes(2)))
  val rightVotes = VoteState(Map(zioHttp -> Votes(2), zioTlsHttp -> Votes(2)))

  println(leftVotes combine rightVotes)
  // Output: VoteState(Map(zio-http -> 6, uzi-http -> 2, zio-tls-http -> 2))
}
```

---

## New Types

ZIO Prelude provides functionality for creating zero overhead new types to allow you to increase the type safety of your application without compromising performance or ergonomics.

A common situation in domain modeling is that we have multiple types in our business domain with the same underlying representation in Scala. For example:

```scala
type AccountNumber  = Int
type SequenceNumber = Int
```

This is not a great situation. We can easily mix up values that represent different types in our business domain, for example providing an `AccountNumber` where we are supposed to use a `SequenceNumber` or vice versa.

The Scala compiler will not help us avoid bugs in these situations because from the perspective of the compiler `AccountNumber` and `SequenceNumber` are the same type, namely `Int`.

To avoid this, we might be tempted to create a separate case class for each of these types like this:

```scala
final case class AccountNumber(value: Int)
final case class SequenceNumber(value: Int)
```

This is an improvement in type safety because `AccountNumber` and `SequenceNumber` are now separate types so providing an `AccountNumber` when we are supposed to use a `SequenceNumber` will result in a compilation error.

However, it comes with costs of its own. Every one of these account numbers and sequence numbers will now allocate an additional object, which can add up over the course of a large application with a complex domain model.

We can try to minimize these allocations using techniques such as extending `AnyVal`, but these approaches can be extremely fragile and can actually result in worse performance than the original code if we are not careful.

Let's see how ZIO Prelude helps us get the best of both worlds here.

## New Types

A new type in ZIO Prelude is a type that has the same underlying representation as another type at runtime but is a separate type at compile time.

We create new types by creating an object that extends `Newtype` and specifying the type that our new type wraps. We then export that type by defining a new type alias that refers to the `Type` within our new object.

```scala

object AccountNumber extends Newtype[Int]
type AccountNumber = AccountNumber.Type

object SequenceNumber extends Newtype[Int]
type SequenceNumber = SequenceNumber.Type
```

The `AccountNumber` and `SequenceNumber` types are now completely distinct from the `Int` type at compile time, even though at runtime they are all `Int` values. 

```scala
val accountNumber: AccountNumber =
  AccountNumber(1)
// accountNumber: AccountNumber = 1

val sequenceNumber: SequenceNumber =
  SequenceNumber(2)
// sequenceNumber: SequenceNumber = 2

def lookup(accountNumber: AccountNumber, sequenceNumber: SequenceNumber) =
  ???
```

We can see this if we try to supply an `AccountNumber` somewhere that a `SequenceNumber` is expected:

```scala
lookup(sequenceNumber, accountNumber)
// error: type mismatch;
//  found   : SequenceNumber
//     (which expands to)  SequenceNumber.Type
//  required: AccountNumber
//     (which expands to)  AccountNumber.Type
// error: type mismatch;
//  found   : AccountNumber
//     (which expands to)  AccountNumber.Type
//  required: SequenceNumber
//     (which expands to)  SequenceNumber.Type
// lookup(sequenceNumber, accountNumber)
//                        ^^^^^^^^^^^^^
```

This is great but how do we construct `AccountNumber` and `SequenceNumber` values and how do we access the underlying `Int` values to work with them?

## Constructing New Types

The easiest way to construct an instance of the new type is to use the `apply` method on the new type object.

```scala
val accountNumber: AccountNumber =
  AccountNumber(1)
// accountNumber: AccountNumber = 1
```

You can think of the new type object as being the "companion object" of the new type and it comes included with an `apply` method to make it easy for us to construct instances of the new type.

If you want to construct a collection of values of the new type from values of the underlying type you can use the `wrapAll` operator.

```scala

val accountNumbers: Chunk[AccountNumber] =
  AccountNumber.wrapAll(Chunk(3, 4, 5))
// accountNumbers: Chunk[AccountNumber] = IndexedSeq(3, 4, 5)
```

The `wrapAll` operator is extremely efficient because it does not have to traverse the collection at all. Internally we know that `AccountNumber` and `Int` are the same type so we can freely convert a `Chunk[Int]` to a `Chunk[AccountNumber]` without having to traverse the collection.

We can also define our own more specialized constructors for our new type in terms of `apply`, `wrap`, and `wrapAll`. The new type object provides a convenient place for us to do this.

For example, we could define a simple constructor for an initial sequence number like this:

```scala

object SequenceNumber extends Newtype[Int] {
  val initial: SequenceNumber =
    SequenceNumber(0)
}
type SequenceNumber = SequenceNumber.Type

val initial: SequenceNumber =
  SequenceNumber.initial
// initial: SequenceNumber = 0
```

This lets us construct instances of the new type in a very ergonomic way, where constructors for the new type are on the new type companion object just like constructors for any other type would be.

## Deconstructing New Types

In addition to constructing instances of the new type we want to be able to access the value underlying the new type to perform operations on it.

We can do this using the `unwrap` operator on the new type companion object.

```scala
val zero: Int =
  SequenceNumber.unwrap(initial)
// zero: Int = 0
```

We can also use the `unwrapAll` operator to convert a collection of values of the new type into the underlying type.

```scala

val sequenceNumbers: Chunk[SequenceNumber] =
  SequenceNumber.wrapAll(Chunk(1, 2, 3))
// sequenceNumbers: Chunk[SequenceNumber] = IndexedSeq(1, 2, 3)

val ints: Chunk[Int] =
  SequenceNumber.unwrapAll(sequenceNumbers)
// ints: Chunk[Int] = IndexedSeq(1, 2, 3)
```

Again this is extremely efficient because we don't have to traverse the collection at all. Internally we know that a `Chunk[SequenceNumber]` is a `Chunk[Int]` so we can just immediately return the original collection unchanged.

## Defining Operators On New Types

By wrapping and unwrapping values of the new type we can define our own operators on new types.

The easiest way to do this is by defining our operators as extension methods inside the new type companion object. These extension methods will automatically be available when we are working with instances of the new type.

```scala

object SequenceNumber extends Newtype[Int] {

  implicit class SequenceNumberSyntax(private val self: SequenceNumber) extends AnyVal {
    def next: SequenceNumber =
      SequenceNumber.wrap(SequenceNumber.unwrap(self) + 1)
  }
}
type SequenceNumber = SequenceNumber.Type

val sequenceNumber: SequenceNumber =
  SequenceNumber(1)
// sequenceNumber: SequenceNumber = 1

val nextSequenceNumber: SequenceNumber =
  sequenceNumber.next
// nextSequenceNumber: SequenceNumber = 2
```

Notice that the `next` operator was available on `SequenceNumber` without having to do any imports because we defined it as an extension method within the `SequenceNumber` object.

The new type companion object is also a convenient place for other operators that we would normally define on the companion object of a type. For example, we could define a `continuous` operator that returns whether a collection of `SequenceNumber` values are continuous.

```scala

object SequenceNumber extends Newtype[Int] {
  def continuous(sequenceNumbers: Iterable[SequenceNumber]): Boolean =
    SequenceNumber.unwrapAll(sequenceNumbers).foldLeft[(Option[Int], Boolean)]((None, true)) {
      case ((None, continuous), current) => (Some(current), continuous)
      case ((Some(previous), continuous), current) => (Some(current), continuous && current == previous + 1)
    }._2
}
type SequenceNumber = SequenceNumber.Type

val sequenceNumbers: Chunk[SequenceNumber] =
  Chunk(SequenceNumber(1), SequenceNumber(2), SequenceNumber(3))
// sequenceNumbers: Chunk[SequenceNumber] = IndexedSeq(1, 2, 3)

val continuous: Boolean =
  SequenceNumber.continuous(sequenceNumbers)
// continuous: Boolean = true
```

Here `continuous` is an operator on a collection of sequence numbers so it didn't make sense to define it as an extension method on an individual sequence number. Defining it in the new type companion object makes it feel like an operator on the companion object of any ordinary data type.

## Defining Instances Of Functional Abstractions For New Types

We can also define instances of functional abstractions for new types such as `Equal`, `Ord`, and `Associative`.

The new type companion object again provides a convenient place to define these instances and ensures that they will automatically be available so that we don't have to import them.

Let's see how we can do this to define an `Equal` instance for `AccountNumber`:

```scala

object AccountNumber extends Newtype[Int] {
  implicit val AccountNumberEqual: Equal[AccountNumber] =
    Equal.default
}
type AccountNumber = AccountNumber.Type

AccountNumber(1) === AccountNumber(1)
// res4: Boolean = true
AccountNumber(1) === AccountNumber(2)
// res5: Boolean = false
```

Attempting to compare two unrelated types results in a compilation error.

```scala
AccountNumber(1) === 1
```

This can be particularly useful because since the representations of the new type and the underlying types are the same at runtime, operators that are not strongly typed like `println` and `==` will not respect the difference between the new type and the underlying type. Using the functional abstractions in ZIO Prelude we can avoid this problem.

## Subtypes

When we use `Newtype` the new type we are creating is completely distinct from the underlying type. This can be useful when we want to "hide" the underlying representation of the new type except when we explicitly unwrap it.

However, it can lead to boilerplate in some cases where we need to explicitly unwrap the new type or reimplement functionality that already exists on the underlying type.

For example, say we want to test whether one `SequenceNumber` is after another. Right now this will not work:

```scala

object SequenceNumber extends Newtype[Int]
type SequenceNumber = SequenceNumber.Type

SequenceNumber(2) > SequenceNumber(1)
```

The `>` operator is defined on `Int` and as far as the Scala compiler is concerned `SequenceNumber` and `Int` are completely unrelated types. Of course we could unwrap each of our sequence numbers or define a new `>` operator on sequence numbers but here we are not taking advantage of the fact that we know that every `SequenceNumber` is an `Int`.

We can fix this by extending `Subtype` instead of `Newtype` when creating our new type.

```scala

object SequenceNumber extends Subtype[Int]
type SequenceNumber = SequenceNumber.Type

SequenceNumber(2) > SequenceNumber(1)
// res7: Boolean = true
```

Now `SequenceNumber` is a type that is different from `Int` but is still a subtype of `Int`.

This means that we can use a `SequenceNumber` any time we need an `Int` and can use operators defined on `Int` on `SequenceNumber`. However, we still get the type safety of not being able to use an `Int` or `AccountNumber` when a `SequenceNumber` is expected.

## Smart Newtypes

So far, all the new types we have created have been distinct from the underlying types but have not imposed any additional constraints on the values that the underlying type can take. For example, the `SequenceNumber` above could in principle be any `Int` value, whether that is `42` or `-1`.

In many cases that makes sense. We often want to use a new type to denote that this particular `Int` value represents a sequence number and shouldn't be confused with another `Int` value that represents an account number without imposing other constraints.

However, in other cases we may want to restrict the values that the underlying type can take. For instance, perhaps a `SequenceNumber` should never be negative.

We can model this in ZIO Prelude by extending `Newtype` and `Subtype`, just as before, and then defining an additional `def assertion` method that describes the constraints on the underlying value. (The syntax differs slightly between Scala 2 and 3 due to changes in the macro API).

```scala

object SequenceNumber extends Subtype[Int] {

  // Scala 2
  override def assertion = assert { 
    greaterThanOrEqualTo(0)
  }
  
  // Scala 3
  // override inline def assertion = 
  //  greaterThanOrEqualTo(0)
}
type SequenceNumber = SequenceNumber.Type
```

Here we created a simple assertion that requires the value be equal to or greater than zero, but we can use much more complex assertions. For example, we could validate an `Email` with the `matches` assertion, which accept a `Regex`.

Now, when we construct new values using `apply`, they will be validated *at compile time*.

```scala
val valid1: SequenceNumber = SequenceNumber(0)
val valid2: SequenceNumber = SequenceNumber(42)

val oops = SequenceNumber(-10)
// Newtype Assertion Failed 
// • -10 did not satisfy greaterThanOrEqualTo(0)
```

Smart Newtypes can only be validated at compile-time when called with literals, such as `9000` or `"Fancy Pants"`. Attempting to wrap a variable or run-time value with `apply` will result in a compilation error warning you about this.

When wrapping variables or run-time values, you can use the `make` or `makeAll` constructors, which will perform the validation at run-time instead.

```scala

val a = 10
// a: Int = 10
val b = 11
// b: Int = 11
val c = 12
// c: Int = 12

val validatedSequenceNumber: Validation[String, SequenceNumber] =
  SequenceNumber.make(a)
// validatedSequenceNumber: Validation[String, SequenceNumber] = Success(
//   log = IndexedSeq(),
//   value = 10
// )

val validateSequenceNumbers: Validation[String, Chunk[SequenceNumber]] =
  SequenceNumber.makeAll(Chunk(a, b, c))
// validateSequenceNumbers: Validation[String, Chunk[SequenceNumber]] = Success(
//   log = IndexedSeq(),
//   value = IndexedSeq(10, 11, 12)
// )
```

We can see that `make` and `makeAll` return a `Validation` that will either be a success with a valid instance of the newtype or a failure with a helpful error message indicating why a valid instance of the newtype could not be constructed.

Note that the `wrap` operator, which bypasses the compile-time check, is `protected` and will only be accessible within the scope of the newtype object unless we choose to expose them. This allows us to skip the compile-time check when we need to, such as in implementing a `.next` method, which we know will be safe, even if the compiler does not.

```scala

object SequenceNumber extends Subtype[Int] {
  override def assertion = assert { 
    greaterThanOrEqualTo(0)
  }
  
  val initial: SequenceNumber =
    SequenceNumber(0)
    
  implicit final class SequenceNumberOps(val self: SequenceNumber) extends AnyVal {
    def next: SequenceNumber = 
      wrap(self + 1)
  }
}
type SequenceNumber = SequenceNumber.Type
```

As another example, we might want to expose an `unsafeMake` operator for our users to allow them to construct instances of the newtype directly without going through `Validation`. As its name implies this places responsibility on the user to ensure that the underlying data is valid but could be necessary for performance or ergonomics in some situations.

We can do that quite easily like this:.

```scala

object SequenceNumber extends Subtype[Int] {
  override def assertion = assert { 
    greaterThanOrEqualTo(0)
  }

  def unsafeMake(n: Int): SequenceNumber =
    SequenceNumber.wrap(n)
}
type SequenceNumber = SequenceNumber.Type

val aTrustedInt = 1
// aTrustedInt: Int = 1
val sequenceNumber: SequenceNumber =
  SequenceNumber.unsafeMake(aTrustedInt)
// sequenceNumber: SequenceNumber = 1
```

Thus, Smart Newtypes give us full ability to implement our own operators and to expose whatever interface we want for our type, from validating input at compile-time, to using `Validation` at run-time, to allowing users to create instances of the refined newtype directly.

---

## ZIO Prelude Resources

- [Refactoring Functional Type Classes](https://www.slideshare.net/jdegoes/refactoring-functional-type-classes) - slides by John De Goes and Adam Fraser,
- [SF Scala: Reimagining Functional Type Classes](https://youtu.be/OwmHgL9F_9Q) - a talk by John A. De Goes and Adam Fraser
- [Exploring ZIO Prelude The game-changer for type classes in Scala](https://youtu.be/OzoMofqsPg8) - talk by Jorge Vásquez
- [What's Ap with ZIO Prelude?](https://justinhj.github.io/2020/08/02/whats-ap-with-zio-prelude.html) - an article by justinhj
- [SF Scala: Reimagining Functional Type Classes](https://www.youtube.com/watch?v=OwmHgL9F_9Q) John A. De Goes and Adam Fraser (August 2020) — In this presentation, John A. De Goes and Adam Fraser introduce a new Scala library with a completely different factoring of functional type classes—one which throws literally everything away and starts from a clean slate. In this new factoring, type classes leverage Scala’s strengths, including variance and modularity. Pieces fit together cleanly and uniformly, and in a way that satisfies existing use cases, but enables new ones never before possible. Finally, type classes are named, organized, and described in a way that makes teaching them easier, without compromising on algebraic principles.
- [The Terror-Free Guide To Introducing Functional Scala At Work](https://www.youtube.com/watch?v=Sinde_P7nmY) by Jorge Vasquez (December 2020) — Too often, our applications are dominated by boilerplate that's not fun to write or test, and that makes our business logic complicated. In object-oriented programming, classes and interfaces help us with abstraction to reduce boilerplate. But, in functional programming, we use type classes. Historically, type classes in functional programming have been very complex and confusing, partially because they import ideas from Haskell that don't make sense in Scala, and partially because of their esoteric origins in category theory. In this presentation, Jorge Vásquez presents a new library called ZIO Prelude, which offers a distinctly Scala take on Functional Abstractions, and you will learn how you can eliminate common types of boilerplate by using it. Come see how you can improve your happiness and productivity with a new take on what it means to do functional programming in Scala!
- [The Terror-Free Guide to Introducing Functional Scala at Work](https://www.slideshare.net/jv2301/the-terrorfree-guide-to-introducing-functional-scala-at-work) - slides by Jorge Vásquez,
- [ZIO WORLD - ZIO Prelude](https://www.youtube.com/watch?v=69ngoqVXKPI) by Jorge Vasquez (March 2020) — In this talk, Jorge Vasques discusses his work bringing refined newtypes to ZIO Prelude, which are working natively on Scala 3 with a beautiful syntax and DSL.
- [Zymposium - ZIO Prelude](https://www.youtube.com/watch?v=M3HmROwOoRU) by Adam Fraser and Kit Langton (May 2021) — We'll see how ZIO Prelude gives us the tools for solving some common problems in day-to-day development. We'll also see how ZIO Prelude provides a set of abstractions we can use for inspiration when implementing our own data types but never forces us to use these abstractions.
- [Zymposium - Prelude Redux (Type-classes without Type-classes)](https://www.youtube.com/watch?v=97Yc0Ub9aZ8) by Adam and Kit Langton (May 2021) — We will see how thinking in terms of producers and consumers of values can give us powerful insights into the structure of our programs and how we can use these to develop composable operators for own data types, regardless of whether or not we choose to depend on a library like ZIO Prelude.

---

## ZPure

A `ZPure[W, S1, S2, R, E, A]` is a description of a computation that requires an environment `R` and an initial state `S1` and either fails with an error of type `E` or succeeds with an updated state of type `S2` and a value of type `A`, in either case also producing a log of type `W`.

Conceptually, we can think of `ZPure` as just a function, though `ZPure` is not actually implemented this way for reasons of efficiency and stack safety.

```scala

case class ZPure[+W, -S1, +S2, -R, +E, +A](run: (R, S1) => (Chunk[W], Either[E, (S2, A)]))
```

The `ZPure` data type models four "capabilities" that a computation can have in addition to just producing a value of type `A`:

- **Errors** - A `ZPure` computation can fail with an error of type `E` similar to an `Either`.
- **Context** - A `ZPure` computation can require some environment of type `R` similar to a `Reader` data type.
- **State** - A `ZPure` computation can update a state `S1` to a new state `S2` similar to a `State` data type.
- **Logging** - A `ZPure` computation can maintain a log of type `W` similar to a `Writer` data type.

Previously, each of these capabilities required a separate data type, such as `Either` for modeling errors.

This was fine when we just needed one of these capabilities but it broke down when we wanted to describe computations that used more than one capability. How were we supposed to describe, for example, a computation that updated state and could also fail?

The answer was to use other data types called _monad transformers_ which "stacked" these capabilities on top of existing data types. So to get state and errors we would stack an `EitherT` monad transformer on top of a `StateT` monad transformer.

This approach has a number of problems.

First, it has terrible performance. Each of these monad transformers is its own data type so evaluating a single step in a monad transformer stack requires digging into multiple layers of data structures just to access a single value, doing something with it, and then rebuilding all the layers of that nested data structure all over again.

Second, monad transformers have very poor ergonomics. They are generally invariant so they have bad type inference and working with these nested structures creates many opportunities for errors, such as constructing the "stack" in the wrong order, that reduce developer productivity and make it harder to onboard new developers.

Third, monad transformer stacks are actually not powerful enough to model what they are trying to describe. Each part of the "stack" only knows about itself and not the other parts of the stack so it is impossible to implement operators that require multiple capabilities.

`ZPure` addresses this problem by including all of these capabilities in one data type using the same technique of "effect rotation" pioneered in ZIO. This results in dramatically higher performance, improved ergonomics, and a much more powerful API.

## Basic Operations

The `ZPure` data type exposes a very similar API to the `ZIO` data type, excluding operators that deal directly with IO or concurrency. So if you know how to use `ZIO` you should be very comfortable working with `ZPure` as well.

Just like `ZIO`, `ZPure` is a description of a computation so the computation will not actually be executed until we run it.

If we are not embedding side effecting code in `ZPure` this laziness will not be observable in the result of a `ZPure` computation but it can be important to keep in mind for performance. For example, if we are doing some expensive but not side effecting computation in the context of `ZPure` and using the result multiple times we want to only run the `ZPure` computation once instead of multiple times.

The operators that execute a `ZPure` computation are all variants of `run` to emphasize that when we call them we are actually running the computation described by the `ZPure` value.

Also like `ZIO`, `ZPure` is stack safe, so we can write recursive programs in terms of `ZPure` without worrying about stack overflow errors.

The main difference between `ZIO` and `ZPure` is that we do not want to embed arbitrary side effecting code in `ZPure`. `ZPure` supports a specific set of capabilities but if we have arbitrary effects we are better off using `ZIO`, which has the power to manage these effects and also more clearly signals our intent.

We can create a computation that succeeds with an existing value using the `succeed` operator.

```scala

val one: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] =
  ZPure.succeed(1)
// one: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] = Succeed(value = 1)
```

Note that the log type `W` here is nothing, indicating that we are not logging anything right now. The `S1`and `S2` types are `Unit` because we are not using `State` in this computation.

The environment type is `Any`, indicating that this computation does not require any environment to be run. And the error type is `Nothing` because this computation cannot fail.

If we find ourselves using the same type parameters over and over it can be helpful to define a type alias to avoid repeating ourselves. There are also several useful type aliases for `ZPure` defined directly in ZIO Prelude.

```scala
type State[S, +A] = zio.prelude.fx.ZPure[Nothing, S, S, Any, Nothing, A]
type Reader[-R, +A] = zio.prelude.fx.ZPure[Nothing, Unit, Unit, R, Nothing, A]
type Writer[+W, +A] = zio.prelude.fx.ZPure[W, Unit, Unit, Any, Nothing, A]
```

The `State` type alias specializes the environment type to `Any` and the error and log types to `Nothing` and just has a single state type 'A'. It is analogous to a function `S => (S, A)`.

The `Reader` type alias has an environment type but specializes the log and error types to `Nothing` and the state types to `Unit`. It is analogous to a function `R => A`.

The `Writer` type alias has a log type but specializes the environment type to `Any`, the error type to `Nothing`, and the state types to `Unit`. It is analogous to a function `() => (W, A)`.

There are also versions of each of these type aliases that support failure, `EState`, `EReader`, and `EWriter`. Using these type aliases can simplify your type signatures and lead to better type inference in some cases. So if you are really only working with one or two of the capabilities of `ZPure` it is definitely worth using these type aliases.

However, for now we will continue to use the full `ZPure` type signature.

We can transform the successful value of a computation using the `map` operator.

```scala
val two: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] =
  one.map(_ + 1)
// two: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] = FlatMap(
//   value = Succeed(value = 1),
//   continue = zio.prelude.fx.ZPure$$Lambda$17353/0x0000000804bf5840@2e4ad19b
// )
```

We can combine the values of two computations with the `zipWith` operator.

```scala
val three: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] =
  one.zipWith(two)(_ + _)
// three: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] = FlatMap(
//   value = Succeed(value = 1),
//   continue = zio.prelude.fx.ZPure$$Lambda$17361/0x0000000804c08840@1fae5a1c
// )
```

We can also chain computations using the `flatMap` operator and Scala's for comprehension syntax.

```scala
val six: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] =
  for {
    x <- one
    y <- two
    z <- three
  } yield x + y + z
// six: ZPure[Nothing, Unit, Unit, Any, Nothing, Int] = FlatMap(
//   value = Succeed(value = 1),
//   continue = <function1>
// )
```

The easiest way to run a computation is to use the `run` method. It requires us to have already provided any required environment and initial state and handled our errors and just produces the final value.

```scala
val value: Int =
  three.run
// value: Int = 3
```

## Working With Errors

Since `ZPure` also has an error type the operators for working with errors are quite similar to the ones on `ZIO`.

We can create a computation that fails with an error using the `fail` operator.

```scala
val fail: ZPure[Nothing, Unit, Unit, Any, String, Nothing] =
  ZPure.fail("fail")
// fail: ZPure[Nothing, Unit, Unit, Any, String, Nothing] = Fail(
//   error = "fail"
// )
```

Notice how now the error type is `String`, indicating that this computation can fail with a `String` error. In this case the success type is `Nothing` since we actually know that this computation can never succeed.

If we are working with code that can throw exceptions we can use the `attempt` operator of `ZPure` to safely import that code into our computation, catching any non-fatal exceptions and translating them into failure values.

```scala
def parseIntThrowable(s: String): ZPure[Nothing, Unit, Unit, Any, Throwable, Int] =
  ZPure.attempt(s.toInt)
```

We can use the `mapError` operator to transform the error type of a computation, for example to map it to a common error type we are using in other parts of our computation.

```scala
def parseInt(s: String): ZPure[Nothing, Unit, Unit, Any, String, Int] =
  parseIntThrowable(s).mapError(_.getMessage)
```

We can recover from errors by using the `catchAll` operator, which allows us to recover from the failure using a new computation.

```scala
def parseIntOrDefault(s: String): ZPure[Nothing, Unit, Unit, Any, Nothing, Int] =
  parseInt(s).catchAll(_ => ZPure.succeed(0))
```

Now the error type is `Nothing`, indicating that we have handled all our errors.

There are many other error handling operators defined on `ZPure`. The `catchSome` operator allows us to only recover from certain errors.

The `fold` and `foldM` operators allow us to handle both the failure and success cases at the same time. And the `orElse` operator allows us to specify a fallback computation that will be run if the original computation fails.

## Working With Context

The environment type `R` is also analogous to the environment type of ZIO.

The main difference is that we tend to use `ZPure` to describe a particular computation in our application rather than as the basis for our entire application architecture. As a result, services tend to be less commonly used in the environment and often the environment just consists of some data.

We also do not use layers to construct the environment. Layers are inherently effectual and we do not want to perform arbitrary effects as part of a `ZPure` computation.

As an example of how we might use the environment type in a `ZPure` computation, we can imagine that our computation describes some logic for working with customer accounts. To perform its logic our computation needs access to various values such as the interest rate to use.

```scala
case class AccountEnvironment(interestRate: Double)
```

We can work with the environment using the same operators we use for `ZIO`, with the caveat described above that we use the more generic environment operators instead of the ones specialized for the module pattern.

We access a service in the environment using the `service` operator, so if we wanted to access the `AccountEnvironment` service we could do it like this:

```scala
val accountEnvironment: ZPure[Nothing, Unit, Unit, AccountEnvironment, Nothing, AccountEnvironment] =
  ZPure.service
// accountEnvironment: ZPure[Nothing, Unit, Unit, AccountEnvironment, Nothing, AccountEnvironment] = Environment(
//   access = zio.prelude.fx.ZPure$ServiceWithPartiallyApplied$$$Lambda$17538/0x0000000804cfc840@27dae542
// )
```

This computation does not use logging or state and cannot fail, but it now depends on an `AccountEnvironment` and just returns the `AccountEnvironment`. Since this computation now succeeds with an `AccountEnvironment` we can use all of our normal operators for transforming success values like `map` and `flatMap` to work with it.

If we just want to do one thing with the environment like get the interest rate we can do this slightly more concisely with the `serviceWith` operator.

```scala
val interestRate: ZPure[Nothing, Unit, Unit, AccountEnvironment, Nothing, Double] =
  ZPure.serviceWith(_.interestRate)
// interestRate: ZPure[Nothing, Unit, Unit, AccountEnvironment, Nothing, Double] = Environment(
//   access = zio.prelude.fx.ZPure$ServiceWithPartiallyApplied$$$Lambda$17538/0x0000000804cfc840@5283d5ad
// )
```

There is also a `serviceWithPure` variant for when we want to perform another computation based on the value from the environment.

```scala
def computeSimpleInterest(balance: Double, days: Int, interestRate: Double): ZPure[Nothing, Unit, Unit, Any, Nothing, Double] =
  ZPure.succeed(balance * days / 365 * interestRate)

def accruedInterest(balance: Double, days: Int): ZPure[Nothing, Unit, Unit, AccountEnvironment, Nothing, Double] =
  ZPure.serviceWithPure(r => computeSimpleInterest(balance, days, r.interestRate))
```

To run a computation we need to provide it with its required environment, which we can do with the `provide` operator.

```scala

val interestComputation: ZPure[Nothing, Unit, Unit, Any, Nothing, Double] =
  accruedInterest(100000, 30).provideEnvironment(ZEnvironment(AccountEnvironment(0.05)))
// interestComputation: ZPure[Nothing, Unit, Unit, Any, Nothing, Double] = Provide(
//   r = ZEnvironment(MdocSession::MdocApp0::AccountEnvironment -> AccountEnvironment(0.05)),
//   continue = FlatMap(
//     value = Environment(
//       access = zio.prelude.fx.ZPure$ServiceWithPurePartiallyApplied$$$Lambda$17539/0x0000000804cff840@50dcdff4
//     ),
//     continue = generalized constraint
//   )
// )
```

Once we have provided our application with our required environment we are ready ro run it.

```scala
val interestDue: Double =
  interestComputation.run
// interestDue: Double = 410.958904109589
```

## Working With State

The state type described by `S1` and `S2` is the first capability provided by `ZPure` that is not directly analogous to `ZIO`.

In `ZIO` we manage state using the `Ref` and `FiberRef` data types so we do not need these additional type parameters. In contrast, in `ZPure` we want to avoid arbitrary side effects so we instead model state as a function that takes the old state and returns the new state using the `S1` and `S2` type parameters.

In practice in most cases `S1` and `S2` will be the same and we will be describing a computation that takes some state and just returns an updated value of the same state type. However, in some cases it can be useful to model the initial and updated state separately, for example to describe a transition from a `Closed` to `Open` state in a type safe way.

To motivate our example here let's define an `AccountState` type.

```scala
case class AccountState(balance: Int, open: Boolean)
```

We will also define a domain specific error type which we will use.

```scala
sealed trait AccountError

case object InsufficientFunds extends AccountError
```

The most basic operators for working with state are `get` and `set`. The `get` operator retrieves the initial state and the `set` operator sets the updated state to the specified value.

Let's use these operators to describe a computation that withdraws funds from the account.

```scala
def withdraw(amount: Int): ZPure[Nothing, AccountState, AccountState, Any, AccountError, Unit] =
  for {
    state <- ZPure.get[AccountState]
    _     <- if (amount > state.balance) ZPure.fail(InsufficientFunds)
             else ZPure.set(AccountState(state.balance - amount, state.open))
  } yield ()
```

This example combines a couple of the features we have talked about so far.

First, we are using a for comprehension to chain together multiple computations

Second, we are using both the error type and the state type here. If the customer has insufficient funds we fail immediately, otherwise we update the state.

One important thing to note here is that since `ZPure` computations are never concurrent we don't need to worry about conflicting modifications to the state.

For example, in the context of `ZIO` code analogous to the above would not be safe because the `get` and `set` would not be performed atomically so we would need to do the entire transaction in a single `modify` operation, returning a value indicating whether there were sufficient funds. In `ZPure` we don't have to do any of that so our code can be simpler.

There are also `update` and `modify` operators on `ZPure`. So for example we could factor out the logic of updating the account balance in the `withdraw` method like this:

```scala
def decrementBalance(amount: Int): ZPure[Nothing, AccountState, AccountState, Any, Nothing, Unit] =
  ZPure.update(state => AccountState(state.balance - amount, state.open))
```

If we are working with state then to actually run our computation we need to provide it with an initial state. The easiest way to do this is with the `provideState` operator, analogous to how we provide the environment.

```scala
val withdrawalComputation: ZPure[Nothing, Any, AccountState, Any, AccountError, Unit] =
  withdraw(10).provideState(AccountState(100, true))
// withdrawalComputation: ZPure[Nothing, Any, AccountState, Any, AccountError, Unit] = FlatMap(
//   value = Modify(
//     run0 = zio.prelude.fx.ZPure$$$Lambda$17546/0x0000000804d11840@16b4a018
//   ),
//   continue = zio.prelude.fx.ZPure$$Lambda$17547/0x0000000804d12840@2e693c8b
// )
```

We will use the `runEither` operator, which is a variant of `run` that allows the computation to fail and just converts the failure to the left side of an `Either`. The `run` and `runEither` operators just return the final value, so let's use the `get` operator to get the final state before we run our computation since we only care about the final state.

```scala moc
val updatedAccountState: Either[AccountError, AccountState] =
  (withdrawalComputation *> ZPure.get).runEither
```

## Working With Logging

The final capability provided by `ZPure` is logging.

We can add to the log with the `log` operator on the `ZPure` companion object.

```scala
def withdrawLog(amount: Int): ZPure[String, AccountState, AccountState, Any, AccountError, Unit] =
  ZPure.log("Attempting to withdraw") *> withdraw(amount) <* ZPure.log(s"Withdrew $amount")
```

We can also use the `log` operator on the `ZPure` trait to log something immediately after a computation has successfully completed.

```scala
def withdrawLog(amount: Int): ZPure[String, AccountState, AccountState, Any, AccountError, Unit] =
  withdraw(amount).log(s"Withdrew $amount")
```

The log is maintained until our computation is done.

This time we will use the `runAll` operator to run our computation. `runAll` is the most general way of running a `ZPure` computation and returns the log as well as either the failure or the success.

```scala

val withdrawalComputationLog: ZPure[String, AccountState, AccountState, Any, AccountError, Unit] =
  withdrawLog(10)
// withdrawalComputationLog: ZPure[String, AccountState, AccountState, Any, AccountError, Unit] = FlatMap(
//   value = FlatMap(
//     value = Modify(
//       run0 = zio.prelude.fx.ZPure$$$Lambda$17545/0x0000000804d11040@43038722
//     ),
//     continue = <function1>
//   ),
//   continue = zio.prelude.fx.ZPure$$Lambda$17548/0x0000000804d13040@70044b3a
// )

val log: Chunk[String] =
  withdrawalComputationLog.runAll(AccountState(100, true))._1
// log: Chunk[String] = IndexedSeq("Withdrew 10")
```

---

## Basics

To build a description of a command:

```scala
val command = Command("cat", "file.txt")
```

`command.run` will return a handle to the process as `ZIO[Blocking, CommandError, Process]`. Alternatively, instead of
flat-mapping and calling methods on `Process`, there are convenience methods on `Command` itself for some common operations:

## Transforming output

### List of lines

To obtain the output as a list of lines with the type `ZIO[Blocking, CommandError, Chunk[String]]`

```scala
command.lines
```

### Stream of lines

To obtain the output as a stream of lines with the type `ZStream[Blocking, CommandError, String]`

```scala
command.linesStream
```

This is particularly useful when dealing with large files and so on as to not use an unbounded amount of memory.

### String

If you don't need a structured type, you can return the entire output as a plain string:

```scala
command.string
```

This defaults to UTF-8. To use a different encoding, specify the charset:

```scala
command.string(StandardCharsets.UTF_16)
```

### Exit code

When you don't care about the output (or there is no output), you can return just the exit code.

```scala
command.exitCode
```

Note that `Command#exitCode` will return the exit code in the ZIO's success channel whether it's 0 or not.
If you want non-zero exit codes to be considered an error, use `Command#successfulExitCode` instead. This will
return a `CommandError.NonZeroErrorCode` in ZIO's error channel when the exit code is not 0:

```scala
for {
  exitCode  <- Command("java", "--non-existent-flag").successfulExitCode
  // Won't reach this 2nd command since the previous command failed with `CommandError.NonZeroErrorCode`:
  exitCode2 <- Command("java", "--non-existent-flag").successfulExitCode
} yield ()
```

### Kill a process

You can kill a process by calling `interrupt` on the running `Fiber`:

```scala
for {
  fiber <- Command("long-running-process").exitCode.forkDaemon
  _     <- ZIO.sleep(5.seconds)
  _     <- fiber.interrupt
  _     <- fiber.join
} yield ()
```

If you use `Command#run` then you receive a handle to underlying `Process` immediately, which means ZIO's built-in
interruption model no longer applies. In this case, if you want to kill a process before it's done terminating,
you can use `kill` (the Unix SIGTERM equivalent) or `killForcibly` (the Unix SIGKILL equivalent):

```scala
for {
  process <- Command("long-running-process").run
  _       <- ZIO.sleep(5.seconds)
  _       <- process.kill
} yield ()
```

### Stream of bytes

If you need lower-level access to the output's stream of bytes, you can access them directly like so:

```scala
command.stream
```

### Access stdout and stderr separately

There are times when you need to process the output of stderr as well.

```scala
for {
  process <- Command("./some-process").run
  stdout  <- process.stdout.string
  stderr  <- process.stderr.string
  // ...
} yield ()
```

## Error handling

Errors are represented as `CommandError` in the error channel instead of `IOException`. Since `CommandError` is an ADT,
you can pattern match on it and handle specific cases rather than trying to parse the guts of `IOException.getMessage`
yourself.

For example, if you want to fallback to running a different program if it doesn't exist on the host machine, you can
match on `CommandError.ProgramNotFound`:

```scala
Command("some-program-that-may-not-exit").string.catchSome {
  case CommandError.ProgramNotFound(_) => Command("fallback-program").string
}
```

---

## Getting Started(Zio-process)

ZIO Process - a purely functional command and process library based on ZIO.
Here's list of contents available:

 - **[Basics](basics.md)** — Creating a description of a command and transforming its output
 - **[Piping](piping.md)** — Creating a pipeline of commands
 - **[Interactive Processes](interactive-processes.md)** — Communicating with an interactive process
 - **[Other](other.md)** — Miscellaneous operations such as settings the working direction, inheriting I/O, etc.

## Installation

Include ZIO Process in your project by adding the following to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-process" % "0.7.2"
```

---

## Introduction to ZIO Process

[ZIO Process](https://zio.dev/zio-process) is a simple ZIO library for interacting with external processes and command-line programs.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-process/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-process_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-process_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-process_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-process_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-process-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-process-docs_2.13) [![ZIO Process](https://img.shields.io/github/stars/zio/zio-process?style=social)](https://github.com/zio/zio-process)

## Introduction

ZIO Process is backed by ZIO Streams, enabling you to work with processes that output gigabytes of data without worrying about exceeding memory constraints.

ZIO Process provides a principled way to call out to external programs from within a ZIO application while leveraging ZIO's capabilities like interruptions and offloading blocking operations to a separate thread pool. We don't need to worry about avoiding these common pitfalls as we would if we were to use Java's `ProcessBuilder` or the `scala.sys.process` API since it is already taken care of for you.

Key features of the ZIO Process:
- **Deep ZIO Integration** — Leverages ZIO to handle interruption and offload blocking operations.
- **ZIO Streams** — ZIO Process is backed by ZIO Streams, which enables us to obtain the command output as streams of bytes or lines. So we can work with processes that output gigabytes of data without worrying about exceeding memory constraints.
- **Descriptive Errors** — In case of command failure, it has a descriptive category of errors.
- **Piping** — It has a simple DSL for piping the output of one command as the input of another.
- **Blocking Operations**

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-process" % "0.7.2" 
```

## Example

Here is a simple example of using ZIO Process:

```scala

object ZIOProcessExample extends ZIOAppDefault {
  val myApp = for {
    fiber <- Command("dmesg", "--follow").linesStream
      .foreach(Console.printLine(_))
      .fork
    cpuModel <- (Command("cat", "/proc/cpuinfo") |
      Command("grep", "model name") |
      Command("head", "-n", "1") |
      Command("cut", "-d", ":", "-f", "2")).string
    _ <- Console.printLine(s"CPU Model: $cpuModel")
    _ <- (Command("pg_dump", "my_database") > new File("dump.sql")).exitCode
    _ <- fiber.join
  } yield ()

  override def run = myApp
}
```

---

## Interactive Processes

Sometimes you want to interact with a process in a back-and-forth manner by sending requests to the process and receiving responses back. For example, interacting with a repl-like process like `node -i`, `python -i`, etc. or an ssh server.

Here is an example of communicating with an interactive NodeJS shell:

```scala
for {
  commandQueue <- Queue.unbounded[Chunk[Byte]]
  process      <- Command("node", "-i").stdin(ProcessInput.fromQueue(commandQueue)).run
  sep          <- System.lineSeparator
  fiber        <- process.stdout.linesStream.foreach { line =>
                    ZIO.debug(s"Response from REPL: $line")
                  }.fork
  _            <- commandQueue.offer(Chunk.fromArray(s"1+1${sep}".getBytes(StandardCharsets.UTF_8)))
  _            <- commandQueue.offer(Chunk.fromArray(s"2**8${sep}".getBytes(StandardCharsets.UTF_8)))
  _            <- commandQueue.offer(Chunk.fromArray(s"process.exit(0)${sep}".getBytes(StandardCharsets.UTF_8)))  
  _            <- fiber.join  
} yield ()
```

You would probably want to create a helper for the repeated code, but this just a minimal example to help get you started.

---

## Other

### Inheriting I/O

If you'd like to run a process and handle its input/output in the current process, you can inherit its I/O. For example, running the Scala REPL:

```scala
Command("scala").inheritIO.exitCode
```

### Providing environment variables

```scala
Command("java", "-version").env(Map("JAVA_HOME" -> javaHome)).string
```

### Specifying the working directory

```scala
Command("ls").workingDirectory(new File("/")).lines
```

### Redirecting output to a file

For example, if you'd like to dump the contents of a PostgreSQL database, you can do the following:

```scala
Command("pg_dump", "my_database").stdout(ProcessOutput.FileRedirect(new File("dump.sql"))).exitCode
```

Alternatively, you can use the bash-like `>` operator if that feels more familiar to you:

```scala
(Command("pg_dump", "my_database") > new File("dump.sql")).exitCode
```

---

## Piping

You can pipe the output of one process as the input to another. For example, if you want to return a list of all running
Java process IDs, you can do the following:

### Manually

```scala
val processes = Command("ps", "-ef").stream
val javaProcesses = Command("grep", "java").stdin(ProcessInput.fromStream(processes)).stream
val processIds = Command("awk", "{print $2}").stdin(ProcessInput.fromStream(javaProcesses)).lines
```

### Using the pipe operator

Rather than connecting the outputs and inputs manually, you can use the `|` operator
(or its named equivalent, `pipe`) like so:

```scala
(Command("ps", "-ef") | Command("grep", "java") | Command("awk", "{print $2}")).lines
```

---

## Introduction to ZIO Profiling

ZIO Profiling is a collection of different profilers for better understanding the runtime behavior of ZIO programs.

[![Concept](https://img.shields.io/badge/Project%20Stage-Concept-orange.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-profiling/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-profiling_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-profiling_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-profiling_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-profiling_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-profiling-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-profiling-docs_2.13) [![ZIO Profiling](https://img.shields.io/github/stars/zio/zio-profiling?style=social)](https://github.com/zio/zio-profiling)

## Introduction

Normal cpu profilers cannot really be used to profile code using an effect system. Profilers operating on a thread level will only see that threads are spending time in the evaluation loop of the effect system, producing profiles that are not useful for application developers.

Instead, profiling a program written using an effect system requires a profiler that is aware of the effect system and can report where the effects were constructed / which user code the effect system is spending time on. ZIO profiling aims to be that library for the ZIO effect system.

The library focuses exclusively on cpu profiling. For heap profiling please consider using other tools such as async-profiler or VisualVM.

## Installation

ZIO Profiling requires you to add both the main library and optionally the compiler plugin to your build.sbt:

```scala
libraryDependencies += "dev.zio" %% "zio-profiling" % "0.3.2"
libraryDependencies += compilerPlugin("dev.zio" %% "zio-profiling-tagging-plugin" % "0.3.2")
```

## Profiling an application and displaying a flamegraph

For this example we are going to use the sampling profiler to measure the cpu time used by parts of a zio program.
All needed definitions can be imported using:
```scala

```

The program we want to instrument simulates performing a short and then a long computation concurrently:
```scala
val fast = ZIO.succeed(Thread.sleep(400))

val slow = ZIO.succeed(Thread.sleep(600))

val program = fast <&> slow
```

In order to profile the program, we wrap it with the `profile` method of the sampling profiler. Once the effect has completed
this will yield the profiling result. We can either manipulate the result in Scala or render it in a number of standard
formats. In this case we are going to write it out in a format supported by https://github.com/brendangregg/FlameGraph, so we
can visualize it as a flamegraph.
```scala
SamplingProfiler
  .profile(program)
  .flatMap(_.stackCollapseToFile("profile.folded"))
```

The resulting file can be converted to a svg using the flamegraph.pl script ([example](img/example_sampling_profile.svg)):
```bash
flamegraph.pl ./examples/profile.folded > profile.svg
```

## Causal Profiling

ZIO Profiling includes experimental support for causal profiling inspired by [coz](https://github.com/plasma-umass/coz).

Usage is similar to the sampling profiler, but instead of displaying the time spent running the program it will give recommendations
which parts of the program to focus on during performance tuning for biggest effect. It achieves this by iteratively artificially speeding
up parts of the program (by slowing down all parts running concurrently) and measuring the effect on overall runtime.

Check out the paper linked in the coz repository for more details about the idea.

We can bring the causal profiler into scope with the following import:
```scala

```

This time we are using a slightly more complicated example:
```scala
val fast = ZIO.succeed(Thread.sleep(40))

val slow1 = ZIO.succeed(Thread.sleep(20))

val slow2 = ZIO.succeed(Thread.sleep(60))

val slow = slow1 <&> slow2

val program = (fast <&> slow) *>
  CausalProfiler.progressPoint("iteration done")

CausalProfiler(iterations = 100)
  .profile(program.forever)
  .flatMap(_.renderToFile("profile.coz"))
```

We also need to weave the `progressPoint` effect into our program. It will be used by the causal profiler to measure progress
of the overall program.

Finally, we can run the program using the causal profiler (notice the use of `program.forever` -- the profiler will automatically interrupt the program, until then it has to keep running)
and save the result to a file.

```scala
CausalProfiler(iterations = 100)
  .profile(prog.forever)
  .flatMap(_.renderToFile("profile.coz"))
```

The file can be viewed using the [Coz Visualizer](https://plasma-umass.org/coz/) ([example](img/example_causal_profile.png)).
As you can see, the profiler correctly tells you that you can get up to a 33% speedup by optimizing the `slow2` effect,
but it's impossible to get a speedup any other way.

## Compiler Plugin

In order to produce actionable output, a profiler not only needs to know which line of code is currently running, but also how that location was reached.

Most profilers rely on the function call hierarchy to determine this information, but the call stack is not really useful for programs using functional effect systems. The reason for this is that the normal function calls are only used to build up the program as a datastructure -- not execute it.

---

It's possible to restore the proper callstack while the effects are actually getting executed (this is the approach taken by the zio.Trace machinery), but that is not the approach taken by ZIO Profiling.

Instead, zio-profiling tracks the current 'callstack' of your program using FiberRefs. For this approach to work every effect should be manually annotated using `zio.profiling.CostCenter.withChildCostCenter`, which will result in a hierarchy of effect tags at runtime.

As this requires modification of large parts of user programs and is bad UX, zio-profiling ships with a compiler plugin (zio-profiling-tagging-plugin) that automates this. Every `def` or `val` that returns a zio effect will be rewritten to be properly tagged. Consider this example for the rewrite that happens:

```scala
val testEffect = ZIO.unit

// gets rewritten to

val testEffect = CostCenter.withChildCostCenter("foo.Foo.testEffect(Foo.scala:12)")(ZIO.unit)
```

To enable the compiler plugin, add the following to the sbt module __containing the code you want to profile__.

```scala
compilerPlugin("dev.zio" %% "zio-profiling-tagging-plugin" % "0.3.2")

```

## Jmh Support

ZIO Profiling offers an integration with the Java Microbenchmark Harness (JMH). In order to profile a jmh benchmark, first ensure that the sources are properly tagged using the tagging plugin. Next, add a dependency to the jmh module to your benchmarking module:
```scala
libraryDependencies += "dev.zio" %% "zio-profiling-jmh" % "0.3.2"
```

In your actual benchmarks, ensure that you are running ZIO effects using the methods in `zio.profiling.jmh.BenchmarkUtils`. A possible benchmark might look like this
```scala
package zio.redis.benchmarks.lists

@State(Scope.Thread)
@BenchmarkMode(Array(Mode.Throughput))
@OutputTimeUnit(TimeUnit.SECONDS)
@Measurement(iterations = 15)
@Warmup(iterations = 15)
@Fork(2)
class BlMoveBenchmarks extends BenchmarkRuntime {

  @Param(Array("500"))
  var count: Int = _

  private var items: List[String] = _

  private val key = "test-list"

  private def execute(query: ZIO[Redis, RedisError, Unit]): Unit =
    BenchmarkUtils.unsafeRun(query.provideLayer(BenchmarkRuntime.Layer))

  @Setup(Level.Trial)
  def setup(): Unit = {
    items = (0 to count).toList.map(_.toString)
    execute(ZIO.serviceWithZIO[Redis](_.rPush(key, items.head, items.tail: _*).unit))
  }

  @TearDown(Level.Trial)
  def tearDown(): Unit =
    execute(ZIO.serviceWithZIO[Redis](_.del(key).unit))

  @Benchmark
  def zio(): Unit = execute(
    ZIO.foreachDiscard(items)(_ =>
      ZIO.serviceWithZIO[Redis](_.blMove(key, key, Side.Left, Side.Right, 1.second).returning[String])
    )
  )
}
```

Once the benchmark is set up properly, you can specify the profiler from the jmh command line. Using sbt-jmh, it might look like this:
```
Jmh/run -i 3 -wi 3 -f1 -t1 -prof zio.profiling.jmh.JmhZioProfiler zio.redis.benchmarks.lists.BlMoveBenchmarks.zio
```

The profiler output will be written to a file in the directory the JVM has been invoked from.

---

## Resources

## Talks

- [Causal Profiling for functional effects](https://youtu.be/rdETYUc8XyI) by Maxim Schuwalow (2021)

## Other

- [Scaladoc](https://zio.github.io/zio-profiling/api/index.html)

---

## Creating Data Sources

To construct a `ZQuery` that executes a request, you first need to create a `DataSource`. A `DataSource[R, A]` requires an environment `R` and is capable of executing requests of type `A`. It is defined in terms of:

- an `identifier` that uniquely identifies the data source
- an effectual function `runAll` from a `Chunk[Chunk[A]]` of requests to a `CompletedRequestMap` of requests and results

The outer `Chunk` represents batches of requests that must be performed sequentially. The inner `Chunk` represents a batch of requests that can be performed in parallel. This allows data sources to introspect on all the requests being executed and optimize the query.

Let's consider `getUserNameById` from the previous example.

We need to define a corresponding request type that extends `Request` for a given response type:

```scala
case class GetUserName(id: Int) extends Request[Throwable, String]
```

Now let's build the corresponding `DataSource`. We will create a `Batched` data source that executes requests that can be performed in parallel in batches but does not further optimize batches of requests that must be performed sequentially. We need to implement the following functions:

```scala
lazy val UserDataSource = new DataSource.Batched[Any, GetUserName] {
  val identifier: String = ???
  def run(requests: Chunk[GetUserName])(implicit trace: Trace): ZIO[Any, Nothing, CompletedRequestMap] = ???
}
```

We will use "UserDataSource" as our identifier. This name should not be reused for other data sources.

```scala
val identifier: String = "UserDataSource"
```

We will define two different behaviors depending on whether we receive a single request or multiple requests at once. For each request, we need to insert into the result map a value of type `Exit` (`fail` for an error and `succeed` for a success).

```scala
def run(requests: Chunk[GetUserName]): ZIO[Any, Nothing, CompletedRequestMap] =
  requests.toList match {
    case request :: Nil =>
      // get user by ID e.g. SELECT name FROM users WHERE id = $id
      val result: Task[String] = ???
      result.exit.map(CompletedRequestMap.single(request, _))
    case batch =>
      // get multiple users at once e.g. SELECT id, name FROM users WHERE id IN ($ids)
      val result: Task[List[(Int, String)]] = ???
      result.foldCause(
        CompletedRequestMap.failCause(requests, _),
        CompletedRequestMap.fromIterableWith(_)(kv => GetUserName(kv._1), kv => Exit.succeed(kv._2))
      )
  }
```

Now to build a `ZQuery`, we can use `ZQuery.fromRequest` and just pass the request and the data source:

```scala
def getUserNameById(id: Int): ZQuery[Any, Throwable, String] =
  ZQuery.fromRequest(GetUserName(id))(UserDataSource)
```

To run a `ZQuery`, simply use `ZQuery#run` which will return a `ZIO[R, E, A]`.

---

## Creating Queries

There are several ways to create a `ZQuery`. We've seen `ZQuery.fromRequest`, but you can also:

- create a query from a pure value with `ZQuery.succeed`
- create a query from an effect with `ZQuery.fromZIO`
- combine multiple queries with `ZQuery.collectAllPar` and `ZQuery.foreachPar` and their sequential equivalents `ZQuery.collectAll` and `ZQuery.foreach`

If you have a `ZQuery`, you can use:

- `map` and `mapError` to modify the result or error type
- `flatMap`, `zipWith`, or `zipWithPar` to combine it with other queries
- `provide` and `provideSome` to eliminate some or all of its environmental requirements

---

## Introduction to ZIO Query

[ZIO Query](https://github.com/zio/zio-query) is a library for writing optimized queries to data sources in a high-level compositional style. It can add efficient pipelining, batching, and caching to any data source. ZIO Query helps us dramatically reduce load on data sources and improve performance.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-query/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-query_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-query_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-query_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-query_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-query-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-query-docs_2.13) [![ZIO Query](https://img.shields.io/github/stars/zio/zio-query?style=social)](https://github.com/zio/zio-query)

## Introduction

Some key features of ZIO Query:

- **Batching** — ZIO Query detects parts of composite queries that can be executed in parallel without changing the semantics of the query.
- **Pipelining** — ZIO Query detects parts of composite queries that can be combined together for fewer individual requests to the data source.
- **Caching** — ZIO Query can transparently cache read queries to minimize the cost of fetching the same item repeatedly in the scope of a query.

Compared with Fetch, ZIO Query supports response types that depend on request types, does not require higher-kinded types and implicits, supports ZIO environment and statically typed errors, and has no dependencies except for ZIO.

A `ZQuery[R, E, A]` is a purely functional description of an effectual query that may contain requests from one or more data sources, requires an environment `R`, and may fail with an `E` or succeed with an `A`.

Requests that can be performed in parallel, as expressed by `zipWithPar` and combinators derived from it, will automatically be batched. Requests that must be performed sequentially, as expressed by `zipWith` and combinators derived from it, will automatically be pipelined. This allows for aggressive data source specific optimizations. Requests can also be deduplicated and cached.

This allows for writing queries in a high level, compositional style, with confidence that they will automatically be optimized. For example, consider the following query from a user service.

Assume we have the following database access layer APIs:

```scala
def getAllUserIds: ZIO[Any, Nothing, List[Int]] = {
  // Get all user IDs e.g. SELECT id FROM users
  ZIO.succeed(???)
}

def getUserNameById(id: Int): ZIO[Any, Nothing, String] = {
  // Get user by ID e.g. SELECT name FROM users WHERE id = $id
  ZIO.succeed(???)
}
```

We can get their corresponding usernames from the database by the following code snippet:

```scala
val userNames = for {
  ids   <- getAllUserIds
  names <- ZIO.foreachPar(ids)(getUserNameById)
} yield names
```

It works, but this is not performant. It is going to query the underlying database _N + 1_ times, one for `getAllUserIds` and one for each call to `getUserNameById`.

In contrast, `ZQuery` will automatically optimize this to two queries, one for `userIds` and one for `userNames`:

```scala
lazy val getAllUserIds: ZQuery[Any, Nothing, List[Int]]    = ???
def getUserNameById(id: Int): ZQuery[Any, Nothing, String] = ???

lazy val userQuery: ZQuery[Any, Nothing, List[String]] = for {
  userIds   <- getAllUserIds
  userNames <- ZQuery.foreachPar(userIds)(getUserNameById)
} yield userNames
```

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-query" % "0.7.7"
```

## Example

Here is an example of using ZIO Query, which optimizes multiple database queries by batching all of them in one query:

```scala

object ZQueryExample extends ZIOAppDefault {
  case class GetUserName(id: Int) extends Request[Throwable, String]

  lazy val UserDataSource: DataSource.Batched[Any, GetUserName] =
    new DataSource.Batched[Any, GetUserName] {
      val identifier: String = "UserDataSource"

      def run(requests: Chunk[GetUserName])(implicit trace: Trace): ZIO[Any, Nothing, CompletedRequestMap] =
        requests.toList match {
          case request :: Nil =>
            val result: Task[String] = {
              // get user by ID e.g. SELECT name FROM users WHERE id = $id
              ZIO.succeed(???)
            }

            result.exit.map(CompletedRequestMap.single(request, _))

          case batch: Seq[GetUserName] =>
            val result: Task[List[(Int, String)]] = {
              // get multiple users at once e.g. SELECT id, name FROM users WHERE id IN ($ids)
              ZIO.succeed(???)
            }

            result.foldCause(
              CompletedRequestMap.failCause(requests, _),
              CompletedRequestMap.fromIterableWith(_)(kv => GetUserName(kv._1), kv => Exit.succeed(kv._2))
            )
        }
    }

  def getUserNameById(id: Int): ZQuery[Any, Throwable, String] =
    ZQuery.fromRequest(GetUserName(id))(UserDataSource)

  val query: ZQuery[Any, Throwable, List[String]] =
    for {
      ids <- ZQuery.succeed(1 to 10)
      names <- ZQuery.foreachPar(ids)(id => getUserNameById(id)).map(_.toList)
    } yield (names)

  def run = query.run.tap(usernames => Console.printLine(s"Usernames: $usernames"))
}
```

## Resources

- [Wicked Fast API Calls with ZIO Query](https://www.youtube.com/watch?v=rUUxDXJMzJo) by Adam Fraser (July 2020) (https://www.youtube.com/watch?v=rUUxDXJMzJo)

---

## Running Queries

There are several ways to run a `ZQuery`:

- `runCache` runs the query using a given pre-populated cache. This can be useful for deterministically "replaying" a query without executing any new requests.
- `runLog` runs the query and returns its result along with the cache containing a complete log of all requests executed and their results. This can be useful for logging or analysis of query execution.
- `run` runs the query and returns its result.

---

## How to Contribute?

Instructions on how to contribute to Quill project.

## Building the project using Docker

The only dependency you need to build Quill locally is [Docker](https://www.docker.com/).
Instructions on how to install Docker can be found [here](https://docs.docker.com/engine/installation/).

If you are running Linux, you should also install Docker Compose separately, as described
[here](https://docs.docker.com/compose/install/).

After installing Docker and Docker Compose you have to setup databases:

```bash
docker compose run --rm setup
```

After that you are ready to build and test the project. The following steps describes how to test the project with
sbt built within docker image. If you would like to use your local sbt,
please visit [Building locally](#building-locally-using-docker-only-for-databases). This is highly recommended
when running Docker on non-linux OS due to high IO overhead from running 
[Docker in virtualized environments](https://docs.docker.com/docker-for-mac/osxfs/#performance-issues-solutions-and-roadmap).

To build and test the project:

```bash
docker compose run --rm sbt sbt test
```

## Building Scala.js targets

The Scala.js targets are disabled by default, use `sbt "project quill-with-js"` to enable them.
The CI build also sets this `project quill-with-js` to force the Scala.js compilation.

## Changing database schema

If any file that creates a database schema was changed then you have to setup the databases again:

```bash
docker compose down && docker compose run --rm setup
```

## Changing docker configuration

If `build/Dockerfile-sbt`, `build/Dockerfile-setup`, `docker-compose.yml` or any file used by them was changed then you have to rebuild docker images and to setup the databases again:

```bash
docker compose down && docker compose build && docker compose run --rm setup
```

## Tests

### Running tests

Run all tests:
```bash
docker compose run --rm sbt sbt test
```

Run specific test:
```bash
docker compose run --rm sbt sbt "test-only io.getquill.context.sql.SqlQuerySpec"
```

Run all tests in specific sub-project:
```bash
docker compose run --rm sbt sbt "project quill-jdbc-zio" test
```

Run specific test in specific sub-project:
```bash
docker compose run --rm sbt sbt "project quill-sqlJVM" "test-only io.getquill.context.sql.SqlQuerySpec"
```

### Debugging tests
1. Run sbt in interactive mode with docker container ports mapped to the host: 
```bash
docker compose run --service-ports --rm sbt
```

2. Attach debugger to port 15005 of your docker host. In IntelliJ IDEA you should create Remote Run/Debug Configuration, 
change it port to 15005.
3. In sbt command line run tests with `test` or test specific spec by passing full name to `test-only`:
```bash
> test-only io.getquill.context.sql.SqlQuerySpec
```

## Pull Request

In order to contribute to the project, just do as follows:

1. Fork the project
2. Build it locally
3. Code
4. Compile (file will be formatted)
5. Run the tests through `docker compose run sbt sbt test`
6. If you made changes in *.md files, run `docker compose run sbt sbt tut` to validate them
7. If everything is ok, commit and push to your fork
8. Create a Pull Request, we'll be glad to review it

## File Formatting 

[scalafmt](https://scalameta.org/scalafmt/) is used as file formatting tool in this project.
Every time you compile the project in sbt, file formatting will be triggered. To manually format
the Quill codebase, run the following command:
```
scalafmt --config .scalafmt.conf
```

## Notes About Git

As you are working on a branch fixing a Quill problem, there will likely be additional changes merged
to the main Quill repo (i.e. getquill/quill) before you finish. The solution to this is to squash your commits, 
and then to replay them on top of the latest changes that have been made to the main Quill repo.

The steps to do that are the following:
1) Squash your commits via an interactive git rebase (see [here](https://medium.com/@slamflipstrom/a-beginners-guide-to-squashing-commits-with-git-rebase-8185cf6e62ec)).
2) If you have not already, add an 'upstream' remote to your repository:
   ```
   git remote add upstream 'git@github.com:getquill/quill.git'
   ``` 
3) Do a git Pull + Rebase from the upstream:
   ```
   git pull --rebase upstream master
   ```

Once you have resolved any conflicts that may have arisen from the rebase, your
branch will be capable of becoming a pull-request.

## Building locally using Docker only for databases

To restart your database service with database ports exposed to your host machine run:

```bash
docker compose down && docker compose run --rm --service-ports setup
```

After that we need to set some environment variables in order to run `sbt` locally.

```bash
export CASSANDRA_HOST=127.0.0.1
export CASSANDRA_PORT=19042
export CASSANDRA_CONTACT_POINT_0=127.0.0.1:19042
export CASSANDRA_DC=datacenter1
export MYSQL_HOST=127.0.0.1
export MYSQL_PORT=13306
export MYSQL_PASSWORD=root
export POSTGRES_HOST=127.0.0.1
export POSTGRES_PORT=15432
export POSTGRES_PASSWORD=postgres
export SQL_SERVER_HOST=127.0.0.1
export SQL_SERVER_PORT=11433
export ORIENTDB_HOST=127.0.0.1
export ORIENTDB_PORT=12424
export ORACLE_HOST=127.0.0.1
export ORACLE_PORT=11521
```

Where `127.0.0.1` is address of local docker.
If you have non-local docker change it depending on your settings.

Finally, you can use `sbt` locally.

### All In One ###

To restart the database services, rebuild them, and start with locally exposed ports run:

docker compose down && docker compose build && docker compose run --rm --service-ports setup

Note: Make sure you have exposed all the ports as mentioned above.

## Debugging using IntelliJ

[IntelliJ](https://www.jetbrains.com/idea/) has a comprehensive debugger that also works with macros which is very
helpful when working on Quill. There are two ways to debug Quill macros using IntelliJ. The first way is to launch SBT in 
debug mode and use IntelliJ to remote debug it. The second way is to launch a debug session 
from IntelliJ from the "Run/Debug Configurations" menu.

### Debug Macros by Remote Debugging SBT

In order to use the debugger you need to run sbt manually so follow the instructions 
[here](#building-locally-using-docker-only-for-databases) first. After this you need to edit `build.sbt` to disable
forking when running tests, this is done by changing `fork := true` to `fork := false` for the tests you want to run.

After this you need to launch sbt with `sbt -jvm-debug 5005`. Note that since the JVM is no longer forked in tests its
recommended to launch sbt with additional memory, i.e. `sbt -jvm-debug 5005 -mem 4096` otherwise sbt may complain about
having memory issues.

Then in IntelliJ you need to
[add a remote configuration](https://www.jetbrains.com/help/idea/run-debug-configuration-remote-debug.html). The default
parameters will work fine (note that we started sbt with the debug port `5005` which is also the default debug port
in IntelliJ). After you have added the configuration you should be able to start it to start debugging! Feel to free
to add breakpoints to step through the code.

Note that its possible to debug macros (you can even
[evaluate expressions](https://www.jetbrains.com/help/idea/evaluating-expressions.html) while paused inside a macro),
however you need to edit the macro being executed after every debug run to force the macro to recompile since macro
invocations are cached on a file basis. You can easily do this just by adding new lines.

### Debug Macros by Launching a Session

Firstly, you will need to build Quill with some additional dependencies that include the file `scala.tools.nsc.Main`.
You can do this adding the argument `-DdebugMacro=true` to the sbt launcher. You can do this in the IntelliJ SBT
menu:

![IntelliJ-SBT-Settings.png](etc/IntelliJ-SBT-Settings.png)

In IntelliJ, go to `Run -> Edit Configurations...` click on the Plus (i.e. `+`) button (or `Add New Configuration`) 
and select `Application`. Then enter the following settings:

```
Main Class: scala.tools.nsc.Main
VM Options: -Dscala.usejavacp=true
Program Arguments: -cp io.getquill.MySqlTest.scala /home/me/projects/quill/quill-sql/src/main/scala/io/getquill/MySqlTest.scala
Use classpath of module: quill-sql
Before launch:
Build, no error check (make sure to set this since you will frequently want to debug the macros even if the build fails)
```

It should look like this:
![IntelliJ-Run-Debug-Config.png](etc/IntelliJ-Run-Debug-Config.png)

> NOTE In this example, our entry-point into Quill-macro-debugging is `MySqlTest.scala`.
> In our IntelliJ application configuration this file name is being explicitly specified. 
> If you wish to easily be able to macro-debug multiple entry-point files, an alternative method would be to
> use some IntelliJ variables to automatically pass whatever file is currently selected. You can do this by using
> the configuration:
> ```
> -cp $FileFQPackage$$FileName$ $FilePath$
> ```
> Also, instead of specifying the path segment `/home/me/projects/quill/` you can use `$ProjectFileDir$`

Then create a file `quill-sql/src/main/scala/io/getquill/MySqlTest.scala` that has the code you wish to debug.
For example:
```scala
object MySqlTest {

  val ctx = new SqlMirrorContext(PostgresDialect, Literal)

  case class Person(name: String, age:Int)

  def main(args:Array[String]):Unit = {    
    val q = quote {
      query[Person].filter(p => p.name == "Joe")
    }
    println( run(q).string )
  }
}
```

Set a breakpoint anywhere in the Quill codebase and run this configuration from the top-right menu shortcut:
![IntelliJ-Debug-App-Launcher](etc/IntelliJ-Debug-App-Launcher.png)

## Additional Debug Arguments

Some additional arguments you can add to your compiler's VM args provide insight into Quill's compilation:

```
-DdebugMacro=true                              // Enables libraries needed to debug via an IntelliJ Application session (default=false)
-DexcludeTests=false                           // Excludes testing code from being build. Useful during development times that require rapid iteration
-Dquill.macro.log.pretty=true                  // Pretty print the SQL Queries that Quill produces (default=false)
-Dquill.macro.log=true                         // Enable/Disable printing of the SQL Queries Quill generates during compile-time (default=true)
-Dquill.trace.enabled=true                     // Global switch that Enables/Disables printing of Quill ASTs during compilation (default=false)
-Dquill.trace.color=true                       // Print Quill ASTs in color (default=false) 
-Dquill.trace.opinion=false                    // Print the parts of Quill ASTs not directly used in the main transformation phases (called Opinions).  (default=false) 
-Dquill.trace.ast.simple=true                  // Print the raw Quill ASTs elements or a more compact view of the AST code (think `show` vs `showRaw` in Scala macros). (default=true) 
-Dquill.trace.types=sql,standard,alias,norm    // What parts of the Quill transformations to print during compilation?
```

In IntelliJ, add them in the SBT settings if your are compiling using SBT:
![IntelliJ-SBT-Settings-Additional.png](etc/IntelliJ-SBT-Settings-Additional.png)

## 'Trick' Debugging via the Dynamic Query API

Since typically the Quill Query Normalizations and Tokenizations run during compile-time,
break-point debugging them requires one of the above two steps. However, you can 'trick'
Quill into running these during compile time by using the `.dynamically` keyword.

For example say you have a query that looks like this, and you would like to see
what happens to the AST in the `ExpandDistinct` phase.
```scala
val q = quote { query[Person].distinct.map(p => p.name) }
val output = run(q)
```
Change the quotation to be dynamic:
```scala
val output = run(q.dynamically)
```
Since the query is now dynamic, you can set a breakpoint in the ExpandDistinct
class and it will be hit during the runtime of your application.

#### Alternate Method

In situations where the .dynamically keyword is not available, e.g. when the 
quoted construct is not a query, add a type annotation to the variable holding the
quotation and this will effectively cause the same behavior.

```scala
val q = quote { query[Person].distinct.map(p => p.name) }
val output = run(q)
```
Add a type annotation which will cause it to be dynamic
```scala
val q: Quoted[Query[String]] = quote { query[Person].distinct.map(p => p.name) }
```
Same as before, since the query is now dynamic, you can set a breakpoint in the ExpandDistinct
class and it will be hit during the runtime of your application.

---

## Additional Resources

## Templates

In order to quickly start with Quill, we have setup some template projects:

* [Play Framework with Quill JDBC](https://github.com/getquill/play-quill-jdbc)
* [Play Framework with Quill async-postgres](https://github.com/jeffmath/play-quill-async-postgres-example)

## Slick comparison

Please refer to [SLICK.md](https://github.com/getquill/quill/blob/master/SLICK.md) for a detailed comparison between Quill and Slick.

## Cassandra libraries comparison

Please refer to [CASSANDRA.md](https://github.com/getquill/quill/blob/master/CASSANDRA.md) for a detailed comparison between Quill and other main alternatives for interaction with Cassandra in Scala.

## Related Projects
* [quill-generic](https://github.com/ajozwik/quill-generic) - Generic DAO Support for Quill.
* [scala-db-codegen](https://github.com/olafurpg/scala-db-codegen) - Code/boilerplate generator from db schema
* [quill-cache](https://github.com/mslinn/quill-cache/) - Caching layer for Quill
* [quill-gen](https://github.com/mslinn/quill-gen/) - a DAO generator for `quill-cache`

## External content

### Talks

- **[Dotty/Scala 3]** Functional Scala 2020 - [Quill, Dotty, And The Awesome Power of 'Inline'](https://www.youtube.com/watch?v=SmBpGkIsJIU)
- **[Intro]** ScalaDays Berlin 2016 - [Scylla, Charybdis, and the mystery of Quill](https://www.youtube.com/watch?v=nqSYccoSeio)
- **[Intro]** Postgres Philly 2019 - [Introduction to Quill](https://www.youtube.com/watch?v=RVs-T5iFdQI)
- ScalaUA 2020 - [Manipulating Abstract Syntax Trees (ASTs) to generate safe SQL Queries with Quill](https://www.youtube.com/watch?v=aY8DrjE9lIY)
- BeeScala 2019 - [Quill + Spark = Better Together](https://www.youtube.com/watch?v=EXISmUXBXu8)
- Scale By the Bay 2019 - [Quill + Doobie = Better Together](https://www.youtube.com/watch?v=1WVjkP_G2cA)
- ScQuilL, Porting Quill to Dotty (Ongoing) - [Quill, Dotty, and Macros](https://www.youtube.com/playlist?list=PLqky8QybCVQYNZY_MNJpkjFKT-dAdHQDX)

### Blog posts

- **[Intro]** Haoyi's Programming Blog - [Working with Databases using Scala and Quill](https://www.lihaoyi.com/post/WorkingwithDatabasesusingScalaandQuill.html)
- Juliano Alves's Blog - [Streaming all the way with ZIO, Doobie, Quill, http4s and fs2](https://juliano-alves.com/2020/06/15/streaming-all-the-way-zio-doobie-quill-http4s-fs2/)
- Juliano Alves's Blog - [Quill: Translating Boolean Literals](https://juliano-alves.com/2020/09/14/quill-translating-boolean-literals/)
- Juliano Alves's Blog - [Quill NDBC Postgres: A New Async Module](https://juliano-alves.com/2019/11/29/quill-ndbc-postgres-a-new-async-module/)
- Juliano Alves's Blog - [Contributing to Quill, a Pairing Session](https://juliano-alves.com/2019/11/18/contributing-to-quill-a-pairing-session/)
- Medium @ Fwbrasil - [quill-spark: A type-safe Scala API for Spark SQL](https://medium.com/@fwbrasil/quill-spark-a-type-safe-scala-api-for-spark-sql-2672e8582b0d)
- Scalac.io blog - [Compile-time Queries with Quill](https://web.archive.org/web/20170512003505/https://blog.scalac.io/2016/07/21/compile-time-queries-with-quill.html)

## Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See [CODE_OF_CONDUCT.md](https://github.com/getquill/quill/blob/master/CODE_OF_CONDUCT.md) for details.

## License

See the [LICENSE](https://github.com/getquill/quill/blob/master/LICENSE.txt) file for details.

---

## Quill Changelog

Starting from the versions superior v4.6.1, the release notes and changelog will be provided in the GitHub Releases of the project.
See https://github.com/zio/zio-quill/releases

# 4.6.1

- [Remove support of Scala 2.11 && Update ZIO to 2.0.12](https://github.com/zio/zio-quill/pull/2722)
  - Remove support of Scala 2.11
  - Update Scala versions
  - Update ZIO to 2.0.12
  - Update zio-logging to 2.1.12
  - Update sbt to 1.7.3
  - Update sbt-scoverage to 2.0.0
  - Update zio-json to 0.5.0

#### Notes:
- Besides for Scala-version support update and the dependency bumps (thank you Guizmaii and Juliano!) the purpose of this
  release is to set the stage of future infrastructure changes e.g. adding Scalafmt to do formatting instead
  of the long-standing scalariform.

# 4.6.0

- [Implementing Postgres json/jsonb encoding via zio-json](https://github.com/zio/zio-quill/pull/2615)

# 4.5.0

- [Remove the need to make things embedded, happens automatically](https://github.com/zio/zio-quill/pull/2607)
- [Date encoders for major date types. Extensible context.](https://github.com/zio/zio-quill/pull/2598)
- [Warning about embedding fields that should have encoders](https://github.com/zio/zio-quill/pull/2610)
- [jasync zio current schema configuration, jasync version  update](https://github.com/zio/zio-quill/pull/2588)

#### Migration Notes:
- It is no longer necessary to do extend `Embedded` for case classes that should be embedded within an entity.
  In the case that the embedded case class "looks" like it should be encoded/decoded (i.e. it has only one field),
  an additional warning has been introduced to notify the user of this potential issue.

# 4.4.1

- [Fixing VALUES-clause update query naming](https://github.com/zio/zio-quill/pull/2595)

# 4.4.0

- [UPDATE with VALUES optimization for Postgres. Various macro refactoring.](https://github.com/zio/zio-quill/pull/2571)

# 4.3.0

- [Values clause batch insert](https://github.com/zio/zio-quill/pull/2565)
- [Slightly better batch logging](https://github.com/zio/zio-quill/pull/2562)
- ["transaction" supports ZIO effects with mixed environments](https://github.com/zio/zio-quill/pull/2515)

# 4.2.0

- [Implement ZIO-Idiomatic JDBC Context](https://github.com/zio/zio-quill/pull/2537)
- [Update idiomatic pattern based on discussions](https://github.com/zio/zio-quill/pull/2546)
- [Implement ZIO idiomatic pattern for cassandra](https://github.com/zio/zio-quill/pull/2549)
- [Add switch to manually disable returning/output-clauses](https://github.com/zio/zio-quill/pull/2550)
- [cassandra - update if exists](https://github.com/zio/zio-quill/pull/2359)
- [Change infix”$content” to sql”$content”](https://github.com/zio/zio-quill/pull/2547)
- [Remove mysql as](https://github.com/zio/zio-quill/pull/2540)

#### Migration Notes:
- The `infix` interpolator is now deprecated because in Scala 2, infix is a keyword. Instead of
  `infix"MyUdf(${person.name})"` use `sql"MyUdf(${person.name})"`. For contexts such as Doobie that already
  have an `sql` interpolator. Import `context.compat._` and use the `qsql` interpolator instead.

# 4.1.0

- [Implement groupByMap and EnableTrace/DisablePhase functionality](https://github.com/zio/zio-quill/pull/2517)

# 4.0.0

- [Bump to ZIO 2.0.0 Release Version](https://github.com/zio/zio-quill/pull/2511)

# 3.19.0

- [Implementing insert/update/delete.returningMany](https://github.com/zio/zio-quill/pull/2509)
- [Update jasync](https://github.com/zio/zio-quill/pull/2374)
- [Add support for SSL Client connections](https://github.com/zio/zio-quill/pull/2491)

# 3.18.0

- [Check all columns for null-ness for Option[Product] to be None](https://github.com/zio/zio-quill/pull/2504)
- [Fixing Correlated Subquery Issues](https://github.com/zio/zio-quill/pull/2489)
- [Corrects Like operator generating wrong SQLs](https://github.com/zio/zio-quill/pull/2502)
- [Implement filterIfDefined](https://github.com/zio/zio-quill/pull/2501)
- [Remove invalid 'AS' for Oracle Queries](https://github.com/zio/zio-quill/pull/2499)
- [Remove twitter-chill library](https://github.com/zio/zio-quill/pull/2500)

#### Version Bumps:
- [sbt-scalajs-crossproject to 1.2.0](https://github.com/zio/zio-quill/pull/2444)
- [logback-classic to 1.2.11](https://github.com/zio/zio-quill/pull/2439)
- [h2 to 2.1.212](https://github.com/zio/zio-quill/pull/2462)
- [zio, zio-streams to 1.0.14](https://github.com/zio/zio-quill/pull/2465)
- [cassandra-driver-core to 3.11.2](https://github.com/zio/zio-quill/pull/2470)
- [java-driver-core to 4.14.1](https://github.com/zio/zio-quill/pull/2469)
- [scala-collection-compat to 2.7.0](https://github.com/zio/zio-quill/pull/2445)
- [mysql-connector-java to 8.0.29](https://github.com/zio/zio-quill/pull/2467)
- [scala3-library, ... to 3.1.2](https://github.com/zio/zio-quill/pull/2464)
- [sbt-sonatype to 3.9.13](https://github.com/zio/zio-quill/pull/2496)
- [postgresql to 42.3.6](https://github.com/zio/zio-quill/pull/2495)

#### Migration Notes:
- As a result of [2504](https://github.com/zio/zio-quill/pull/2504), the handling of optional-product rows (technically parts of rows) is now different. Whereas before, if any non-optional column of an optional-product row was null, then entre optional-product would be null. Now however, an optional-product will only be null if every column inside is null. For example, before, if a query returning `Person(name:Option(Name(first:String, last:String)), age: Int)` resulted in the row `ResultRow("Joe", null, 123)` before the entity would be decoded into `Person(None, 123)` (i.e. the optional-product `Option[Name]` would decode to `None`).
  Now however, `Option[Name]` only decodes to `None` if every column inside it is null. This means that the `ResultRow("Joe", null, 123)` decodes to `Person(Name("Joe", 0 /*default-placeholder for null*/), 123)`. Only when the both `first` and `last` columns in Name are null i.e. `ResultRow(null, null, 123)` will the result be: `Person(None, 123)`. Have a look at the PR [2504](https://github.com/zio/zio-quill/pull/2504) as well as it's corresponding issue [2505](https://github.com/zio/zio-quill/issues/2505) for more details on how this works and the rationale for it.

# 3.16.5

- [Re-integrating Doobie support](https://github.com/zio/zio-quill/pull/2478)

# 3.16.4

- [Support Spark 3.2.x and Scala 2.13 in quill-spark](https://github.com/zio/zio-quill/pull/2460)
- [Fix dynamic query quat error](https://github.com/zio/zio-quill/pull/2458)
- [Implement DistinctOn](https://github.com/zio/zio-quill/pull/2375)
- [Sync Contexts with ProtoQuill and move out ProtoQuill conflicts](https://github.com/zio/zio-quill/pull/2472)
- [Fix the typo `ZioCassandraSession`  in document.](https://github.com/zio/zio-quill/pull/2303)

# 3.16.3

- [Remove anonymous class made by Property.Opinionated and make NullValue case object](https://github.com/zio/zio-quill/pull/2426)

#### Note
* This change is to allow ProtoQuill transition to BooPickle AST Serialization in https://github.com/zio/zio-protoquill/pull/72

# 3.16.2

- [Remove second apply method on ast.CaseClass due to incremental-compile errors](https://github.com/zio/zio-quill/pull/2423)

# 3.16.1

- [Top-Level quat passing to query expansions](https://github.com/zio/zio-quill/pull/2420)

# 3.16.0

- [Remove deprecated EntityQuery.update/insert APIs](https://github.com/zio/zio-quill/pull/2412)

#### Migration Notes
* This change removes the deprecated `EntityQuery.insert(CaseClass)` and `EntityQuery.update(CaseClass)` APIs that have been
  updated to `EntityQuery.insertValue(CaseClass)` and `EntityQuery.updateValue(CaseClass)`.
  This is the only change in this release so that you can update when ready.
  This change is needed due to the upstream Dotty issue: lampepfl/dotty#14043.

# 3.15.0

- [cassandra - update if exists](https://github.com/zio/zio-quill/pull/2359)
- [Change update to updateValue](https://github.com/zio/zio-quill/pull/2405)

#### Migration Notes
* Similar to `EntityQuery.insert(CaseClass)`, the method `EntityQuery.update(CaseClass)` e.g. `query[Person].update(Person("Joe", 123))` has been replaced with `updateValue`.
  The original `insert` method has been deprecated and will be removed in an upcoming Quill release.

# 3.14.1

- [Fix Infix Ast Quat issues](https://github.com/zio/zio-quill/pull/2403)

# 3.14.0

- [Single path-element alias in top-level selects](https://github.com/zio/zio-quill/pull/2393)
- [Update zio-logging](https://github.com/zio/zio-quill/pull/2392)

# 3.13.0

- [JAsync ZIO implementation](https://github.com/zio/zio-quill/pull/2267)
- [cassandra-alpakka](https://github.com/zio/zio-quill/pull/2365)
- [Need to change EntityQuery.insert(CaseClass) to EntityQuery.insertValue(CaseClass) for upstream Scala 3 issues.](https://github.com/zio/zio-quill/pull/2379)
- [Update ScalaJS to latest](https://github.com/zio/zio-quill/pull/2364)
- [Work on removing tuple elaboration](https://github.com/zio/zio-quill/pull/2381)
- [Option to Disable Nested Subexpansion](https://github.com/zio/zio-quill/pull/2383)
- [Remove deprecated async modules](https://github.com/zio/zio-quill/pull/2362)
- [Add Scala 3 cross-build for quill-engine](https://github.com/zio/zio-quill/pull/2361)
- [Move quill-core-portable & quill-sql-portable to common quill-engine module](https://github.com/zio/zio-quill/pull/2360)
- [Sheath leaf map clauses that cannot be reduced so still have their column in queries](https://github.com/zio/zio-quill/pull/2352)

#### Migration Notes
* The method `EntityQuery.insert(CaseClass)` e.g. `query[Person].insert(Person("Joe", 123))` has been replaced with `insertValue`.
  The original `insert` method has been deprecated and will be removed in the next Quill release.
* The `quill-async` modules using Mauricio's deprecated library ([here](https://github.com/mauricio/postgresql-async))
  have been removed. Please move to the `quill-jasync` libraries as soon as possible.
* Quill for ScalaJS has been updated to ScalaJS 1.8.
* `quill-core-portable` and `quill-sql-portable` are now merged into a cross-built `quill-engine` module.
* In 3.12.0 addition of field-aliases has been introduced in sub-queries but [#2340](https://github.com/zio/zio-quill/issues/2340)
  then occurred. A compile-time switch `-Dquill.query.subexpand=false` has been introduced to disable the feature
  until it can be fixed.

# 3.12.0

- [cassandra - Datastax4x upgrade](https://github.com/getquill/quill/pull/2315)
- [Implement dynamic query caching](https://github.com/getquill/quill/pull/2088)
- [Fix Quat-based query schema rename issue](https://github.com/getquill/quill/pull/2322)
- [forUpdate via infix](https://github.com/getquill/quill/pull/2268)
- [Disable file infra log by default](https://github.com/getquill/quill/pull/2332)
- [Fix `QIO.apply` function type](https://github.com/getquill/quill/pull/2318)
- [doc: update CODEGEN.md](https://github.com/getquill/quill/pull/2316)

#### Migration Notes - Datastax Drivers:

The Datastax drivers have been moved to Version 4, this adds support for many new features with the caveat that the configuration
file format must be changed. In Version 4, the Datastax standard configuration file format and properties
are in the HOCON format. They are used to configure the driver.

Sample HOCON:
```hocon
MyCassandraDb {
  preparedStatementCacheSize=1000
  keyspace=quill_test

  session {
    basic.contact-points = [ ${?CASSANDRA_CONTACT_POINT_0}, ${?CASSANDRA_CONTACT_POINT_1} ]
    basic.load-balancing-policy.local-datacenter = ${?CASSANDRA_DC}
    basic.request.consistency = LOCAL_QUORUM
    basic.request.page-size = 3
  }

}
```

The `session` entry values and keys are described in the datastax documentation:
[Reference configuration](https://docs.datastax.com/en/developer/java-driver/4.13/manual/core/configuration/reference/)

The ZioCassandraSession constructors:

```scala
 val zioSessionLayer: ZLayer[Any, Throwable, CassandraZioSession] =
  CassandraZioSession.fromPrefix("MyCassandraDb")
run(query[Person])
  .provideCustomLayer(zioSessionLayer)
```

Additional parameters can be added programmatically:
```scala
 val zioSessionLayer: ZLayer[Any, Throwable, CassandraZioSession] =
  CassandraZioSession.fromContextConfig(LoadConfig("MyCassandraDb").withValue("keyspace", ConfigValueFactory.fromAnyRef("data")))
run(query[Person])
  .provideCustomLayer(zioSessionLayer)
```

`session.queryOptions.fetchSize=N` config entry should be replaced by
`basic.request.page-size=N`

```hocon
testStreamDB {
  preparedStatementCacheSize=1000
  keyspace=quill_test

  session {
    ...
    basic.request.page-size = 3
  }
  ...
}
```

#### Migration Notes - Query Log File:

Production of the query-log file `queries.txt` has been disabled by default due to issues with SBT
and metals. In order to use it, launch the compiler JVM (e.g. SBT) with the argument `-Dquill.log.file=my_queries.sql`
or set the `quill_log_file` environment variable (e.g. `export quill_log_file=my_queries.sql`).

#### Migration Notes - Monix:

The monix context wrapper `MonixJdbcContext.Runner` has been renamed to `MonixJdbcContext.EffectWrapper`.
The type `Runner` needs to be used by ProtoQuill to define quill-context-specific execution contexts.

# 3.11.0

- [Implement `transaction` on outer zio-jdbc-context using fiber refs](https://github.com/getquill/quill/pull/2302)
- [Feature Request: write compile-time queries to a file](https://github.com/getquill/quill/issues/1715)
- [`transaction` supports ZIO effects with mixed environments](https://github.com/getquill/quill/pull/2304)
- [Apple M1 Build Updates & Instructions](https://github.com/getquill/quill/pull/2296)

#### Migration Notes:

All ZIO JDBC context `run` methods have now switched from have switched their dependency (i.e. `R`) from `Connection` to
`DataSource`. This should clear up many innocent errors that have happened because how this `Connection` is supposed
to be provided was unclear. As I have come to understand, nearly all DAO service patterns involve grabbing a connection from a
pooled DataSource, doing one single crud operation, and then returning the connection back to the pool. The new JDBC ZIO context
memorialize this pattern.

* The signature of `QIO[T]` has been changed from `ZIO[Connection, SQLException, T]` to `ZIO[DataSource, SQLException, T]`.
  a new type-alias `QCIO[T]` (lit. Quill Connection IO) has been introduced that represents `ZIO[Connection, SQLException, T]`.

* If you are using the `.onDataSource` command, migration should be fairly easy. Whereas previously, a usage of quill-jdbc-zio 3.10.0
  might have looked like this:
  ```scala
  object MyPostgresContext extends PostgresZioJdbcContext(Literal); import MyPostgresContext._
  val zioDS = DataSourceLayer.fromPrefix("testPostgresDB")

  val people = quote {
    query[Person].filter(p => p.name == "Alex")
  }

  MyPostgresContext.run(people).onDataSource
    .tap(result => printLine(result.toString))
    .provideCustomLayer(zioDs)
  ```
  In 3.11.0 simply remove the `.onDataSource` in order to use the new context.
  ```scala
  object MyPostgresContext extends PostgresZioJdbcContext(Literal); import MyPostgresContext._
  val zioDS = DataSourceLayer.fromPrefix("testPostgresDB")

  val people = quote {
    query[Person].filter(p => p.name == "Alex")
  }

  MyPostgresContext.run(people)  // Don't need `.onDataSource` anymore
    .tap(result => printLine(result.toString))
    .provideCustomLayer(zioDs)
  ```

* If you are creating a Hikari DataSource directly, passing of the dependency is now also simpler. Instead having to pass
  the Hikari-pool-layer into `DataSourceLayer`, just provide the Hikari-pool-layer directly.

  From this:
  ```scala
  def hikariConfig = new HikariConfig(JdbcContextConfig(LoadConfig("testPostgresDB")).configProperties)
  def hikariDataSource: DataSource with Closeable = new HikariDataSource(hikariConfig)

  val zioConn: ZLayer[Any, Throwable, Connection] =
    Task(hikariDataSource).toLayer >>> DataSourceLayer.live

  MyPostgresContext.run(people)
    .tap(result => printLine(result.toString))
    .provideCustomLayer(zioConn)
  ```
  To this:
  ```scala
  def hikariConfig = new HikariConfig(JdbcContextConfig(LoadConfig("testPostgresDB")).configProperties)
  def hikariDataSource: DataSource with Closeable = new HikariDataSource(hikariConfig)

  val zioDS: ZLayer[Any, Throwable, DataSource] =
    Task(hikariDataSource).toLayer // Don't need `>>> DataSourceLayer.live` anymore!

  MyPostgresContext.run(people)
    .tap(result => printLine(result.toString))
    .provideCustomLayer(zioConn)
  ```

* If you want to provide a `java.sql.Connection` to a ZIO context directly, you can still do it using the `underlying` variable.
  ```
  object Ctx extends PostgresZioJdbcContext(Literal); import MyPostgresContext._
  Ctx.underlying.run(qr1)
    .provide(zio.Has(conn: java.sql.Connection))
  ```

* Also, when using an underlying context, you can still use `onDataSource` to go from a `Connection` dependency
  back to a `DataSource` dependency (note that it no longer has to be `with Closable`).
  ```
    object Ctx extends PostgresZioJdbcContext(Literal); import MyPostgresContext._
    Ctx.underlying.run(qr1)
      .onDataSource
      .provide(zio.Has(ds: java.sql.DataSource))
    ```

* Finally, that the `prepare` methods have been unaffected by this change. They still require a `Connection`
  and have the signature `ZIO[Connection, SQLException, PreparedStatement]`. This is because in order to work
  with the result of this value (i.e. to work with `PreparedStatement`), the connection that created it must
  still be open.

# 3.10.0

- [Defunct AsyncZioCache accidentally returned in #2174. Remove it.](https://github.com/getquill/quill/pull/2246)
- [Open connection on ZIO blocking pool](https://github.com/getquill/quill/pull/2244)
- [Line-up core API with ProtoQuill so child contexts can have same code](https://github.com/getquill/quill/pull/2231)

#### Migration Notes:

No externally facing API changes have been made.
This release aligns Quill's internal Context methods with the API defined in ProtoQuill and introduces
a root-level context (in the `quill-sql-portable` module) that will be shared together with ProtoQuill.
Two arguments `info: ExecutionInfo` and `dc: DatasourceContext` have been introduced to all `execute___`
and `prepare___` methods. For Scala2-Quill, these arguments should be ignored as they contain no
relevant information. ProtoQuill uses them in order to pass Ast information as well as whether
the query is Static or Dynamic into execute and prepare methods. In the future, Scala2-Quill may be enhanced
to use them as well.

# 3.9.0

- [Pass Session to all Encoders/Decoders allowing UDT Encoding without local session variable in contexts e.g. ZIO and others](https://github.com/getquill/quill/pull/2219)
- [Fixing on-conflict case with querySchema/schemaMeta renamed columns](https://github.com/getquill/quill/pull/2218)

#### Migration Notes:

This release modifies Quill's core encoding DSL however this is very much an internal API.
If you are using MappedEncoder, which should be the case for most users, you will be completely unaffected.
The MappedEncoder signatures remain the same.

Quill's core encoding API has changed:
```scala
// From:
type BaseEncoder[T] = (Index, T, PrepareRow) => PrepareRow
type BaseDecoder[T] = (Index, ResultRow) => T
// To:
type BaseEncoder[T] = (Index, T, PrepareRow, Session) => PrepareRow
type BaseDecoder[T] = (Index, ResultRow, Session) => T
```
That means that internal signature of all encoders has also changed. For example, the JdbcEncoder has changed:
```scala
// From:
case class JdbcEncoder[T](sqlType: Int, encoder: BaseEncoder[T]) extends BaseEncoder[T] {
  override def apply(index: Index, value: T, row: PrepareRow) =
    encoder(index + 1, value, row)
}
// To:
case class JdbcEncoder[T](sqlType: Int, encoder: BaseEncoder[T]) extends BaseEncoder[T] {
  override def apply(index: Index, value: T, row: PrepareRow, session: Session) =
    encoder(index + 1, value, row, session)
}
```
If you are writing encoders that directly implement `BaseEncoder`, they will have to be modified with an
additional `session: Session` parameter.
> The actual type that `Session` is will vary. For JDBC this will be `Connection`, for `Cassandra` this will be some
implementation of `CassandraSession`, for other systems that use a entirely different session paradigm
this will just be `Unit`.

Again, if you are using MappedEncoders for all of your custom encoding needs, you will not be affected by this change.

# 3.8.0

- [Use ZIO-Native Iterator chunking for JDBC result sets](https://github.com/getquill/quill/pull/2196)
- [Remove 'with Blocking' from all signatures](https://github.com/getquill/quill/pull/2174)
- [Update Microsoft SQL Server Docker image](https://github.com/getquill/quill/pull/2183)

#### Migration Notes:
The `quill-jdbc-zio` contexts' `.run` method was designed to work with ZIO in an idiomatic way. As such, the environment variable
of their return type including the `zio.blocking.Blocking` dependency. This added a significant amount of complexity.
Instead of `ZIO[Connection, SQLException, T]`, the return type became `ZIO[Connection with Blocking, SQLException, T]`.
Instead of `ZIO[DataSource with Closeable, SQLException, T]`, the return type became `ZIO[DataSource with Closeable with Blocking, SQLException, T]`.
Various types such as `QConnection` and `QDataSource` were created in order to encapsulate these concepts but this only led to additional confusion.
Furthermore, actually supplying a `Connection` or `DataSource with Closeable` required first peeling off the `with Blocking` clause, calling a `.provide`,
and then appending it back on. The fact that a Connection needs to be opened from a Data Source (which will typically be a Hikari connection pool)
further complicates the problem because this aforementioned process needs to be done twice. All of leads to the clear conclusion that the `with Blocking`
construct has bad ergonomics. For this reason, the ZIO team has decided to drop the concept of `with Blocking` in ZIO 2 altogether.

As a result of this, I have decided to drop the `with Blocking` construct in advance. Quill queries resulting from the `run(qry)` command and
still run on the blocking pool but `with Blocking` is not included in the signature. This also means that and the need for `QConnection` and `QDataSource` disappears since they are now just `Connection` and `Datasource with Closeable`
respectively. This also means that all the constructors on the corresponding objects e.g. `QDataSource.fromPrefix("myDB")` are not consistent with
any actual construct in QIO, therefore they are not needed either.

Instead, I have introduced a simple layer-constructor called `DataSourceLayer` which has a `.live` implementation which converts
`ZIO[Connection, SQLException, T]` to `ZIO[DataSource with Closeable, SQLException, T]` by taking a connection from the
data-source and returning it immediately afterward, this is the analogue of what `QDataSource.toConnection` use to do.
You can use it like this:
```scala
def hikariDataSource: DataSource with Closeable = ...
val zioConn: ZLayer[Any, Throwable, Connection] =
  Task(hikariDataSource).toLayer >>> DataSourceLayer.live
run(people)
  .provideCustomLayer(zioConn)
```

You can also use the extension method `.onDataSource` (or `.onDS` for short) to do the same thing:
```scala
def hikariDataSource: DataSource with Closeable = ...
run(people)
  .onDataSource
  .provide(Has(hikariDataSource))
```

Also, constructor-methods `fromPrefix`, `fromConfig`, `fromJdbcConfig` and `fromDataSource` are available on
`DataSourceLayer` to construct instances of `ZLayer[DataSource with Closeable, SQLException, Connection]`.
Combined with the `toDataSource` construct, these provide a simple way to construct various Hikari pools from
a corresponding typesafe-config file `application.conf`.
```scala
run(people)
  .onDataSource
  .provideLayer(DataSourceLayer.fromPrefix("testPostgresDB"))
```

Also note that the objects `QDataSource` and `QConnection` have not yet been removed. Instead, all of their methods
have been marked as deprecated and a comment on what calls using `DataSourceLayer`/`onDataSource` to use instead
have been added.

#### Cassandra:

Similar changes have been made in quill-cassandra-zio. `CassandraZioSession with Blocking` has been replaced
with just `CassandraZioSession` so now this is much easier to provide:

```scala
val session: CassandraZioSession = _
run(people)
  .provide(Has(session))
```

The CassandraZioSession constructors however are all still fine to use:

```scala
 val zioSessionLayer: ZLayer[Any, Throwable, CassandraZioSession] =
   CassandraZioSession.fromPrefix("testStreamDB")
run(query[Person])
  .provideCustomLayer(zioSessionLayer)
```

# 3.7.2

- [Fix FutureAsyncCache](https://github.com/getquill/quill/pull/2162)
- [Catch close() exceptions](https://github.com/getquill/quill/pull/2166)
- [Make all AST classes final](https://github.com/getquill/quill/pull/2165)

# 3.7.1

- [Add a single ZIO-inspired root type QAC for all actions](https://github.com/getquill/quill/pull/2130)
- [Adding fromDataSource ZIO API and example](https://github.com/getquill/quill/pull/2131)

# 3.7.0

- [ZIO Cassandra](https://github.com/getquill/quill/pull/2106)
- [Zio](https://github.com/getquill/quill/pull/1989)

Migration Notes:
In order to properly accommodate a good ZIO experience, several refactorings had to be done to various
internal context classes, none of these changes modify class structure in a breaking way.

The following was done for quill-jdbc-zio
- Query Preparation base type definitions have been moved out of `JdbcContextSimplified` into `JdbcContextBase`
  which inherits a class named `StagedPrepare` which defines prepare-types (e.g. `type PrepareQueryResult = Session => Result[PrepareRow]`).
- This has been done so that the ZIO JDBC Context can define prepare-types via the ZIO `R` parameter instead of
  a lambda parameter (e.g. `ZIO[QConnection, SQLException, PrepareRow]` a.k.a. `QIO[PrepareRow]`).
- In order prevent user-facing breaking changes. The contexts in `BaseContexts.scala` now extend from both `JdbcContextSimplified` (indirectly)
  and `JdbcContextBase` thus preserving the `Session => Result[PrepareRow]` prepare-types.
- The context `JdbcContextSimplified` now contains the `prepareQuery/Action/BatchAction` methods used by all contexts other than the ZIO
  contexts which define these methods independently (since they use the ZIO `R` parameter).
- All remaining context functionality (i.e. the `run(...)` series of functions) has been extracted out into `JdbcRunContext` which the
  ZIO JDBC Contexts in `ZioJdbcContexts.scala` as well as all the other JDBC Contexts now extend.

Similarly for quill-cassandra-zio
- The CassandraSessionContext on which the CassandraMonixContext and all the other Cassandra contexts are based on keeps internal state (i.e. session, keyspace, caches).
- This state was pulled out as separate classes e.g. `SyncCache`, `AsyncFutureCache` (the ZIO equivalent of which is `AsyncZioCache`).
- Then a `CassandraZioSession` is created which extends these state-containers however, it is not directly a base-class of the `CassandraZioContext`.
- Instead it is returned as a dependency from the CassandraZioContext run/prepare commands as part of the type
  `ZIO[CassandraZioSession with Blocking, Throwable, T]` (a.k.a `CIO[T]`). This allows the primary context CassandraZioContext to be stateless.

# 3.6.1

- [Memoize Passed-By-Name Quats of Asts Ident, Entity, and Others](https://github.com/getquill/quill/pull/2084)
- [Minor Quat Fixes and More Tests](https://github.com/getquill/quill/pull/2057)

Migration Notes:

 - Memoization of Quats should improve performance of dynamic queries based on some profiling analysis. This
   change should not have any user-facing changes.

# 3.6.0
This description is an aggregation of the 3.6.0-RC1, RC2 and RC3 as well as several new items.

 - [Quat Enhancements to Support Needed Spark Use Cases](https://github.com/getquill/quill/pull/2010)
 - [Add support for scala 2.13 to quill-cassandra-lagom](https://github.com/getquill/quill/pull/1909)
 - [Change all Quat fields to Lazy](https://github.com/getquill/quill/pull/2004)
 - [Smart serialization based on number of Quat fields](https://github.com/getquill/quill/pull/1997)
 - [Better Dynamic Query DSL For Quats on JVM](https://github.com/getquill/quill/pull/1993)
 - [Fix incorrect Quat.Value parsing issues](https://github.com/getquill/quill/pull/1987)
 - [Fix Query in Nested Operation and Infix](https://github.com/getquill/quill/pull/1980)
 - [Fix Logic table, replicate Option.getOrElse optimization to Boolean Quats](https://github.com/getquill/quill/pull/1975)
 - [Fixes + Enhancements to Boolean Optional APIs](https://github.com/getquill/quill/pull/1970)
 - [Fix for Boolean Quat Issues](https://github.com/getquill/quill/pull/1967)

Migration Notes:

 - The Cassandra base UDT class `io.getquill.context.cassandra.Udt` has been moved to `io.getquill.Udt`.
 - When working with databases which do not support boolean literals (SQL Server, Oracle, etc...) infixes representing booleans
   will be converted to equality-expressions.

   For example:
   ```
   query[Person].filter(p => sql"isJoe(p.name)".as[Boolean])
   // SELECT ... FROM Person p WHERE isJoe(p.name)
   // Becomes> SELECT ... FROM Person p WHERE 1 = isJoe(p.name)
   ```
   This is because the aforementioned databases do not directly support boolean literals (i.e. true/false) or expressions
   that yield them.

   In some cases however, it is desirable for the above behavior not to happen and for the whole infix statement to be treated
   as an expression. For example
   ```
   query[Person].filter(p => sql"${p.age} > 21".as[Boolean])
   // We Need This> SELECT ... FROM Person p WHERE p.age > 21
   // Not This> SELECT ... FROM Person p WHERE 1 = p.age > 21
   ```
   In order to have this behavior, instead of `sql"...".as[Boolean]`, use `sql"...".asCondition`.
   ```
   query[Person].filter(p => sql"${p.age} > 21".asCondition)
   // We Need This> SELECT ... FROM Person p WHERE p.age > 21
   ```
   If the condition represents a pure function, be sure to use `sql"...".pure.asCondition`.

# 3.6.0-RC3

 - [Add support for scala 2.13 to quill-cassandra-lagom](https://github.com/getquill/quill/pull/1909)
 - [Change all Quat fields to Lazy](https://github.com/getquill/quill/pull/2004)
 - [Smart serialization based on number of Quat fields](https://github.com/getquill/quill/pull/1997)
 - [Better Dynamic Query DSL For Quats on JVM](https://github.com/getquill/quill/pull/1993)

# 3.6.0-RC2

- [Fix incorrect Quat.Value parsing issues](https://github.com/getquill/quill/pull/1987)

Migration Notes:

 - When working with databases which do not support boolean literals (SQL Server, Oracle, etc...) infixes representing booleans
   will be converted to equality-expressions.

   For example:
   ```
   query[Person].filter(p => sql"isJoe(p.name)".as[Boolean])
   // SELECT ... FROM Person p WHERE isJoe(p.name)
   // Becomes> SELECT ... FROM Person p WHERE 1 = isJoe(p.name)
   ```
   This is because the aforementioned databases do not directly support boolean literals (i.e. true/false) or expressions
   that yield them.

   In some cases however, it is desirable for the above behavior not to happen and for the whole infix statement to be treated
   as an expression. For example
   ```
   query[Person].filter(p => sql"${p.age} > 21".as[Boolean])
   // We Need This> SELECT ... FROM Person p WHERE p.age > 21
   // Not This> SELECT ... FROM Person p WHERE 1 = p.age > 21
   ```
   In order to have this behavior, instead of `sql"...".as[Boolean]`, use `sql"...".asCondition`.
   ```
   query[Person].filter(p => sql"${p.age} > 21".asCondition)
   // We Need This> SELECT ... FROM Person p WHERE p.age > 21
   ```
   If the condition represents a pure function, be sure to use `sql"...".pure.asCondition`.
 - This release is not binary compatible with any Quill version before 3.5.3.
 - Any code generated by the Quill Code Generator with `quote { ... }` blocks will have to be regenerated with this
   Quill version if generated before 3.5.3.
 - In most SQL dialects (i.e. everything except Postgres) boolean literals and expressions yielding them are
    not supported so statements such as `SELECT foo=bar FROM ...` are not supported. In order to get equivalent logic,
    it is necessary to user case-statements e.g.
    ```sql
    SELECT CASE WHERE foo=bar THEN 1 ELSE 0`.
    ```
    On the other hand, in a WHERE-clause, it is the opposite:
    ```sql
    SELECT ... WHERE CASE WHEN (...) foo ELSE bar`
    ```
    is invalid and needs to be rewritten.
    Naively, a `1=` could be inserted:
    ```sql
    SELECT ... WHERE 1 = (CASE WHEN (...) foo ELSE bar)
    ```
    Note that this behavior can disabled via the `-Dquill.query.smartBooleans` switch
    when issued during compile-time for compile-time queries and during runtime for runtime
    queries.

    Additionally, in certain situations, it is far more preferable to express this without the `CASE WHEN` construct:
    ```sql
    SELECT ... WHERE ((...) && foo) || !(...) && foo
    ```
   This is because CASE statements in SQL are not sargable and generally [cannot be well optimized](https://dba.stackexchange.com/questions/209025/sargability-of-queries-against-a-view-with-a-case-expression).

 - A large portion of the Quill DSL has been moved outside of QueryDsl into the top level under the `io.getquill` package. Due to this change, it may be necessary to import `io.getquill.Query` if you are not already importing `io.getquill._`.

# 3.6.0-RC1

- [Fix Query in Nested Operation and Infix](https://github.com/getquill/quill/pull/1980)
- [Fix Logic table, replicate Option.getOrElse optimization to Boolean Quats](https://github.com/getquill/quill/pull/1975)
- [Fixes + Enhancements to Boolean Optional APIs](https://github.com/getquill/quill/pull/1970)
- [Fix for Boolean Quat Issues](https://github.com/getquill/quill/pull/1967)

Migration Notes:

 - This release is not binary compatible with any Quill version before 3.5.3.
 - Any code generated by the Quill Code Generator with `quote { ... }` blocks will have to be regenerated with this
   Quill version if generated before 3.5.3.
 - In most SQL dialects (i.e. everything except Postgres) boolean literals and expressions yielding them are
    not supported so statements such as `SELECT foo=bar FROM ...` are not supported. In order to get equivalent logic,
    it is necessary to user case-statements e.g.
    ```sql
    SELECT CASE WHERE foo=bar THEN 1 ELSE 0`.
    ```
    On the other hand, in a WHERE-clause, it is the opposite:
    ```sql
    SELECT ... WHERE CASE WHEN (...) foo ELSE bar`
    ```
    is invalid and needs to be rewritten.
    Naively, a `1=` could be inserted:
    ```sql
    SELECT ... WHERE 1 = (CASE WHEN (...) foo ELSE bar)
    ```
    Note that this behavior can disabled via the `-Dquill.query.smartBooleans` switch
    when issued during compile-time for compile-time queries and during runtime for runtime
    queries.

    Additionally, in certain situations, it is far more preferable to express this without the `CASE WHEN` construct:
    ```sql
    SELECT ... WHERE ((...) && foo) || !(...) && foo
    ```
   This is because CASE statements in SQL are not sargable and generally [cannot be well optimized](https://dba.stackexchange.com/questions/209025/sargability-of-queries-against-a-view-with-a-case-expression).

 - A large portion of the Quill DSL has been moved outside of QueryDsl into the top level under the `io.getquill` package. Due to this change, it may be necessary to import `io.getquill.Query` if you are not already importing `io.getquill._`.

# 3.5.3

Please skip this release and proceed directly to the 3.6.0-RC line. This release was originally a test-bed for the new Quats-based functionality which was supposed to be a strictly internal mechanism. Unfortunately multiple issues were found. They will be addressed in the 3.6.X line.

- [Adding Quill-Application-Types (Quats) to AST](https://github.com/getquill/quill/pull/1911)
- [Translate boolean literals](https://github.com/getquill/quill/pull/1923)
- [breakdown caseclasses in groupBy clause](https://github.com/getquill/quill/pull/1939)
- [allowed distinct to be placed on an infix](https://github.com/getquill/quill/pull/1935)
- [Change Subquery Expansion to be Quat-based](https://github.com/getquill/quill/pull/1920)
- [Use quats to expand nested queries in Spark](https://github.com/getquill/quill/pull/1932)
- [Fixed bug where alias of filter clause did not match alias of inner query.](https://github.com/getquill/quill/pull/1924)
- [Add default implementations so Query can be more easily inherited from Dotty](https://github.com/getquill/quill/pull/1879)
- [Monix streaming with NDBC](https://github.com/getquill/quill/pull/1750)
- [Fix SqlServer snake case - OUTPUT i_n_s_e_r_t_e_d.id](https://github.com/getquill/quill/pull/1867)

Migration Notes:`
 - Quill 3.5.3 is source-compatible but not binary-compatible with Quill 3.5.2.
 - Any code generated by the Quill Code Generator with `quote { ... }` blocks will have to be regenerated with
 Quill 3.5.3 as the AST has substantially changed.
 - The implementation of Quill Application Types (Quats) has changed the internals of nested query expansion. Queries
 with a `querySchema` or a `schemaMeta` will be aliased between nested clauses slightly differently.
 Given:
   ```
   case class Person(firstName:String, lastName:String)
   val ctx = new SqlMirrorContext(PostgresDialect, Literal)
   ```
   Before:
   ```
   SELECT x.first_name, x.last_name FROM (
     SELECT x.first_name, x.last_name FROM person x) AS x
   ```
   After:
   ```
   SELECT x.firstName, x.lastName FROM (
     SELECT x.first_name AS firstName, x.last_name AS lastName FROM person x) AS x
   ```
   Note however that the semantic result of the queries should be the same. No user-level code change for this
   should be required.

# 3.5.2

- [Add support jasync-sql for postgres](https://github.com/getquill/quill/pull/1793)
- [Add quill-jasync-mysql](https://github.com/getquill/quill/pull/1813)
- [Delete returning](https://github.com/getquill/quill/pull/1870)
- [Fix SqlServer snake case - OUTPUT i_n_s_e_r_t_e_d.id](https://github.com/getquill/quill/pull/1867)
- [Add translate to NDBC Context](https://github.com/getquill/quill/pull/1865)
- [Apply NamingStrategy after applying prefix](https://github.com/getquill/quill/pull/1807)
- [Remove use of `Row#getAnyOption` from `FinaglePostgresDecoders`](https://github.com/getquill/quill/pull/1848)
- [Better error message about lifting for enum types](https://github.com/getquill/quill/pull/1803)
- [More 2.13 modules](https://github.com/getquill/quill/pull/1753)

Migration Notes:
 - Much of the content in `QueryDsl` has been moved to the top-level for better portability with the upcoming Dotty
implementation. This means that things like `Query` are no longer part of `Context` but now are directly in the
`io.getquill` package. If you are importing `io.getquill._` your code should be unaffected.
 - Custom decoders written for Finagle Postgres no longer require a `ClassTag`.

# 3.5.1

- [Fix Ndbc runtime logging](https://github.com/getquill/quill/pull/1748)
- [Support for onConflictIgnore to H2 in PostgreSQL mode](https://github.com/getquill/quill/pull/1731)
- [Update Returning](https://github.com/getquill/quill/pull/1720)

# 3.5.0

- [Ndbc Postgres Support](https://github.com/getquill/quill/pull/1702)
- [MS SQL Server returning via OUTPUT](https://github.com/getquill/quill/pull/1681)
- [Pretty Print SQL Queries](https://github.com/getquill/quill/pull/1646)
- [Fix shadowing via aggressive uncapture](https://github.com/getquill/quill/pull/1713)
- [Fix Issues with Short](https://github.com/getquill/quill/pull/1698)
- [Pull Oracle jdbc driver from Maven Central](https://github.com/getquill/quill/pull/1677)

# 3.4.10

- [support scala 2.13](https://github.com/getquill/quill/pull/1644)
- [Remove redundant null checks](https://github.com/getquill/quill/pull/1648)
- [Improve error message](https://github.com/getquill/quill/pull/1503)

# 3.4.9

- [Properly unhide properties in nested queries](https://github.com/getquill/quill/pull/1637)

# 3.4.8

- [Additional Fixes for Embedded Entities in Nested Queries](https://github.com/getquill/quill/pull/1628)
- [Fix java.sql.SQLException corner case](https://github.com/getquill/quill/pull/1614)
- [Feature/local time support](https://github.com/getquill/quill/pull/1596)
- [Update monix-eval, monix-reactive to 3.0.0](https://github.com/getquill/quill/pull/1607)

Documentation Updates:
- [Update database drivers versions to latest in docs](https://github.com/getquill/quill/pull/1617)
- [Document remote debugging in IntelliJ](https://github.com/getquill/quill/pull/1615)

Migration Notes:
- Monix 3.0.0 is not binary compatible with 3.0.0-RC3 which was a dependency of Quill 3.4.7.
If you are using the Quill Monix modules, please update your dependencies accordingly.

# 3.4.7

- [Fix RenameProperties with Schema Re-Arrangement](https://github.com/getquill/quill/pull/1613)

# 3.4.6

- [Fixing Embedded Coproduct Column Duplication Issue](https://github.com/getquill/quill/pull/1604)

# 3.4.5

- [Express Infix Clauses not in Select List](https://github.com/getquill/quill/pull/1597)

# 3.4.4

- [added MappedEncoding for encoding and decoding OffsetDateTime](https://github.com/getquill/quill/pull/1595)
- [Fix onConflict with custom schemaMeta](https://github.com/getquill/quill/pull/1589)

# 3.4.3

- [Nested Query does not work correctly with NamingSchemas](https://github.com/getquill/quill/issues/1577)

# 3.4.2

- [Fix NamingStrategy override of querySchema](https://github.com/getquill/quill/pull/1560)
- [Monix JDBC scheduling fixes ](https://github.com/getquill/quill/pull/1546)

Migration Notes:
- `NamingStrategy` is no longer applied on column and table names defined in `querySchema`, all
column and table names defined in `querySchema` are now final. If you are relying on this behavior to
name your columns/tables correctly, you will need to update your `querySchema` objects.

# 3.4.1

- [Fix property order expanded sub-queries](https://github.com/getquill/quill/pull/1541)

Migration Notes:
- Nested sub-queries will now have their terms re-ordered in certain circumstances although the functionality
of the entire query should not change. If you have deeply nested queries with Infixes, double check that
they are in the correct position.

# 3.4.0

- [Nest queries with infix values to allow infix-impurity](https://github.com/getquill/quill/pull/1534)

Migration Notes:
- Infixes are now not treated as pure functions by default. This means wherever they are used, nested queries may be created.
You can use `.pure` (e.g. `sql"MY_PURE_UDF".pure.as[T]`) to revert to the previous behavior. See the
[Infix](https://github.com/getquill/quill/tree/36842c4801c95a7609ba94c450645f3c022b3e2e#infix)
section of the documentation for more detail.

# 3.3.0

- [Returning Record](https://github.com/getquill/quill/pull/1489)
- [Change == and != to be Scala-idiomatic ](https://github.com/getquill/quill/pull/1453)
- [Optimize === comparisons when ANSI behavior assumed](https://github.com/getquill/quill/pull/1454)
- [API to get PreparedStatement from Query for Low Level Use-cases](https://github.com/getquill/quill/pull/1459)
- [Add BoundStatement support for all context.](https://github.com/getquill/quill/pull/1425)
- [Only decode when field is non-null](https://github.com/getquill/quill/pull/1514)
- [Fix support of nested transactions in Finagle-Postgres](https://github.com/getquill/quill/pull/1501)
- [Returning shadow fix](https://github.com/getquill/quill/pull/1509)
- [Fix SQL Server Subqueries with Order By](https://github.com/getquill/quill/pull/1478)
- [Explicitly pass AsyncContext type params](https://github.com/getquill/quill/pull/1457)
- [Remove unneeded Tuple reduction clause](https://github.com/getquill/quill/pull/1481)
- [Fix join subquery+map+distinct and sortBy+distinct](https://github.com/getquill/quill/pull/1462)
- [Fix Java9 depreciation message](https://github.com/getquill/quill/pull/1504)

Noteworthy Version Bumps:
- monix - 3.0.0-RC3
- cassandra-driver-core - 3.7.2
- orientdb-graphdb - 3.0.21
- postgresql - 42.2.6
- sqlite-jdbc - 3.28.0

Migration Notes:
- The `returning` method no long excludes the specified ID column from the insertion as it used to.
Use the `returningGenerated` method in order to achieve that.
See the 'Database-generated values' section of the documentation for more detail.
- The `==` method now works Scala-idiomatically.
That means that when two `Option[T]`-wrapped columns are compared, `None == None` will now yield `true`.
The `===` operator can be used in order to compare `Option[T]`-wrapped columns in a ANSI-SQL
idiomatic way i.e. `None == None := false`. See the 'equals' section of the documentation for more detail.

# 3.2.0

- [Allow == for Option[T] and/or T columns](https://github.com/getquill/quill/pull/1437)
- [Introducing Code Generator](https://github.com/getquill/quill/pull/1396)
- [Fix variable shadowing issue in action metas](https://github.com/getquill/quill/pull/1412)
- [Change effect to protected](https://github.com/getquill/quill/pull/1413)
- [Update spark-sql to 2.4.1](https://github.com/getquill/quill/pull/1398)
- [Update orientdb-graphdb to 3.0.17](https://github.com/getquill/quill/pull/1379)
- [Update sqlite-jdbc to 3.27.2.1](https://github.com/getquill/quill/pull/1392)

# 3.1.0

- [oracle support](https://github.com/getquill/quill/pull/1295)
- [quill cassandra for lagom](https://github.com/getquill/quill/pull/1299)
- [Fix the problem with re-preparing already prepared statements](https://github.com/getquill/quill/issues/1268)
- [Rely on ANSI null-fallthrough where possible](https://github.com/getquill/quill/pull/1341)
- [Fix for non-fallthrough null operations in map/flatMap/exists](https://github.com/getquill/quill/pull/1302)
- [Move basic encoders into EncodingDsl](https://github.com/getquill/quill/pull/1327)
- [Make string column name as property](https://github.com/getquill/quill/pull/1332)
- [Update MySQL driver/datasource](https://github.com/getquill/quill/pull/1326)
- [Provide a better "Can't tokenize a non-scalar lifting" error message](https://github.com/getquill/quill/pull/1311)

# 3.0.1

- [Fix Monix JDBC Connection Leak](https://github.com/getquill/quill/pull/1313)
- [Action task needs to flatMap](https://github.com/getquill/quill/pull/1307)

# 3.0.0

- [First-class support for dynamic queries](https://github.com/getquill/quill/pull/1262)
- [support dynamic strings within infix](https://github.com/getquill/quill/pull/1256)
- [Create a streaming module for Monix over JDBC - combined approach](https://github.com/getquill/quill/pull/1263)
- [Better implementation of Spark nested objects.](https://github.com/getquill/quill/pull/1227)
- [Spark 2.4 (with Scala 2.12 support)](https://github.com/getquill/quill/pull/1250)
- [Create quill-cassandra-monix](https://github.com/getquill/quill/pull/1278)
- [Move `io.getquill.CassandraStreamContext` into `quill-cassandra-streaming-monix` module](https://github.com/getquill/quill/pull/1259)
- [filterIf method for dynamic queries](https://github.com/getquill/quill/pull/1281)
- [Make UDT encoding to support options](https://github.com/getquill/quill/pull/1266)
- [fix column name conflict](https://github.com/getquill/quill/pull/1254)
- [#1204 add explicit `AS` for aliases (except table context)](https://github.com/getquill/quill/pull/1252)
- [sqlite dialect - translate boolean literals into 1/0](https://github.com/getquill/quill/pull/1248)
- [sqlite dialect - ignore null ordering](https://github.com/getquill/quill/pull/1247)
- [fail is property is not a case accessor](https://github.com/getquill/quill/pull/1246)
- [verify table references](https://github.com/getquill/quill/pull/1244)
- [fix property renaming for nested queries within infixes](https://github.com/getquill/quill/pull/1243)
- [expand map.distinct](https://github.com/getquill/quill/pull/1242)
- [quill-spark: fix groupby with multiple columns](https://github.com/getquill/quill/pull/1240)
- [quill-spark: escape strings](https://github.com/getquill/quill/pull/1239)
- [StatementInterpolator performance improvements](https://github.com/getquill/quill/pull/1280)
- [fix async transactions for scala future + io monad](https://github.com/getquill/quill/pull/1187)
- [Update orientdb-graphdb to 3.0.13](https://github.com/getquill/quill/pull/1286)
- [update guava version to 27.0.1-jre](https://github.com/getquill/quill/pull/1294)
- [documentation improvements](https://github.com/getquill/quill/pull/1241)

### Migration notes

- `io.getquill.CassandraStreamContext` is moved into `quill-cassandra-monix` module and now uses Monix 3.
- `io.getquill.CassandraMonixContext` has been introduced which should eventually replace `io.getquill.CassandraStreamContext`.
- Spark queries with nested objects will now rely on the star `*` operator and `struct` function to generate sub-schemas as opposed to full expansion of the selection.
- Most functionality from `JdbcContext` has been moved to `JdbcContextBase` for the sake of re-usability. `JdbcContext` is only intended to be used for synchronous JDBC.

# 2.6.0

- [add noFailFast option to FinagleMysqlContextConfig](https://github.com/getquill/quill/pull/1212)
- [add transactionWithIsolation to FinagleMysqlContext](https://github.com/getquill/quill/pull/1213)
- [Add encoding between java.time.ZonedDateTime and java.util.Date](https://github.com/getquill/quill/pull/1189)
- [Fix Infix causing ignoring renamings](https://github.com/getquill/quill/pull/1183)
- [Cassandra async improvements](https://github.com/getquill/quill/pull/1149)
- [Add upsert support for SQLite](https://github.com/getquill/quill/pull/1191)
- [add IO.lift](https://github.com/getquill/quill/pull/1146)
- [Minor performance improvements](https://github.com/getquill/quill/pull/1148)
- [Add encoder/decoder for Byte](https://github.com/getquill/quill/pull/1143)
- [Use Option.getOrElse(boolean) to generate ... OR IS [NOT] NULL queries](https://github.com/getquill/quill/pull/1142)
- [Upgrade finagle to 18.8.0](https://github.com/getquill/quill/pull/1144)
- [Fix renaming fields with schema/query meta for queries where unary/binary operation produces nested query](https://github.com/getquill/quill/pull/1134)
- [scala-js 0.6.24](https://github.com/getquill/quill/pull/1125)
- [Add question mark escaping for Spark](https://github.com/getquill/quill/pull/1112)
- [Allow mapping MySQL `TIMESTAMP` and `DATETIME` to Joda `DateTime` type.](https://github.com/getquill/quill/pull/1116)
- [added error message example in the documentation.](https://github.com/getquill/quill/pull/1113)
- [Wrong timeout configs](https://github.com/getquill/quill/pull/1089)
- [Fix unnecessary nesting of infix queries](https://github.com/getquill/quill/pull/1131)

### Migration notes

- When the infix starts with a query, the resulting sql query won't be nested

# 2.5.4

- [Adds master-slave capability to FinagleMysqlContext](https://github.com/getquill/quill/pull/1079)
- [Fix concatenation operator for SQL Server](https://github.com/getquill/quill/commit/83b80a1e894e53608e1ccca80a85038e7351b0e1)
- [Use PreparedStatement.getConnection for JDBC Array Encoders](https://github.com/getquill/quill/pull/1086)
- [CassandraSessionContext : change session to a lazy val](https://github.com/getquill/quill/pull/1082)

# 2.5.0, 2.5.1, 2.5.2, and 2.5.3

Broken releases, do not use.

# 2.4.2

- [Fix for Nested Objects with Spark](https://github.com/getquill/quill/pull/1074)

# 2.4.1

- [Add support of upsert for Postgres and MySQL](https://github.com/getquill/quill/pull/987)
- [Add flatMap, flatten, getOrElse and Option.apply](https://github.com/getquill/quill/pull/1044)
- [`quill-cassandra`: Add encoding for `Byte` and `Short`](https://github.com/getquill/quill/pull/1049)
- [Fix renaming aggregated properties in groupBy with custom querySchema](https://github.com/getquill/quill/pull/1038)
- [Change referencing `super.prepare` call to `this.prepare` in quill-cassandra contexts](https://github.com/getquill/quill/pull/1067)
- [Add connectTimeout option into FinagleMysqlContextConfig](https://github.com/getquill/quill/pull/1070)

# 2.3.3

- [Dependency updates](https://github.com/getquill/quill/pull/1042)
- [update finagle-postgres to 0.7.0](https://github.com/getquill/quill/pull/1030)
- [fixing unions with Ad-Hoc tuples](https://github.com/getquill/quill/pull/1026)
- [Fix removing assignment in returning insert if embedded field has columns with the same name as in parent case class](https://github.com/getquill/quill/pull/1010)

# 2.3.2

- [Simplify multiple `AND` `OR` sql generation](https://github.com/getquill/quill/pull/1003)
- [Fix SQLServer take/drop SQL syntax](https://github.com/getquill/quill/pull/1005)
- [Fix for Ad-Hoc Case Class producing Dynamic Queries](https://github.com/getquill/quill/pull/1000)
- [Fix throwing exception instead of failed future in cassandra async prepare](https://github.com/getquill/quill/pull/996)
- [Fix invalid alias with distinct](https://github.com/getquill/quill/pull/986)
- [Log errors instead of throwing exception directly in several places](https://github.com/getquill/quill/pull/995)
- [Update finagle to 17.12.0](https://github.com/getquill/quill/pull/999)

# 2.3.1

- [Fix Ad-Hoc Case Classes for Spark](https://github.com/getquill/quill/pull/981)
- [Make the error reporting of comparing `Option` to `null` to point actual position](https://github.com/getquill/quill/pull/982)
- [Fix postgres query probing failing for queries with wildcards](https://github.com/getquill/quill/pull/983)
- [Dependency updates](https://github.com/getquill/quill/pull/977)
- [Update finagle to 17.11.0](https://github.com/getquill/quill/pull/976)

# 2.3.0

- [Ad-Hoc Tuple Support in Quotations](https://github.com/getquill/quill/pull/957)
- [fix regression: allow table references](https://github.com/getquill/quill/pull/968)

# 2.2.0

- [Fix StackOverflowError in select distinct with aggregation](https://github.com/getquill/quill/pull/954)
- [Add support of java.time.Instant/java.time.LocalDate for quill-casandra](https://github.com/getquill/quill/pull/953)
- [Fix select query for unlimited optional embedded case classes](https://github.com/getquill/quill/pull/955)
- [`concatMap`, `startsWith`, and `split` support](https://github.com/getquill/quill/pull/956)
- [Upgrade finagle to 17.10.0](https://github.com/getquill/quill/pull/959)

# 2.1.0

- [Spark SQL support](https://github.com/getquill/quill/pull/941)
- [Add support of postgres sql arrays operators](https://github.com/getquill/quill/pull/940)
- [Fix reversed log parameter binds](https://github.com/getquill/quill/pull/931)
- [Fix renaming properties for unlimited optional and raw `Embedded` case classes](https://github.com/getquill/quill/pull/934)
- [Improve coverage](https://github.com/getquill/quill/pull/933)
- [Dependency updates](https://github.com/getquill/quill/pull/937)
- [Converge of PostgreSQL and MySQL behavior](https://github.com/getquill/quill/pull/938)

# 2.0.0

We're proud to announce the Quill 2.0. **All** bugs were fixed, so this release doesn't have any known bugs!

- [IO monad](https://github.com/getquill/quill/pull/881)
- [fall back to dynamic queries if dialect/naming isn't available](https://github.com/getquill/quill/pull/886)
- [Cassandra UDT encoding](https://github.com/getquill/quill/pull/852)
- [Add support of 'contains' operation on Cassandra collections](https://github.com/getquill/quill/pull/813)
- [Add org.joda.time.DateTime and java.time.ZonedDateTime encoding for quill-async-postgres](https://github.com/getquill/quill/pull/896)
- [Update dependencies](https://github.com/getquill/quill/pull/927)
- [give a better error message for option.get](https://github.com/getquill/quill/pull/905)
- [Remove OrientDB async context](https://github.com/getquill/quill/pull/884)
- [remove anonymous class support](https://github.com/getquill/quill/pull/891)
- [Remove client.ping from the FinagleMysqlContext constructor](https://github.com/getquill/quill/pull/895)

## Fixes

[#872](https://github.com/getquill/quill/pull/872), [#874](https://github.com/getquill/quill/pull/874), [#875](https://github.com/getquill/quill/pull/875), [#877](https://github.com/getquill/quill/pull/877), [#879](https://github.com/getquill/quill/pull/879), [#889](https://github.com/getquill/quill/pull/889), [#890](https://github.com/getquill/quill/pull/890), [#892](https://github.com/getquill/quill/pull/892), [#894](https://github.com/getquill/quill/pull/894), [#897](https://github.com/getquill/quill/pull/897), [#899](https://github.com/getquill/quill/pull/899), [#900](https://github.com/getquill/quill/pull/900), [#903](https://github.com/getquill/quill/pull/903), [#902](https://github.com/getquill/quill/pull/902), [#904](https://github.com/getquill/quill/pull/904), [#906](https://github.com/getquill/quill/pull/906), [#907](https://github.com/getquill/quill/pull/907), [#908](https://github.com/getquill/quill/pull/908), [#909](https://github.com/getquill/quill/pull/909), [#910](https://github.com/getquill/quill/pull/910), [#913](https://github.com/getquill/quill/pull/913), [#915](https://github.com/getquill/quill/pull/915), [#917](https://github.com/getquill/quill/pull/917), [#920](https://github.com/getquill/quill/pull/920), [#921](https://github.com/getquill/quill/pull/921), [#925](https://github.com/getquill/quill/pull/925), [#928](https://github.com/getquill/quill/pull/928)

### Migration notes

- Sources now take a parameter for idiom and naming strategy instead of just type parameters. For instance, `new SqlSource[MysqlDialect, Literal]` becomes `new SqlSource(MysqlDialect, Literal)`.
- Composite naming strategies don't use mixing anymore. Instead of the type `Literal with UpperCase`, use parameter value `NamingStrategy(Literal, UpperCase)`.
- Anonymous classes aren't supported for function declaration anymore. Use a method with a type parameter instead. For instance, replace `val q = quote { new { def apply[T](q: Query[T]) = ... } }` by `def q[T] = quote { (q: Query[T] => ... }`

# 1.4.0

- [Allow unlimited nesting of embedded case classes and optionals](https://github.com/getquill/quill/pull/660)
- [Accept traversables for batch action](https://github.com/getquill/quill/pull/682)
- [Add joda time encoding to `quill-async`](https://github.com/getquill/quill/pull/837)
- [Remove unnecessary `java.sql.Types` usage in JDBC decoders](https://github.com/getquill/quill/pull/841)
- [Add mappedEncoder and mappedDecoder for AnyVal](https://github.com/getquill/quill/pull/839)
- [Support contains, exists, forall for optional embedded case classes with optional fields](https://github.com/getquill/quill/pull/838)
- [Improve error message for "Can't expand nested value ..." error](https://github.com/getquill/quill/pull/840)
- [Improve error message for query probing](https://github.com/getquill/quill/pull/845)
- [Report the exactly tree position while typechecking the query](https://github.com/getquill/quill/pull/854)
- [Fix inserting single auto generated column](https://github.com/getquill/quill/pull/850)
- [Update finagle to 7.0.0](https://github.com/getquill/quill/pull/858)
- [Dependency updates](https://github.com/getquill/quill/pull/856)

### Migration notes

- `quill-async` contexts: `java.time.LocalDate` now supports only `date` sql types, `java.time.LocalDateTime` -
only `timestamp` sql types. Joda times follow this conventions accordingly.
Exception is made to `java.util.Date` it supports both `date` and `timestamp` types due to
historical moments (`java.sql.Timestamp` extents `java.util.Date`).
- `quill-jdbc` encoders do not accept `java.sql.Types` as a first parameter anymore.

# 1.3.0

- [SQLServer support](https://github.com/getquill/quill/pull/731)
- [OrientDB support](https://github.com/getquill/quill/pull/816)
- [Query bind variables logging](https://github.com/getquill/quill/pull/809)
- [Add url configuration property for quill-async](https://github.com/getquill/quill/pull/821)
- [Add support infix for batch actions](https://github.com/getquill/quill/pull/824)
- [Better support for empty lifted queries](https://github.com/getquill/quill/pull/794)
- [SQLLite 3.18.0](https://github.com/getquill/quill/pull/792)
- [Fix nested query stack overflow](https://github.com/getquill/quill/pull/786)
- [Performance optimization of Interleave](https://github.com/getquill/quill/pull/811)
- [Performance optimization of ReifyStatement](https://github.com/getquill/quill/pull/823)
- [Fix invalid nested queries with take/drop](https://github.com/getquill/quill/pull/779)
- [Fix NPE when using nested quoted binding](https://github.com/getquill/quill/pull/817)
- [Make `withConnection` method protected in AsyncContext](https://github.com/getquill/quill/pull/826)

# 1.2.1

- [upgrade finagle-postgres to 0.4.2](https://github.com/getquill/quill/pull/759)
- [add collections support for row elements (SQL Arrays, Cassandra Collection)](https://github.com/getquill/quill/pull/758)
- [allow querySchema/schemaMeta to rename optional embedded case classes](https://github.com/getquill/quill/pull/767)
- [make Quill compatible with Scala 2.12.2](https://github.com/getquill/quill/pull/770)
- [upgrade finagle-mysql to 6.44.0](https://github.com/getquill/quill/pull/772)

# 1.1.1

**see migration notes below**

- [avoid dynamic query generation for option.contains](https://github.com/getquill/quill/pull/696)
- [fix forall behaviour in quotation](https://github.com/getquill/quill/pull/699)
- [change query compilation log level to debug](https://github.com/getquill/quill/pull/709)
- [fix infix query compilation](https://github.com/getquill/quill/pull/711)
- [add support for Cassandra DATE type](https://github.com/getquill/quill/pull/723)
- [fix finagle timezone issues](https://github.com/getquill/quill/pull/722)
- [add max prepare statement configuration](https://github.com/getquill/quill/pull/722)
- [upgrade finagle-mysql to 6.43.0](https://github.com/getquill/quill/pull/726)
- [fix compilation issue when import List type](https://github.com/getquill/quill/pull/735)
- [upgrade cassandra-driver to 3.2.0](https://github.com/getquill/quill/pull/736)
- [apply NamingStrategy to returning column](https://github.com/getquill/quill/pull/734)
- [upgrade scala to 2.11.11](https://github.com/getquill/quill/pull/752)
- [fix finagle mysql context constructor with timezone](https://github.com/getquill/quill/pull/744)
- [rename Cassandra property address translater to translator](https://github.com/getquill/quill/pull/757)
- [fix timezone handling for finagle-mysql)](https://github.com/getquill/quill/pull/754)

### Migration notes

- Cassandra context property `ctx.session.addressTranslater` is renamed to `ctx.session.addressTranslator`

# 1.1.0

**see migration notes below**

- [materialize encoding for generic value classes](https://github.com/getquill/quill/pull/657)
- [sbt option to hide debug messages during compilation](https://github.com/getquill/quill/pull/633)
- [support Option.contains](https://github.com/getquill/quill/pull/669)
- [recursive optional nested expanding](https://github.com/getquill/quill/pull/681)
- [apply naming strategy to column alias](https://github.com/getquill/quill/pull/676)
- [fix existing and add missing encoders and decoders for java.util.UUID](https://github.com/getquill/quill/pull/665)
- [upgrade finagle-postgres to 0.3.2](https://github.com/getquill/quill/pull/692)

### Migration notes

- JDBC contexts are implemented in separate classes - `PostgresJdbcContext`, `MysqlJdbcContext`, `SqliteJdbcContext`, `H2JdbcContext`
- all contexts are supplied with default `java.util.UUID` encoder and decoder

# 1.0.1

- [include SQL type info in Encoder/Decoder](https://github.com/getquill/quill/pull/588)
- [make encoder helpers and wrapper type public for quill-finagle-postgres](https://github.com/getquill/quill/pull/608)
- [fix property renaming normalization order](https://github.com/getquill/quill/pull/609)
- [workaround compiler bug involving reflective calls](https://github.com/getquill/quill/pull/612)
- [fix flat joins support](https://github.com/getquill/quill/pull/613)
- [encoders and decoders refactoring](https://github.com/getquill/quill/pull/614)
- [avoid alias conflict for multiple nested explicit joins](https://github.com/getquill/quill/pull/616)
- [avoid merging filter condition into a groupBy.map](https://github.com/getquill/quill/pull/618)
- [move Embedded from `io.getquill.dsl.MetaDsl` inner context to `io.getquill` package](https://github.com/getquill/quill/pull/621)
- [make `currentConnection` protected](https://github.com/getquill/quill/pull/628)
- [add abstract encoders/decoders to CassandraContext and uuid mirror encoder/decoder](https://github.com/getquill/quill/pull/635)
- [made the SQL types for AsyncEncoder/AsyncDecoder generic](https://github.com/getquill/quill/pull/637)

# 1.0.0-RC1 - 20-Oct-2016

- [introduce `finagle-postgres`](https://github.com/getquill/quill/pull/560)
- [introduce meta dsl](https://github.com/getquill/quill/pull/554)
- [expand meta dsl](https://github.com/getquill/quill/pull/590)
- [encoder for java 8 LocalDate & LocalDateTime](https://github.com/getquill/quill/pull/583)
- [Upgraded to Monix 2.x](https://github.com/getquill/quill/pull/578)
- [Make withClient function not private](https://github.com/getquill/quill/pull/575)
- [pass ssl settings to async driver](https://github.com/getquill/quill/pull/587)

### Migration notes

- New API for schema definition: `query[Person].schema(_.entity("people").columns(_.id -> "person_id")` becomes `querySchema[Person]("People", _.id -> "person_id")`. Note that the entity name ("People") is now always required.
- `WrappedValue[T]` no longer exists, Quill can now automatically encode `AnyVal`s.

# 0.10.0 - 5-Sep-2016

**see migration notes below**

* [check types when parsing assignments and equality operations](https://github.com/getquill/quill/pull/532)
* [Update finagle-mysql to finagle 6.37.0](https://github.com/getquill/quill/pull/549/files)
* [Split quill-async into quill-async-mysql and quill-async-postgres](https://github.com/getquill/quill/pull/540)
* [cql: support `+` operator](https://github.com/getquill/quill/pull/530)
* [cassandra context constructor with ready-made Cluster](https://github.com/getquill/quill/pull/529)
* [support forced nested queries](https://github.com/getquill/quill/pull/527)
* [support mapped encoding definition without a context instance](https://github.com/getquill/quill/pull/526)
* [fix class cast exception for returned values](https://github.com/getquill/quill/pull/536)
* [fix free variables detection for the rhs of joins](https://github.com/getquill/quill/pull/528)

### Migration notes

- `mappedEncoding` has been renamed to `MappedEncoding`.
- The way we add async drivers has been changed. To add mysql async to your project use `quill-async-mysql` and for postgre async `quill-async-postgres`. It is no longer necessary to add `quill-async` by yourself.
- Action assignments and equality operations are now typesafe. If there's a type mismatch between the operands, the quotation will not compile.

# 0.9.0 - 22-Aug-2016

**see migration notes below**

* [new encoding, macros refactoring, and additional fixes](https://github.com/getquill/quill/pull/512)
* [Refactor generated to returning keyword in order to return the correct type](https://github.com/getquill/quill/pull/444)
* [Allow finagle-mysql to use Long with INT columns](https://github.com/getquill/quill/pull/467)
* [create sub query if aggregation on distinct query](https://github.com/getquill/quill/pull/472)
* [upgrade dependency to finagle 6.36.0](https://github.com/getquill/quill/pull/479)
* [Make decoder function public](https://github.com/getquill/quill/pull/487)
* [set the scope of all cassandra context type definitions to public](https://github.com/getquill/quill/pull/492)
* [make the cassandra decoder fail when encountering a column with value null](https://github.com/getquill/quill/pull/499)
* [fix Option.\{isEmpty, isDefined, nonEmpty\} show on action.filter](https://github.com/getquill/quill/pull/505)
* [Encoder fix](https://github.com/getquill/quill/pull/503/files)
* [enclose operand-queries of SetOperation in parentheses](https://github.com/getquill/quill/pull/510)

### Migration notes

* The fallback mechanism that looks for implicit encoders defined in the context instance has been removed. This means that if you don't `import context._`, you have to change the specific imports to include the encoders in use.
* `context.run` now receives only one parameter. The second parameter that used to receive runtime values now doesn't exist anymore. Use [`lift` or `liftQuery`](https://github.com/getquill/quill/#bindings) instead.
* Use [`liftQuery` + `foreach`](https://github.com/getquill/quill/#bindings) to perform batch actions and define contains/in queries.
* `insert` now always receives a parameter, that [can be a case class](https://github.com/getquill/quill/#actions).
- Non-lifted collections aren't supported anymore. Example: `query[Person].filter(t => List(10, 20).contains(p.age))`. Use `liftQuery` instead.
* `schema(_.generated())` has been replaced by [`returning`](https://github.com/getquill/quill/#schema).

# 0.8.0 / 17-Jul-2016

**see migration notes below**

* [introduce contexts](https://github.com/getquill/quill/pull/417)
* [sqlite support](https://github.com/getquill/quill/pull/449)
* [scala.js support](https://github.com/getquill/quill/pull/452)
* [support `toInt` and `toLong`](https://github.com/getquill/quill/pull/428)
* [quill-jdbc: support nested `transaction` calls](https://github.com/getquill/quill/pull/430)
* [fix bind order for take/drop with extra param](https://github.com/getquill/quill/pull/429)
* [quotation: allow lifting of `AnyVal`s ](https://github.com/getquill/quill/pull/421)
* [make liftable values work for the cassandra module](https://github.com/getquill/quill/pull/425)
* [apply intermediate map before take/drop](https://github.com/getquill/quill/pull/419)
* [support decoding of optional single-value case classes](https://github.com/getquill/quill/pull/420)
* [make type aliases for `run` results public](https://github.com/getquill/quill/pull/440)
* [fail compilation if query is defined outside a `quote`](https://github.com/getquill/quill/pull/433)
* [fix empty sql string](https://github.com/getquill/quill/pull/443)

### Migration notes

This version [introduces `Context`](https://github.com/getquill/quill/pull/417) as a replacement for `Source`. This change makes the quotation creation dependent on the context to open the path for a few refactorings and improvements we're planning to work on before the `1.0-RC1` release.

Migration steps:

- Remove any import that is not `import io.getquill._`
- Replace the `Source` creation by a `Context` creation. See the [readme](https://github.com/getquill/quill#contexts) for more details. All types necessary to create the context instances are provided by `import io.getquill._`.
- Instead of importing from `io.getquill._` to create quotations, import from you context instance `import myContext._`. The context import will provide all types and methods to interact with quotations and the database.
- See the documentation about [dependent contexts](https://github.com/getquill/quill#dependent-contexts) in case you get compilation errors because of type mismatches.

# 0.7.0 / 2-Jul-2016

* [transform quoted reference](https://github.com/getquill/quill/pull/416)
* [simplify `finagle-mysql` action result type](https://github.com/getquill/quill/pull/358)
* [provide default values for plain-sql query execution](https://github.com/getquill/quill/pull/360)
* [quotation: fix binding conflict](https://github.com/getquill/quill/pull/363)
* [don't consider `?` a binding if inside a quote](https://github.com/getquill/quill/pull/361)
* [fix query generation for wrapped types](https://github.com/getquill/quill/pull/364)
* [use querySingle/query for parametrized query according to return type](https://github.com/getquill/quill/pull/375)
* [remove implicit ordering](https://github.com/getquill/quill/pull/378)
* [remove implicit from max and min](https://github.com/getquill/quill/pull/384)
* [support explicit `Predef.ArrowAssoc` call](https://github.com/getquill/quill/pull/386)
* [added handling for string lists in ClusterBuilder](https://github.com/getquill/quill/pull/395)
* [add naming strategy for pluralized table names](https://github.com/getquill/quill/pull/396)
* [transform ConfiguredEntity](https://github.com/getquill/quill/pull/409)

# 0.6.0 / 9-May-2016

* [explicit bindings using `lift`](https://github.com/getquill/quill/pull/335/files#diff-04c6e90faac2675aa89e2176d2eec7d8R157)
* [Code of Conduct](https://github.com/getquill/quill/pull/296)
* [dynamic type parameters](https://github.com/getquill/quill/pull/351)
* [support contains for Traversable](https://github.com/getquill/quill/pull/290)
* [`equals` support](https://github.com/getquill/quill/pull/328)
* [Always return List for any type of query](https://github.com/getquill/quill/pull/324)
* [quill-sql: support value queries](https://github.com/getquill/quill/pull/354)
* [quill-sql: `in`/`contains` - support empty sets](https://github.com/getquill/quill/pull/329)
* [Support `Ord` quotation](https://github.com/getquill/quill/pull/301)
* [`blockParser` off-by-one error](https://github.com/getquill/quill/pull/292)
* [show ident instead of indent.toString](https://github.com/getquill/quill/pull/307)
* [decode bit as boolean](https://github.com/getquill/quill/pull/308)

# 0.5.0 / 17-Mar-2016

* [Schema mini-DSL and generated values](https://github.com/getquill/quill/pull/226/files#diff-04c6e90faac2675aa89e2176d2eec7d8R212)
* [Support for inline vals in quotation blocks](https://github.com/getquill/quill/pull/271/files#diff-02749abf4d0d51be99715cff7074bc9eR775)
* [Support for Option.\{isEmpty, nonEmpty, isDefined\}](https://github.com/getquill/quill/pull/238/files#diff-02749abf4d0d51be99715cff7074bc9eR688)
* [Tolerant function parsing in option operation](https://github.com/getquill/quill/pull/243/files#diff-6858983f3617753cfb9852426edaa121R481)
* [quill-sql: rename properties and assignments](https://github.com/getquill/quill/pull/250)
* [quill-cassandra: rename properties and assignments](https://github.com/getquill/quill/pull/254)
* [Fix log category](https://github.com/getquill/quill/pull/259)
* [Accept unicode arrows](https://github.com/getquill/quill/pull/257)
* [Add set encoder to SqlSource](https://github.com/getquill/quill/pull/258)
* [Don't quote the source creation tree if query probing is disabled](https://github.com/getquill/quill/pull/268)
* [Bind `drop.take` according to the sql terms order](https://github.com/getquill/quill/pull/278)
* [Avoid silent error when importing the source's implicits for the encoding fallback resolution](https://github.com/getquill/quill/pull/279)
* [Quotation: add identifier method to avoid wrong type refinement inference](https://github.com/getquill/quill/pull/280)
* [Unquote multi-param quoted function bodies automatically](https://github.com/getquill/quill/pull/277)

# 0.4.1 / 28-Feb-2016

* [quill-sql: h2 dialect](https://github.com/getquill/quill/pull/189)
* [support for auto encoding of wrapped types](https://github.com/getquill/quill/pull/199/files#diff-04c6e90faac2675aa89e2176d2eec7d8R854)
* [non-batched actions](https://github.com/getquill/quill/pull/202/files#diff-04c6e90faac2675aa89e2176d2eec7d8L485)
* `distinct` support [[0]](https://github.com/getquill/quill/pull/186/files#diff-02749abf4d0d51be99715cff7074bc9eR38) [[1]](https://github.com/getquill/quill/pull/212)
* [postgres naming strategy](https://github.com/getquill/quill/pull/218)
* [quill-core: unquote quoted function bodies automatically](https://github.com/getquill/quill/pull/222)
* [don't fail if the source annotation isn't available](https://github.com/getquill/quill/pull/210)
* [fix custom aggregations](https://github.com/getquill/quill/pull/207)
* [quill-finagle-mysql: fix finagle mysql execute result loss](https://github.com/getquill/quill/pull/197)
* [quill-cassandra: stream source - avoid blocking queries](https://github.com/getquill/quill/pull/185)

# 0.4.0 / 19-Feb-2016

* [new sources creation mechanism](https://github.com/getquill/quill/pull/136)
* [simplified join syntax](https://github.com/getquill/quill/commit/bfcfe49fbdbda04cce7fe7e7d382fb1adbfcbd7f)
* [Comparison between Quill and other alternatives for CQL](https://github.com/getquill/quill/pull/164)
* [`contains` operator (sql `in`)](https://github.com/getquill/quill/pull/165/files#diff-04c6e90faac2675aa89e2176d2eec7d8R377)
* [unary sql queries](https://github.com/getquill/quill/pull/179/files#diff-02749abf4d0d51be99715cff7074bc9eR207)
* [query probing is now opt-in](https://github.com/getquill/quill/pull/176/files#diff-04c6e90faac2675aa89e2176d2eec7d8R453)
* [quill-cassandra: upgrade Datastax Java Driver to version 3.0.0](https://github.com/getquill/quill/pull/171)
* [support implicit quotations with type parameters](https://github.com/getquill/quill/pull/163)
* [quill-cassandra: UUID support](https://github.com/getquill/quill/pull/142)
* [quill-async: more reasonable numeric type decodes](https://github.com/getquill/quill/pull/139)

# 0.3.1 / 01-Feb-2016

* [fix #134 - ignore the property `queryProbing` when creating the hikari data source](https://github.com/getquill/quill/issues/134)

# 0.3.0 / 26-Jan-2016

* [quill-cassandra: first version of the module featuring async and sync sources](https://github.com/getquill/quill/#cassandra-sources)
* [quill-cassandra: reactive streams support via Monix](https://github.com/getquill/quill/#cassandra-sources)
* [quill-core: updates using table columns](https://github.com/getquill/quill/commit/0681b21aad8d75cb7793840c4f905b80645872cc#diff-04c6e90faac2675aa89e2176d2eec7d8R458)
* [quill-core: explicit inner joins](https://github.com/getquill/quill/commit/902eb858e0e844f41978f8179156da9c69f2d847#diff-2e097508346e0e431a36abcb2c1cc4cbR270)
* [quill-core: configuration option to disable the compile-time query probing](https://github.com/getquill/quill/commit/130919d62a1f852c2d26203c035361ccb3284e53#diff-04c6e90faac2675aa89e2176d2eec7d8L840)
* [quill-core: `if/`else` support (sql `case`/`when`)](https://github.com/getquill/quill/commit/16674ba77fdc880a64af719d150560351ac6a8f6#diff-2e097508346e0e431a36abcb2c1cc4cbR598)
* [quill-async: uuid encoding](https://github.com/getquill/quill/commit/743227aaa3ec76cefcffb405ac658069d90118fc#diff-7bfbe03bba9c515d3f16f88115eb2f9fR24)
* [quill-core: custom ordering](https://github.com/getquill/quill/commit/2fe7556279c5919aa9c1e22bf9c8caf4c67e53e7#diff-04c6e90faac2675aa89e2176d2eec7d8R257)
* [quill-core: expressions in sortBy](https://github.com/getquill/quill/commit/0dbb492de7334cb8ad34dc5c6246ec6908d328bc#diff-2e097508346e0e431a36abcb2c1cc4cbR107)

# 0.2.1 / 28-Dec-2015

* [expire and close compile-time sources automatically](https://github.com/getquill/quill/issues/10)
* [Aggregation sum should return an Option](https://github.com/getquill/quill/pull/69)
* [Changed min/max implicit from Numeric to Ordering](https://github.com/getquill/quill/pull/70)
* [provide implicit to query case class companion objects directly](https://github.com/getquill/quill/pull/73)
* [don't fuse multiple `sortBy`s](https://github.com/getquill/quill/issues/71)
* [actions now respect the naming strategy](https://github.com/getquill/quill/issues/74)

# 0.2.0 / 24-Dec-2015

* [Insert/update using case class instances](https://github.com/getquill/quill/commit/aed630bdb514b3d71a3a3cc47299ff28c0472023)
* [Better IntelliJ IDEA support](https://github.com/getquill/quill/issues/23)
* [Implicit quotations](https://github.com/getquill/quill/commit/1991d694a2bdad645d6d169acefba51f90acde62#diff-6858983f3617753cfb9852426edaa121R491)
* [`like` operator](https://github.com/getquill/quill/commit/f05763ff6cfbe850d7cab2e15d570603cad194c4#diff-04c6e90faac2675aa89e2176d2eec7d8R420)
* [string interpolation support](https://github.com/getquill/quill/commit/c510ee8a6daa98caf45743fd7fc75230cbb3d71e#diff-6858983f3617753cfb9852426edaa121R290)
* [Finagle pool configuration](https://github.com/getquill/quill/pull/60)
* [Allow empty password in Finagle Mysql client](https://github.com/getquill/quill/pull/59)
* Bug fixes:
	* https://github.com/getquill/quill/issues/53
	* https://github.com/getquill/quill/issues/62
	* https://github.com/getquill/quill/issues/63

# 0.1.0 / 27-Nov-2015

* Initial release

---

## Code Generation

Quill now has a highly customizable code generator. Currently, it only supports JDBC but it will soon be extended to other contexts. With a minimal amount of configuration, the code generator takes schemas like this:

```sql
-- Using schema 'public'

create table public.Person (
  id int primary key auto_increment,
  first_name varchar(255),
  last_name varchar(255),
  age int not null
);

create table public.Address (
  person_fk int not null,
  street varchar(255),
  zip int
);
```

Producing objects like this:

```scala
// src/main/scala/com/my/project/public/Person.scala
package com.my.project.public

case class Person(id: Int, firstName: Option[String], lastName: Option[String], age: Int)
```

```scala
// src/main/scala/com/my/project/public/Address.scala
package com.my.project.public

case class Address(personFk: Int, street: Option[String], zip: Option[Int])
```

## Quill Code Generator

This library gives you a few options as to what kind of
schema to generate from JDBC metadata for Quill. You can choose to generate
simple case classes that are controlled entirely but a Quill Naming Strategy,
or a combination of case classes and `querySchema`s.
You can also choose whether they should written to one file, multiple files, or
just a list of strings (useful for executing directly into a repl).
Thanks to the Slick code generator creators for providing inspiration for this library!

Currently the code generator is only available for JDBC databases but it will
be extended in the future for Cassandra as well as others.

You can import the Code Generator using maven:
````xml
<dependency>
  <groupId>io.getquill</groupId>
  <artifactId>quill-codegen-jdbc_2.13</artifactId>
  <version>3.10.0</version>
</dependency>
```` 

Or using sbt:
````scala
libraryDependencies += "io.getquill" %% "quill-codegen-jdbc" % "4.8.6"
````

## SimpleJdbcCodegen

This code generator generates simple case classes, each representing a table
in a database. It does not generate Quill `querySchema` objects.
Create one or multiple CodeGeneratorConfig objects
and call the `.writeFiles` or `.writeStrings` methods
on the code generator to generate the code.

Given the following schema:
````sql
-- Using schema 'public'

create table public.Person (
  id int primary key auto_increment,
  first_name varchar(255),
  last_name varchar(255),
  age int not null
);

create table public.Address (
  person_fk int not null,
  street varchar(255),
  zip int
);
````

You can invoke the SimpleJdbcCodegen like so:

````scala
// provide DB credentials with a com.typesafe.config.Config object
// (under the hood the credentials are used to create a HikariPool DataSource)

val snakecaseConfig = LoadConfig(configPrefix: String)
val gen = new SimpleJdbcCodegen(snakecaseConfig, "com.my.project") {
    override def nameParser = SnakeCaseNames
}
gen.writeFiles("src/main/scala/com/my/project")

// or, provide an initialized DataSource

val pgDataSource = new PGSimpleDataSource()
pgDataSource.setURL(
  "jdbc:postgresql://127.0.0.1:5432/quill_codegen_example?ssl=false",
)
pgDataSource.setUser("my_user")
pgDataSource.setPassword("my_password")
val gen = new SimpleJdbcCodegen(pgDataSource, "com.my.project") {
    override def nameParser = SnakeCaseNames
}
gen.writeFiles("src/main/scala/com/my/project")
````

You can parse column and table names using either the `SnakeCaseNames` or the and the `LiteralNames` parser
which are used with the respective Quill Naming Strategies. They cannot be customized further with this code generator.

The following case classes will be generated
````scala
// src/main/scala/com/my/project/public/Person.scala
package com.my.project.public
  
case class Person(id: Int, firstName: Option[String], lastName: Option[String], age: Int)
````

````scala
// src/main/scala/com/my/project/public/Address.scala
package com.my.project.public
  
case class Address(personFk: Int, street: Option[String], zip: Option[Int])
````

If you wish to generate schemas with custom table or column names, you need to use the `ComposeableTraitsJdbcCodegen`
in order to generate your schemas with `querySchema` objects.

## Composeable Traits Codegen

The `ComposeableTraitsJdbcCodegen` enables more customized code generation.
It allows you to determine the tables to generate entity classes for,
their naming strategy, the types for columns in Scala,
and generates the necessary `querySchema` object in order to map the fields.
Additionally, it generates a database-independent query schema trait which can be composed
with a `Context` object of your choice.

Given the following schema:
````sql
create table public.Person (
  id int primary key auto_increment,
  first_name varchar(255),
  last_name varchar(255),
  age int not null
);

create table public.Address (
  person_fk int not null,
  street varchar(255),
  zip int
);
````

Here is a example of how you could use the `ComposeableTraitsJdbcCodegen` in order to replace the
`first_name` and `last_name` properties with `first` and `last`.

````scala
val gen = new ComposeableTraitsJdbcCodegen(
  configOrDataSource,
  packagePrefix = "com.my.project",
  nestedTrait = true) {

  override def nameParser: NameParser = CustomNames(
    columnParser = col => col.columnName.toLowerCase.replace("_name", "")
  )

  override def packagingStrategy: PackagingStrategy = PackagingStrategy.ByPackageHeader.TablePerSchema(packagePrefix)
}
gen.writeFiles("src/main/scala/com/my/project")
````

The following schema should be generated as a result.
````scala
package com.my.project.public

case class Person(id: Int, first: Option[String], last: Option[String], age: Int)

case class Address(person_fk: Int, street: Option[String], zip: Option[Int])

// Note that by default this is formatted as "${namespace}Extensions"
trait PublicExtensions[Idiom <: io.getquill.idiom.Idiom, Naming <: io.getquill.NamingStrategy] {
  this:io.getquill.context.Context[Idiom, Naming] =>

  object PublicSchema {
    object PersonDao {
        def query = quote {
            querySchema[Person](
              "public.person",
              _.id -> "id",
              _.first -> "first_name",
              _.last -> "last_name",
              _.age -> "age"
            )

          }

      }

      object AddressDao {
        def query = quote {
            querySchema[Address](
              "public.address",
              _.person_fk -> "person_fk",
              _.street -> "street",
              _.zip -> "zip"
            )

          }

      }
  }
}
````

Later when declaring your Quill database context you can compose the context with
the `PublicExtensions` like so:
````scala
object MyCustomContext extends SqlMirrorContext[H2Dialect, Literal](H2Dialect, Literal)
  with PublicExtensions[H2Dialect, Literal]
````

`ComposeableTraitsJdbcCodegen` is designed to be customizable via composition. This is a longer list of customizable strategies:

```scala

new ComposeableTraitsJdbcCodegen(...) {

  // whether to generate Scala code for a table
  override def filter(tc: RawSchema[JdbcTableMeta, JdbcColumnMeta]): Boolean = ???

  // how to name table / columns in Scala
  override def nameParser: NameParser = ???

  // how to organize generated code into files / packages
  override def packagingStrategy: PackagingStrategy = ???

  // what JVM types (classes) to use for DB column
  // e.g. one may want to translate Postgres `timestamptz` to java.time.OffsetDateTime
  override def typer: Typer = ???

  // what to do when `typer` above cannot find an appropriate type and returned None
  override def unrecognizedTypeStrategy: UnrecognizedTypeStrategy = ???
}

```

## Stereotyping
Frequently in corporate databases, the same kind of table is duplicated across multiple schemas, databases, etc...
for different business units. Typically, all the duplicates of the table will have nearly the same columns
with just minor differences. Stereotyped code-generation aims to take the 'lowest common denominator' of all these schemas
in order to produce a case class that can be used across all of them.

Examine the following H2 DDL:
````sql
create table Alpha.Person (
  id int primary key auto_increment,
  first_name varchar(255),
  last_name varchar(255),
  age int not null,
  foo varchar(255),
  num_trinkets int,
  trinket_type varchar(255) not null
);

create table Bravo.Person (
  id int primary key auto_increment,
  first_name varchar(255),
  bar varchar(255),
  last_name varchar(255),
  age int not null,
  num_trinkets bigint not null,
  trinket_type int not null
);
````

* Firstly, note that `Alpha.Person` and `Bravo.Person` have the exact same columns except for `foo` and `bar` respectively.
  If a common table definition `Person` is desired, these columns must be omitted.
* Secondly, note that their columns `num_trinkets` and `trinket_type` have different types.
  If a common table definition `Person` is desired, these columns must be expanded to the widest
  datatype of the two which is this case `bigint` for `num_trinkets` and `varchar(255)` for `trinket_type`.

Both of the above actions are automatically performed by the `ComposeableTraitsJdbcCodegen`
(and `SimpleJdbcCodegen`) automatically when multiple tables with the same name are detected or if you
rename them using a custom `namingStrategy` causing this to happen.
Here is an example of how that is done:

````scala
val gen = new ComposeableTraitsJdbcCodegen(twoSchemaConfig, "com.my.project") {
  override def namingStrategy: EntityNamingStrategy = CustomStrategy()
  override val namespacer: Namespacer =
    ts => if (ts.tableSchema.toLowerCase == "alpha" || ts.tableSchema.toLowerCase == "bravo") "common" else ts.tableSchema.toLowerCase
    
  // Be sure to set the querySchemaNaming correctly so that the different
  // querySchemas generated won't all be called '.query' in the common object (which would
  // case an un-compile-able schema to be generated).
  override def querySchemaNaming: QuerySchemaNaming = `[namespace][Table]`
}

gen.writeFiles("src/main/scala/com/my/project")
````

The following will then be generated. Note how `numTrinkets` is a `Long` (i.e. an SQL `bigint`) type and `trinketType` is a `String`
(i.e. an SQL varchar),

````scala
package com.my.project.common
  
case class Person(id: Int, firstName: Option[String], lastName: Option[String], age: Int, numTrinkets: Option[Long], trinketType: String)
  
trait CommonExtensions[Idiom <: io.getquill.idiom.Idiom, Naming <: io.getquill.NamingStrategy] {
  this:io.getquill.context.Context[Idiom, Naming] =>
  
  object PersonDao {
      def alphaPerson = quote {
          querySchema[Person](
            "ALPHA.PERSON",
            _.id -> "ID",
            _.firstName -> "FIRST_NAME",
            _.lastName -> "LAST_NAME",
            _.age -> "AGE",
            _.numTrinkets -> "NUM_TRINKETS",
            _.trinketType -> "TRINKET_TYPE"
          )
        }
  
      def bravoPerson = quote {
          querySchema[Person](
            "BRAVO.PERSON",
            _.id -> "ID",
            _.firstName -> "FIRST_NAME",
            _.lastName -> "LAST_NAME",
            _.age -> "AGE",
            _.numTrinkets -> "NUM_TRINKETS",
            _.trinketType -> "TRINKET_TYPE"
          )
        }
    }
}
````

Later when declaring your quill database context you can compose the context with
the `CommonExtensions` like so:

````scala
object MyCustomContext extends SqlMirrorContext[H2Dialect, Literal](H2Dialect, Literal)
  with CommonExtensions[H2Dialect, Literal]
````

---

## Compiler performance

Quill will probably make the slow scala compiler even slower, since a lot of additional `Parsing`, `Typechecking`, `Implicit resolution` works introduced to expand a Query.

Following tips may help improving compilation time.

## Use `-Yprofile-trace` scalac options.
With `-Yprofile-trace` option, a chrome trace file will be produced after compilation.
It will help figure out what slowing down the compiler.

Note, this option need some tweak if you are running on java 9 or newer version.

## Split large module into multiple submodules
Since scalac is not fully parallelized, split into independent submodules can significantly reduce build time on multi-core cpu.

## Define decoder/encoder directly instead of `MappedEncoding`

`MappedEncoding` introduce more implicit resolutions, which may slow down compiler.

It is possible to define instance directly.

```scala
case class FooId(id: Long)
implicit val fooIdEncoder: Encoder[FooId] = mappedEncoder(MappedEncoding[FooId, Long](_.id), longEncoder)
implicit val fooIdDecoder: Decoder[FooId] = mappedDecoder(MappedEncoding[Long, FooId](FooId(_)), longDecoder)
```

## Share `QueryMeta` instance

`QueryMeta` generation requires `Decoder` resolution, tree generation, typechecking, etc which can be very slow.

`QueryMeta` is not shared by default, so define shared `QueryMeta` instance may reduce build time.

```scala
val ctx = SqlMirrorContext(MirrorIdiom, Literal)

// Prevent using default macro generated query meta instance.
// Use `_` instead of `*` if `-Xsource:3` not enabled.

 // Instance type must not be specified here, otherwise it will become dynamic query.
implicit val orderQueryMeta = ctx.materializeQueryMeta[Order]

ctx.run {
  query[Order]
}
```
Note, to use [`-Xsource:3`](https://github.com/scala/scala/pull/10439) scalac options, `-Xmigration` or `-Wconf:cat=scala3-migration:w` is required.

Otherwise, it will not compile due to lack of explicit type of implicit definition.

---

## Contexts

Contexts represent the database and provide an execution interface for queries.

## Mirror context

Quill provides a mirror context for testing purposes. Instead of running the query, the mirror context returns a structure with the information that would be used to run the query. There are three mirror context instances:

- `io.getquill.MirrorContext`: Mirrors the quotation AST
- `io.getquill.SqlMirrorContext`: Mirrors the SQL query
- `io.getquill.CassandraMirrorContext`: Mirrors the CQL query

## Dependent contexts

The context instance provides all methods and types to interact with quotations and the database.
Contexts can be imported and passed around normally in constructors and function arguments.

```
class MyContext extends SqlMirrorContext(MirrorSqlDialect, Literal)
case class MySchema(c: MyContext) {

  val people = quote {
    querySchema[Person]("people")
  }
}
case class MyDao(c: MyContext, schema: MySchema) {
  def allPeople = c.run(schema.people)
}
```

### Context Traits

One way to compose applications with this kind of context is to use traits with an abstract context variable:

```scala
class MyContext extends SqlMirrorContext(MirrorSqlDialect, Literal)

trait MySchema {

  val c: MyContext

  val people = quote {
    querySchema[Person]("people")
  }
}

case class MyDao(c: MyContext) extends MySchema {

  def allPeople =
    c.run(people)
}
```

### Modular Contexts

Another simple way to modularize Quill code is by extending `Context` as a self-type and applying mixins. Using this strategy,
it is possible to create functionality that is fully portable across databases and even different types of databases
(e.g. creating common queries for both Postgres and Spark).

For example, create the following abstract context:

```scala
trait ModularContext[I <: Idiom, N <: NamingStrategy] { this: Context[I, N] =>
  def peopleOlderThan = quote {
    (age:Int, q:Query[Person]) => q.filter(p => p.age > age)
  }
}
```

Let's see how this can be used across different kinds of databases and Quill contexts.

#### Use `ModularContext` in a mirror context:

```scala
// Note: In some cases need to explicitly specify [MirrorSqlDialect, Literal].
val ctx =
  new SqlMirrorContext[MirrorSqlDialect, Literal](MirrorSqlDialect, Literal)
    with ModularContext[MirrorSqlDialect, Literal]

println( run(peopleOlderThan(22, query[Person])).string )
```

#### Use `ModularContext` to query a Postgres Database

```scala
val ctx =
  new PostgresJdbcContext[Literal](Literal, ds)
    with ModularContext[PostgresDialect, Literal]

val results = run(peopleOlderThan(22, query[Person]))
```

#### Use `ModularContext` to query a Spark Dataset

```scala
object CustomQuillSparkContext extends QuillSparkContext
  with ModularContext[SparkDialect, Literal]

val results = run(peopleOlderThan(22, liftQuery(dataset)))
```

## Spark Integration

Quill provides a fully type-safe way to use Spark's highly-optimized SQL engine. It's an alternative to `Dataset`'s weakly-typed API.

### Importing Quill Spark

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-spark" % "4.8.6"
)
```

### Usage

Unlike the other modules, the Spark context is a companion object. Also, it does not depend on a spark session. To use it, add the following import:

```scala

// Create your Spark Context
val session =
  SparkSession.builder()
    .master("local")
    .appName("spark test")
    .getOrCreate()

// The Spark SQL Context must be provided by the user through an implicit value:
implicit val sqlContext = session.sqlContext

// Import the Quill Spark Context

```

> Note Unlike the other modules, the Spark context is a companion object. Also, it does not depend on a spark session.

> Also Note: Quill decoders and meta instances are not used by the quill-spark module, Spark's `Encoder`s are used instead.

### Using Quill-Spark

The `run` method returns a `Dataset` transformed by the Quill query using the SQL engine.

```scala
// Typically you start with some type dataset.
val peopleDS: Dataset[Person] = spark.read.parquet("path/to/people").as[Person]
val addressesDS: Dataset[Address] = spark.read.parquet("path/to/addresses").as[Address]

// The liftQuery method converts Datasets to Quill queries:
val people = quote { liftQuery(peopleDS) }
val addresses = quote { liftQuery(addressesDS) }

val peopleAndAddresses = quote {
  (people join addresses).on((p, a) => p.id == a.ownerFk)
}

val peopleAndAddressesDS: Dataset[(Person, Address)] = run(peopleAndAddresses)
```

#### Simplify it
Since the `run` method allows for Quill queries to be specified directly, and `liftQuery` can be used inside
of any Quoted block, you can shorten various steps of the above workflow:

```scala
val peopleDS: Dataset[Person] = spark.read.parquet("path/to/people")
val addressesDS: Dataset[Address] = spark.read.parquet("path/to/addresses")

val peopleAndAddressesDS: Dataset[(Person, Address)] = run {
  liftQuery(peopleDS)
    .join(liftQuery(addressesDS))
    .on((p, a) => p.id == a.ownerFk)
}
```

Here is an example of a Dataset being converted into Quill, filtered, and then written back out.

```scala

def filter(myDataset: Dataset[Person], name: String): Dataset[Int] =
  run {
    liftQuery(myDataset).filter(_.name == lift(name)).map(_.age)
  }
// SELECT x1.age _1 FROM (?) x1 WHERE x1.name = ?
```

#### Workflow

Due to the design of Quill-Spark, it can be used interchangeably throughout your Spark workflow:
- Lift a Dataset to Query to do some filtering and sub-selecting
  (with [Predicate and Filter Pushdown!](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-Optimizer-PushDownPredicate.html)).
- Then covert it back to a Dataset to do Spark-Specific operations.
- Then convert it back to a Query to use Quills great Join DSL...
- Then convert it back to a Dataset to write it to a file or do something else with it...

### Custom Functions

TODO UDFs and UDAFs

### Restrictions

#### Top Level Classes
Spark only supports using top-level classes as record types. That means that
when using `quill-spark` you can only use a top-level case class for `T` in `Query[T]`.

TODO Get the specific error

#### Lifted Variable Interpolation

The queries printed from `run(myQuery)` during compile time escape question marks via a backslash them in order to
be able to substitute liftings properly. They are then returned back to their original form before running.
```scala

def filter(myDataset: Dataset[Person]): Dataset[Int] =
  run {
    liftQuery(myDataset).filter(_.name == "?").map(_.age)
  }
// This is generated during compile time:
// SELECT x1.age _1 FROM (?) x1 WHERE x1.name = '\?'
// It is reverted upon run-time:
// SELECT x1.age _1 FROM (ds1) x1 WHERE x1.name = '?'
```

## SQL Contexts

Example:

```scala
lazy val ctx = new MysqlJdbcContext(SnakeCase, "ctx")
```

### Dialect

The SQL dialect parameter defines the specific database dialect to be used. Some context types are specific to a database and thus not require it.

Quill has five built-in dialects:

- `io.getquill.H2Dialect`
- `io.getquill.MySQLDialect`
- `io.getquill.PostgresDialect`
- `io.getquill.SqliteDialect`
- `io.getquill.SQLServerDialect`
- `io.getquill.OracleDialect`

### Naming strategy

The naming strategy parameter defines the behavior when translating identifiers (table and column names) to SQL.

|           strategy                  |          example              |
|-------------------------------------|-------------------------------|
| `io.getquill.naming.Literal`        | some_ident  -> some_ident     |
| `io.getquill.naming.Escape`         | some_ident  -> "some_ident"   |
| `io.getquill.naming.UpperCase`      | some_ident  -> SOME_IDENT     |
| `io.getquill.naming.LowerCase`      | SOME_IDENT  -> some_ident     |
| `io.getquill.naming.SnakeCase`      | someIdent   -> some_ident     |
| `io.getquill.naming.CamelCase`      | some_ident  -> someIdent      |
| `io.getquill.naming.MysqlEscape`    | some_ident  -> \`some_ident\` |
| `io.getquill.naming.PostgresEscape` | $some_ident -> $some_ident    |

Multiple transformations can be defined using `NamingStrategy()`. For instance, the naming strategy

```NamingStrategy(SnakeCase, UpperCase)```

produces the following transformation:

```someIdent -> SOME_IDENT```

The transformations are applied from left to right.

### Configuration

The string passed to the context is used as the key in order to obtain configurations using the [typesafe config](https://github.com/typesafehub/config) library.

Additionally, the contexts provide multiple constructors. For instance, with `JdbcContext` it's possible to specify a `DataSource` directly, without using the configuration:

```scala
def createDataSource: javax.sql.DataSource with java.io.Closeable = ???

lazy val ctx = new MysqlJdbcContext(SnakeCase, createDataSource)
```

## quill-jdbc

The `quill-jdbc` module provides a simple blocking JDBC context for standard use-cases. For transactions, the JDBC connection is kept in a thread-local variable.

Quill uses [HikariCP](https://github.com/brettwooldridge/HikariCP) for connection pooling. Please refer to HikariCP's [documentation](https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby) for a detailed explanation of the available configurations.

Note that there are `dataSource` configurations, that go under `dataSource`, like `user` and `password`, but some pool settings may go under the root config, like `connectionTimeout`.

#### transactions

The `JdbcContext` provides thread-local transaction support:

```
ctx.transaction {
  ctx.run(query[Person].delete)
  // other transactional code
}
```

The body of `transaction` can contain calls to other methods and multiple `run` calls since the transaction is propagated through a thread-local.

### MySQL (quill-jdbc)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "mysql" % "mysql-connector-java" % "8.0.17",
  "io.getquill" %% "quill-jdbc" % "4.8.6"
)
```

#### context definition
```scala
lazy val ctx = new MysqlJdbcContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.dataSourceClassName=com.mysql.cj.jdbc.MysqlDataSource
ctx.dataSource.url=jdbc:mysql://host/database
ctx.dataSource.user=root
ctx.dataSource.password=root
ctx.dataSource.cachePrepStmts=true
ctx.dataSource.prepStmtCacheSize=250
ctx.dataSource.prepStmtCacheSqlLimit=2048
ctx.connectionTimeout=30000
```

### Postgres (quill-jdbc)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "org.postgresql" % "postgresql" % "42.2.8",
  "io.getquill" %% "quill-jdbc" % "4.8.6"
)
```

#### context definition
```scala
lazy val ctx = new PostgresJdbcContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource
ctx.dataSource.user=root
ctx.dataSource.password=root
ctx.dataSource.databaseName=database
ctx.dataSource.portNumber=5432
ctx.dataSource.serverName=host
ctx.connectionTimeout=30000
```

### Sqlite (quill-jdbc)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "org.xerial" % "sqlite-jdbc" % "3.28.0",
  "io.getquill" %% "quill-jdbc" % "4.8.6"
)
```

#### context definition
```scala
lazy val ctx = new SqliteJdbcContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.driverClassName=org.sqlite.JDBC
ctx.jdbcUrl=jdbc:sqlite:/path/to/db/file.db
```

### H2 (quill-jdbc)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "com.h2database" % "h2" % "1.4.199",
  "io.getquill" %% "quill-jdbc" % "4.8.6"
)
```

#### context definition
```scala
lazy val ctx = new H2JdbcContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.dataSourceClassName=org.h2.jdbcx.JdbcDataSource
ctx.dataSource.url=jdbc:h2:mem:yourdbname
ctx.dataSource.user=sa
```

### SQL Server (quill-jdbc)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "com.microsoft.sqlserver" % "mssql-jdbc" % "7.4.1.jre8",
  "io.getquill" %% "quill-jdbc" % "4.8.6"
)
```

#### context definition

```scala
lazy val ctx = new SqlServerJdbcContext(SnakeCase, "ctx")
```

### Oracle (quill-jdbc)

Quill supports Oracle version 12c and up although due to licensing restrictions, version 18c XE is used for testing.

Note that the latest Oracle JDBC drivers are not publicly available. In order to get them,
you will need to connect to Oracle's private maven repository as instructed [here](https://docs.oracle.com/middleware/1213/core/MAVEN/config_maven_repo.htm#MAVEN9012).
Unfortunately, this procedure currently does not work for SBT. There are various workarounds
available for this situation [here](https://stackoverflow.com/questions/1074869/find-oracle-jdbc-driver-in-maven-repository?rq=1).

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "com.oracle.jdbc" % "ojdbc8" % "18.3.0.0.0",
  "io.getquill" %% "quill-jdbc" % "4.8.6"
)
```

#### context definition

```scala
lazy val ctx = new OracleJdbcContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.dataSourceClassName=com.microsoft.sqlserver.jdbc.SQLServerDataSource
ctx.dataSource.user=user
ctx.dataSource.password=YourStrongPassword
ctx.dataSource.databaseName=database
ctx.dataSource.portNumber=1433
ctx.dataSource.serverName=host
```

## ZIO (quill-jdbc-zio)

Quill context that executes JDBC queries inside of ZIO. Unlike most other contexts
that require passing in a `java.sql.DataSource` when the context is created, this context's
run methods return a ZIO that has a DataSource resource dependency.
Naturally, this should be provided later on in your application
(see `ZioJdbc` for helper methods that assist in doing this).

Since resource dependency is `DataSource` the result of a `run` call is `ZIO[DataSource, SQLException, T]`.
This means that if you have a `DataSource` object, you can just provide it!

```scala
def ds: DataSource = _
run(people).provide(Has(ds))
```

> Since most quill-zio methods return `ZIO[DataSource, SQLException, T]`
> the type `QIO[T]` i.e. Quill-IO has been defined as an alias.
>
> For underlying-contexts (see below) that depend on `Connection`,
> the alias `QCIO[T]` (i.e. Quill-Connection-IO) has been defined
> for `ZIO[Connection, SQLException, T]`.

Since in most JDBC use-cases, a connection-pool datasource (e.g. Hikari) is used,
constructor-methods `fromPrefix`, `fromConfig`, `fromJdbcConfig` are available on
`DataSourceLayer` to construct instances of a `ZLayer[Any, SQLException, DataSource]`
which can be easily used to provide a DataSource dependency.
You can use them like this:
```scala

val zioDs = DataSourceLayer.fromPrefix("testPostgresDB")
MyZioContext.run(query[Person]).provideCustomLayer(zioDS)
```

If in some rare cases, you wish to provide a `java.sql.Connection` to a `run` method directly, you can delegate
to the underlying-context. This is a more low-level context whose `run` methods have a `Connection` resource.
Here is an example of how this can be done.

```scala
def conn: Connection = _ // If you are starting with a connection object

// Import encoders/decoders of the underlying context. Do not import quote/run/prepare methods to avoid conflicts.

MyZioContext.underlying.run(people).provide(Has(conn))
```

If you are working with an underlying-context and want to provide a DataSource instead of a connection,
you can use the `onDataSource` method. Note however that this is *only* needed when working with an underlying-context.
When working with a normal context, `onDataSource` is not available or necessary
(since for a  normal contexts `R` will be `DataSource`).

```scala
val ds: DataSource = _

// Import encoders/decoders of the underlying context. Do not import quote/run/prepare methods to avoid conflicts.

MyZioContext.underlying.run(people).onDataSource.provide(Has(ds))
```

> Also note that if you are using a Plain Scala app however, you will need to manually run it i.e. using zio.Runtime
> ```scala
> Runtime.default.unsafeRun(MyZioContext.run(query[Person]).provideLayer(zioDS))
> ```

#### DAO helper

One additional useful pattern is to use `import io.getquill.context.qzio.ImplicitSyntax.Implicit` to provide
an implicit DataSource to one or multiple `run(qry)` calls in a context. This is very useful when creating
DAO patterns that will reuse a DataSource many times:

```scala
case class MyQueryService(ds: DataSource with Closeable) { // I.e. our DAO

 implicit val env = Implicit(Has(ds)) // This will be looked up in each `.implicitDS` call

 val joes = Ctx.run(query[Person].filter(p => p.name == "Joe")).implicitDS
 val jills = Ctx.run(query[Person].filter(p => p.name == "Jill")).implicitDS
 val alexes = Ctx.run(query[Person].filter(p => p.name == "Alex")).implicitDS
}
```

More examples of a Quill-JDBC-ZIO app [quill-jdbc-zio/src/test/scala/io/getquill/examples](https://github.com/getquill/quill/tree/master/quill-jdbc-zio/src/test/scala/io/getquill/examples).

#### streaming

The `ZioJdbcContext` can stream using zio.ZStream:

```
ctx.stream(query[Person])             // returns: ZStream[Connection, Throwable, Person]
  .run(Sink.collectAll).map(_.toList) // returns: ZIO[Connection, Throwable, List[T]]
```

#### transactions

The `ZioJdbcContext`s provide support for transactions without needing thread-local storage or similar
because they propagate the resource dependency in the ZIO effect itself (i.e. the `Connection` in `Zio[Connection, _, _]`).
As with the other contexts, if an exception is thrown anywhere inside a task or sub-task within a `transaction` block, the entire block
will be rolled back by the database.

Basic syntax:
```
val trans =
  ctx.transaction {
    for {
      _ <- ctx.run(query[Person].delete)
      _ <- ctx.run(query[Person].insertValue(Person("Joe", 123)))
      p <- ctx.run(query[Person])
    } yield p
  } //returns: ZIO[Connection, Throwable, List[Person]]

val result = Runtime.default.unsafeRun(trans.onDataSource.provide(ds)) //returns: List[Person]
```

#### json

The Zio Quill Postgres supports JSON encoding/decoding via the zio-json library. Just wrap your object in a `JsonValue` instance
and then define encoders/decoders via zio-json's `JsonEncoder`/`JsonDecoder` modules.

```scala

case class Person(name: String, age: Int)
case class MyTable(name: String, value: JsonValue[Person])

val joe = Person("Joe", 123)
val joeRow = MyTable("SomeJoe", JsonValue(joe))

// Declare an encoder/decoder for `Person` via zio-json
implicit val personEncoder: JsonEncoder[Person] = DeriveJsonEncoder.gen[Person]
implicit val personDecoder: JsonDecoder[Person] = DeriveJsonDecoder.gen[Person]

val myApp: ZIO[Any, SQLException, List[MyTable]] =
  for {
    // You can then insert the value:
    _ <- context.run(query[MyTable].insertValue(lift(joeRow)))
    // As well read it:
    value <- context.run(query[MyTable])
  } yield (value)
```

You can also encode/decode objects that have the type `zio.json.ast.Json` directly.
```scala

case class MyTable(name: String, value: JsonValue[Json])

// i.e. {name:"Joe", age:123}
val jsonJoe = Json.Obj(Chunk("name" -> Json.Str("Joe"), "age" -> Json.Num(123)))
val joeRow = MyTable("SomeJoe", JsonValue(jsonJoe))

testContext.run(jsonAstQuery.insertValue(lift(joeRow)))
```

### MySQL (quill-jdbc-zio)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "mysql" % "mysql-connector-java" % "8.0.17",
  "io.getquill" %% "quill-jdbc-zio" % "4.8.6"
)
```

#### context definition

```scala
val ctx = new MysqlZioJdbcContext(SnakeCase)
// Also can be static:
object MyContext extends MysqlZioJdbcContext(SnakeCase)
```

#### application.properties
```
ctx.dataSourceClassName=com.mysql.cj.jdbc.MysqlDataSource
ctx.dataSource.url=jdbc:mysql://host/database
ctx.dataSource.user=root
ctx.dataSource.password=root
ctx.dataSource.cachePrepStmts=true
ctx.dataSource.prepStmtCacheSize=250
ctx.dataSource.prepStmtCacheSqlLimit=2048
ctx.connectionTimeout=30000
```

### Postgres (quill-jdbc-zio)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "org.postgresql" % "postgresql" % "42.2.8",
  "io.getquill" %% "quill-jdbc-zio" % "4.8.6"
)
```

#### context definition

```scala
val ctx = new PostgresZioJdbcContext(SnakeCase)
// Also can be static:
object MyContext extends PostgresZioJdbcContext(SnakeCase)
```

#### application.properties
```
ctx.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource
ctx.dataSource.user=root
ctx.dataSource.password=root
ctx.dataSource.databaseName=database
ctx.dataSource.portNumber=5432
ctx.dataSource.serverName=host
ctx.connectionTimeout=30000
```

### Sqlite (quill-jdbc-zio)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "org.xerial" % "sqlite-jdbc" % "3.28.0",
  "io.getquill" %% "quill-jdbc-zio" % "4.8.6"
)
```

#### context definition
```scala
val ctx = new SqlitezioJdbcContext(SnakeCase)
// Also can be static:
object MyContext extends SqlitezioJdbcContext(SnakeCase)
```

#### application.properties
```
ctx.driverClassName=org.sqlite.JDBC
ctx.jdbcUrl=jdbc:sqlite:/path/to/db/file.db
```

### H2 (quill-jdbc-zio)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "com.h2database" % "h2" % "1.4.199",
  "io.getquill" %% "quill-jdbc-zio" % "4.8.6"
)
```

#### context definition

```scala
val ctx = new H2ZioJdbcContext(SnakeCase)
// Also can be static:
object MyContext extends H2ZioJdbcContext(SnakeCase)
```

#### application.properties
```
ctx.dataSourceClassName=org.h2.jdbcx.JdbcDataSource
ctx.dataSource.url=jdbc:h2:mem:yourdbname
ctx.dataSource.user=sa
```

### SQL Server (quill-jdbc-zio)

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "com.microsoft.sqlserver" % "mssql-jdbc" % "7.4.1.jre8",
  "io.getquill" %% "quill-jdbc-zio" % "4.8.6"
)
```

#### context definition
```scala
val ctx = new SqlServerZioJdbcContext(SnakeCase)
// Also can be static:
object MyContext extends SqlServerZioJdbcContext(SnakeCase)
```

#### application.properties
```
ctx.dataSourceClassName=com.microsoft.sqlserver.jdbc.SQLServerDataSource
ctx.dataSource.user=user
ctx.dataSource.password=YourStrongPassword
ctx.dataSource.databaseName=database
ctx.dataSource.portNumber=1433
ctx.dataSource.serverName=host
```

### Oracle (quill-jdbc-zio)

Quill supports Oracle version 12c and up although due to licensing restrictions, version 18c XE is used for testing.

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "com.oracle.jdbc" % "ojdbc8" % "18.3.0.0.0",
  "io.getquill" %% "quill-jdbc-zio" % "4.8.6"
)
```

#### context definition
```scala
val ctx = new OracleZioJdbcContext(SnakeCase)
// Also can be static:
object MyContext extends OracleZioJdbcContext(SnakeCase)
```

#### application.properties
```
ctx.dataSourceClassName=oracle.jdbc.xa.client.OracleXADataSource
ctx.dataSource.databaseName=xe
ctx.dataSource.user=database
ctx.dataSource.password=YourStrongPassword
ctx.dataSource.driverType=thin
ctx.dataSource.portNumber=1521
ctx.dataSource.serverName=host
```

## quill-doobie

Quill 3.16.5 and above supports Doobie starting 1.0.0-RC1. You can use quill quotes to construct `ConnectionIO` programs.
Quill provides statement construction and type mapping, and doobie takes care of statement execution.

> Note that if you are migrating from the original `doobie-quill` integration (e.g. [here](https://search.maven.org/search?q=a:doobie-quill_2.12))
just add the below dependency and replace the `doobie.quill` package with `io.getquill.doobie`.
(If you are using the package provided by kubukoz (i.e. [here](https://github.com/polyvariant/doobie-quill)), then replace `org.polyvariant` with `io.getquill.doobie`.)

In order to use this feature, add the following dependency.
```
libraryDependencies += "io.getquill" %% "quill-doobie" % "4.6.1-SNAPSHOT"
```

The examples below require the following imports.

```

```

We can now construct a `DoobieContext` for our back-end database and import its members, as we would with a traditional Quill context. The options are `H2`, `MySQL`, `Oracle`, `Postgres`, `SQLite`, and `SQLServer`.

```
val dc = new DoobieContext.Postgres(Literal) // Literal naming scheme

```

> Instead of using Quill's `sql"MyUDF(${something})"` interpolator, use `qsql"MyUDF(${something})"` since we have excluded it.

We will be using the `country` table from our test database, so we need a data type of that name, with fields whose names and types line up with the table definition.

```scala
case class Country(code: String, name: String, population: Int)
```

We're now ready to construct doobie programs using Quill quotes. Note the return types from `run`, which are normal doobie types. You can freely mix Quill quotes into existing doobie programs.

#### running and streaming

```scala
val q1 = quote { query[Country].filter(_.code == "GBR") }

// Select all at once
run(q1)

// Stream in chunks of 16
stream(q1, 16)
```

#### actions

A simple update.

```scala
val u1 = quote { query[Country].filter(_.name like "U%").update(_.name -> "foo") }

// Update yielding count of affected rows
run(u1)
```

A batch update.

```scala
val u2 = quote {
  liftQuery(List("U%", "A%")).foreach { pat =>
    query[Country].filter(_.name like pat).update(_.name -> "foo")
  }
}

// Update yielding list of counts of affected rows
run(u2)
```

Now we will look at batch updates with generated keys. For this we will create a new table.

```sql
CREATE TABLE Foo (
  id    SERIAL,
  value VARCHAR(42)
)
```

And a related data type.

```scala
case class Foo(id: Int, value: String)
```

We can now write an update returning generated keys.

```scala
val u3 = quote {
  query[Foo].insert(lift(Foo(0, "Joe"))).returning(_.id)
}

// Update yielding a single id
run(u3)
```

And a batch update returning generated keys.

```scala
val u4 = quote {
  liftQuery(List(Foo(0, "Joe"), Foo(0, "Bob"))).foreach { a =>
    query[Foo].insert(a).returning(_.id)
  }
}

// Update yielding a list of ids
run(u4)
```

## Finagle Contexts

Support for the Twitter Finagle library is available with MySQL and Postgres databases.

### quill-finagle-mysql

#### transactions

The finagle context provides transaction support through a `Local` value. See twitter util's [scaladoc](https://github.com/twitter/util/blob/ee8d3140ba0ecc16b54591bd9d8961c11b999c0d/util-core/src/main/scala/com/twitter/util/Local.scala#L96) for more details.

```
ctx.transaction {
  ctx.run(query[Person].delete)
  // other transactional code
}
```

#### streaming

The finagle context allows streaming a query response, returning an `AsyncStream` value.

```
ctx.stream(query[Person]) // returns: Future[AsyncStream[Person]]
  .flatMap(_.toSeq())
```

The body of `transaction` can contain calls to other methods and multiple `run` calls since the transaction is automatically propagated through the `Local` value.

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-finagle-mysql" % "4.8.6"
)
```

#### context definition
```scala
lazy val ctx = new FinagleMysqlContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.dest=localhost:3306
ctx.user=root
ctx.password=root
ctx.database=database
ctx.pool.watermark.low=0
ctx.pool.watermark.high=10
ctx.pool.idleTime=5 # seconds
ctx.pool.bufferSize=0
ctx.pool.maxWaiters=2147483647
```

### quill-finagle-postgres

#### transactions

The finagle context provides transaction support through a `Local` value. See twitter util's [scaladoc](https://github.com/twitter/util/blob/ee8d3140ba0ecc16b54591bd9d8961c11b999c0d/util-core/src/main/scala/com/twitter/util/Local.scala#L96) for more details.

```
ctx.transaction {
  ctx.run(query[Person].delete)
  // other transactional code
}
```

The body of `transaction` can contain calls to other methods and multiple `run` calls since the transaction is automatically propagated through the `Local` value.

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-finagle-postgres" % "4.8.6"
)
```

#### context definition
```scala
lazy val ctx = new FinaglePostgresContext(SnakeCase, "ctx")
```

#### application.properties
```
ctx.host=localhost:3306
ctx.user=root
ctx.password=root
ctx.database=database
ctx.useSsl=false
ctx.hostConnectionLimit=1
ctx.numRetries=4
ctx.binaryResults=false
ctx.binaryParams=false
```

## quill-cassandra

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-cassandra" % "4.8.6"
)
```

#### synchronous context
```scala
lazy val ctx = new CassandraSyncContext(SnakeCase, "ctx")
```

#### asynchronous context
```scala
lazy val ctx = new CassandraAsyncContext(SnakeCase, "ctx")
```

The configurations are set using runtime reflection on the [`Cluster.builder`](https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Cluster.Builder.html) instance. It is possible to set nested structures like `queryOptions.consistencyLevel`, use enum values like `LOCAL_QUORUM`, and set multiple parameters like in `credentials`.

#### application.properties
```
ctx.keyspace=quill_test
ctx.preparedStatementCacheSize=1000
ctx.session.contactPoint=127.0.0.1
ctx.session.withPort=9042
ctx.session.queryOptions.consistencyLevel=LOCAL_QUORUM
ctx.session.withoutMetrics=true
ctx.session.withoutJMXReporting=false
ctx.session.credentials.0=root
ctx.session.credentials.1=pass
ctx.session.maxSchemaAgreementWaitSeconds=1
ctx.session.addressTranslator=com.datastax.driver.core.policies.IdentityTranslator
```

## quill-cassandra-zio

Quill context that executes Cassandra queries inside of ZIO. Unlike most other contexts
that require passing in a Data Source, this context takes in a `CassandraZioSession`
as a resource dependency which can be provided later (see the `CassandraZioSession` object for helper methods
that assist in doing this).

The resource dependency itself is just a `CassandraZioSession` hence `run(qry)` and other methods in this context will return
`ZIO[CassandraZioSession, Throwable, T]`.  The type `CIO[T]` i.e. Cassandra-IO is an alias for this.
Providing a `CassandraZioSession` dependency is now very simple:

```scala
val session: CassandraZioSession = _
run(people)
  .provide(Has(session))
```

Various methods in the `io.getquill.CassandraZioSession` can assist in simplifying it's creation, for example, you can
provide a `Config` object instead of a `CassandraZioSession` like this:

```scala
 val zioSessionLayer: ZLayer[Any, Throwable, CassandraZioSession] =
   CassandraZioSession.fromPrefix("testStreamDB")
run(query[Person])
  .provideCustomLayer(zioSessionLayer)
```
> (Note that the resulting CassandraZioSession has a closing bracket)

If you are using a Plain Scala app, you will need to manually run it e.g. using zio.Runtime
```scala
 Runtime.default.unsafeRun(MyZioContext.run(query[Person]).provideCustomLayer(zioSessionLayer))
```

#### DAO helper

One additional useful pattern is to use `import io.getquill.context.qzio.ImplicitSyntax.Implicit` to provide
an implicit CassandraZioSession to one or multiple `run(qry)` calls in a context. This is very useful when creating
DAO patterns that will reuse a CassandraZioSession many times:

```scala
case class MyQueryService(cs: CassandraZioSession) {

  implicit val env = Implicit(Has(cs))

  def joes = Ctx.run { query[Person].filter(p => p.name == "Joe") }.implicitly
  def jills = Ctx.run { query[Person].filter(p => p.name == "Jill") }.implicitly
  def alexes = Ctx.run { query[Person].filter(p => p.name == "Alex") }.implicitly
}
```

More examples of a Quill-Cassandra-ZIO app [quill-cassandra-zio/src/test/scala/io/getquill/context/cassandra/zio/examples](https://github.com/getquill/quill/tree/master/quill-cassandra-zio/src/test/scala/io/getquill/context/cassandra/zio/examples).

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-cassandra-zio" % "4.8.6"
)
```

## quill-cassandra-pekko

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-cassandra-pekko" % "4.8.6"
)
```

See [Pekko Cassandra](https://pekko.apache.org/docs/pekko-connectors/current/cassandra.html#apache-cassandra) documentation page for more information.

#### context

```scala

val system: ActorSystem = ???
val pekkoSessionSettings = CassandraSessionSettings("quill-test.pekko.cassandra")
val pekkoSession: CassandraSession = CassandraSessionRegistry.get(system).sessionFor(pekkoSessionSettings)

lazy val ctx = new CassandraAlpakkaContext(SnakeCase, pekkoSession, preparedStatementCacheSize = 100)
```

#### application.properties
```
// pekko cassandra session with keyspace
quill-test.pekko.cassandra: ${pekko.cassandra} { // inheritance of pekko.cassandra session configuration
  // custom datastax driver setup
  datastax-java-driver-config = quill-test-datastax-java-driver
}

quill-test-datastax-java-driver {
  basic {
    // keyspace at datastax driver setup, as there is not different option now
    session-keyspace = "quill_test"
  }
}
```

## OrientDB Contexts

#### sbt dependencies

```
libraryDependencies ++= Seq(
  "io.getquill" %% "quill-orientdb" % "4.8.6"
)
```

#### synchronous context
```scala
lazy val ctx = new OrientDBSyncContext(SnakeCase, "ctx")
```

The configurations are set using [`OPartitionedDatabasePool`](https://orientdb.com/javadoc/latest/com/orientechnologies/orient/core/db/OPartitionedDatabasePool.html) which creates a pool of DB connections from which an instance of connection can be acquired. It is possible to set DB credentials using the parameter called `username` and `password`.

#### application.properties
```
ctx.dbUrl=remote:127.0.0.1:2424/GratefulDeadConcerts
ctx.username=root
ctx.password=root
```

---

## Extending Quill

## Infix

Infix is a very flexible mechanism to use non-supported features without having to use plain queries in the target language. It allows the insertion of arbitrary strings within quotations.

For instance, quill doesn't support the `FOR UPDATE` SQL feature. It can still be used through infix and implicit classes:

```scala
implicit final class ForUpdate[T](private val q: Query[T]) extends AnyVal {
  def forUpdate = quote(sql"$q FOR UPDATE".as[Query[T]])
}

val a = quote {
  query[Person].filter(p => p.age < 18).forUpdate
}

ctx.run(a)
// SELECT p.name, p.age FROM person p WHERE p.age < 18 FOR UPDATE
```

The `forUpdate` quotation can be reused for multiple queries.

Queries that contain `infix` will generally not be flattened since it is not assumed that the contents
of the infix are a pure function.
> Since SQL is typically less performant when there are many nested queries,
be careful with the use of `infix` in queries that have multiple `map`+`filter` clauses.

```scala
final case class Data(id: Int)
final case class DataAndRandom(id: Int, value: Int)

// This should be alright:
val q = quote {
  query[Data].map(e => DataAndRandom(e.id, sql"RAND()".as[Int])).filter(r => r.value <= 10)
}
run(q)
// SELECT e.id, e.value FROM (SELECT RAND() AS value, e.id AS id FROM Data e) AS e WHERE e.value <= 10

// This might not be:
val q = quote {
  query[Data]
    .map(e => DataAndRandom(e.id, sql"SOME_UDF(${e.id})".as[Int]))
    .filter(r => r.value <= 10)
    .map(e => DataAndRandom(e.id, sql"SOME_OTHER_UDF(${e.value})".as[Int]))
    .filter(r => r.value <= 100)
}
// Produces too many layers of nesting!
run(q)
// SELECT e.id, e.value FROM (
//   SELECT SOME_OTHER_UDF(e.value) AS value, e.id AS id FROM (
//     SELECT SOME_UDF(e.id) AS value, e.id AS id FROM Data e
//   ) AS e WHERE e.value <= 10
// ) AS e WHERE e.value <= 100
```

If you are sure that the content of your infix is a pure function, you can use the `pure` method
in order to indicate to Quill that the infix clause can be copied in the query. This gives Quill much
more leeway to flatten your query, possibly improving performance.

```scala
val q = quote {
  query[Data]
    .map(e => DataAndRandom(e.id, sql"SOME_UDF(${e.id})".pure.as[Int]))
    .filter(r => r.value <= 10)
    .map(e => DataAndRandom(e.id, sql"SOME_OTHER_UDF(${e.value})".pure.as[Int]))
    .filter(r => r.value <= 100)
}
// Copying SOME_UDF and SOME_OTHER_UDF allows the query to be completely flattened.
run(q)
// SELECT e.id, SOME_OTHER_UDF(SOME_UDF(e.id)) FROM Data e
// WHERE SOME_UDF(e.id) <= 10 AND SOME_OTHER_UDF(SOME_UDF(e.id)) <= 100
```

### Infixes With Conditions

#### Summary
Use `sql"...".asCondition` to express an infix that represents a conditional expression.

#### Explanation

When synthesizing queries for databases which do not have proper boolean-type support (e.g. SQL Server,
Oracle etc...) boolean infix clauses inside projections must become values.
Typically this requires a `CASE WHERE ... END`.

Take the following example:
```scala
final case class Node(name: String, isUp: Boolean, uptime:Long)
final case class Status(name: String, allowed: Boolean)
val allowedStatus:Boolean = getState

quote {
  query[Node].map(n => Status(n.name, n.isUp == lift(allowedStatus)))
}
run(q)
// This is invalid in most databases:
//   SELECT n.name, n.isUp = ?, uptime FROM Node n
// It will be converted to this:
//   SELECT n.name, CASE WHEN (n.isUp = ?) THEN 1 ELSE 0, uptime FROM Node n
```
However, in certain cases, infix clauses that express conditionals should actually represent
boolean expressions for example:
```scala
final case class Node(name: String, isUp: Boolean)
val maxUptime:Boolean = getState

quote {
  query[Node].filter(n => sql"${n.uptime} > ${lift(maxUptime)}".as[Boolean])
}
run(q)
// Should be this:
//  SELECT n.name, n.isUp, n.uptime WHERE n.uptime > ?
// However since sql"...".as[Boolean] is treated as a Boolean Value (as opposed to an expression) it will be converted to this:
//  SELECT n.name, n.isUp, n.uptime WHERE 1 == n.uptime > ?
```

In order to avoid this problem, use sql"...".asCondition so that Quill understands that the boolean is an expression:
```scala
quote {
  query[Node].filter(n => sql"${n.uptime} > ${lift(maxUptime)}".asCondition)
}
run(q) // SELECT n.name, n.isUp, n.uptime WHERE n.uptime > ?
```

### Dynamic infix

Infix supports runtime string values through the `#$` prefix. Example:

```scala
def test(functionName: String) =
  ctx.run(query[Person].map(p => sql"#$functionName(${p.name})".as[Int]))
```

### Implicit Extensions

You can use implicit extensions in quill in several ways.
> NOTE. In ProtoQuill extensions must be written using the Scala 3 `extension` syntax and implicit class extensions are not supported. Please see [Extensions in ProtoQuill/Scala3](#extensions-in-protoquillscala3) below for more info.

##### Standard quoted extension:
```scala
implicit final class Ext(private val q: Query[Person]) extends AnyVal {
  def olderThan(age: Int) = quote {
    query[Person].filter(p => p.age > lift(age))
  }
}
run(query[Person].olderThan(44))
// SELECT p.name, p.age FROM Person p WHERE p.age > ?
```

##### Higher-order quoted extension:
```scala
implicit final class Ext(private val q: Query[Person]) extends AnyVal {
  def olderThan = quote {
    (age: Int) =>
      query[Person].filter(p => p.age > lift(age))
  }
}
run(query[Person].olderThan(44))
// SELECT p.name, p.age FROM Person p WHERE p.age > 44

run(query[Person].olderThan(lift(44)))
// SELECT p.name, p.age FROM Person p WHERE p.age > ?
```
The advantage of this approach is that you can choose to either lift or use a constant.

##### Scalar quoted extension:

Just as `Query` can be extended, scalar values can be similarly extended.
```scala
implicit final class Ext(private val i: Int) extends AnyVal {
  def between = quote {
    (a: Int, b:Int) =>
      i > a && i < b
  }
}
run(query[Person].filter(p => p.age.between(33, 44)))
// SELECT p.name, p.age FROM Person p WHERE p.age > 33 AND p.age < 44
```

##### Extensions in ProtoQuill/Scala3:
In ProtoQuill, the implicit class pattern for extensions is not supported. Please switch to using Scala 3 extension methods combined with inline definitions to achieve the same functionality.

```scala 3
extension (q: Query[Person]) {
  inline def olderThan(inline age: Int) = quote {
    query[Person].filter(p => p.age > lift(age))
  }
}
run(query[Person].olderThan(44))
// SELECT p.name, p.age FROM Person p WHERE p.age > ?
```

### Raw SQL queries

You can also use infix to port raw SQL queries to Quill and map it to regular Scala tuples.

```scala
val rawQuery = quote {
  (id: Int) => sql"""SELECT id, name FROM my_entity WHERE id = $id""".as[Query[(Int, String)]]
}
ctx.run(rawQuery(1))
//SELECT x._1, x._2 FROM (SELECT id AS "_1", name AS "_2" FROM my_entity WHERE id = 1) x
```

Note that in this case the result query is nested.
It's required since Quill is not aware of a query tree and cannot safely unnest it.
This is different from the example above because infix starts with the query `sql"$q...` where its tree is already compiled

### Database functions

A custom database function can also be used through infix:

```scala
val myFunction = quote {
  (i: Int) => sql"MY_FUNCTION($i)".as[Int]
}

val q = quote {
  query[Person].map(p => myFunction(p.age))
}

ctx.run(q)
// SELECT MY_FUNCTION(p.age) FROM Person p
```

### Comparison operators

The easiest way to use comparison operators with dates is to import them from the `extras` module.
```scala
final case class Person(name: String, bornOn: java.util.Date)

val ctx = new SqlMirrorContext(PostgresDialect, Literal)

run(query[Person].filter(p => p.bornOn > lift(myDate)))
```
> Note that in ProtoQuill you should import `io.getquill.extras._` since they are now global.

#### Using Ordered

You can also define an implicit-class that converts your Date/Numeric type to a `Ordered[YourType]`
which will also give it `>`, `<` etc... comparison operators.

```scala
implicit final class LocalDateTimeOps(val value: MyCustomNumber) extends Ordered[MyCustomNumber] {
  def compare(that: MyCustomNumber): Int = value.compareTo(that)
}
```
Note that Quill will not actually use this `compare` method, that is strictly for your own
data needs. You can technically define it as `def compare(that: MyCustomNumber): Int = ???`
because Quill will never actually invoke this function. It uses the `LocalDateTimeOps`
merely as a sort of marker to know that the `>`, `<` operators etc... can be transpiled to SQL.

> Note that although in ProtoQuill implicit-class based approaches are generally not supported,
> this particular pattern will work as well.

#### Using Implicit Classes

Finally, can implement comparison operators (or any other kinds of operators) by defining
implicit conversion and using infix.

```scala
implicit final class DateQuotes(private val left: MyCustomDate) extends AnyVal {
  def >(right: MyCustomDate) = quote(sql"$left > $right".as[Boolean])
  def <(right: MyCustomDate) = quote(sql"$left < $right".as[Boolean])
}
```

### batch with infix

```scala
implicit final class OnDuplicateKeyIgnore[T](private val q: Insert[T]) extends AnyVal {
  def ignoreDuplicate = quote(sql"$q ON DUPLICATE KEY UPDATE id=id".as[Insert[T]])
}

ctx.run(
  liftQuery(List(
    Person(1, "Test1", 30),
    Person(2, "Test2", 31)
  )).foreach(row => query[Person].insertValue(row).ignoreDuplicate)
)
```

## Custom encoding

Quill uses `Encoder`s to encode query inputs and `Decoder`s to read values returned by queries. The library provides a few built-in encodings and two mechanisms to define custom encodings: mapped encoding and raw encoding.

### Custom Encoders

If the correspondent database type is already supported, use `encoder.contramap` and `decoder.map` to define encoders/decoders for
the custom type. In this example, `String` is already supported by Quill and the `UUID` encoding from/to `String` is defined through a custom encoding:
```scala

implicit val encodeUUID: Encoder[UUID] = 
  implicitly[Encoder[String]].contramap((id: UUID) => id.toString)
implicit val decodeUUID: Decoder[UUID] =
  implicitly[Decoder[String]].map((idStr: String) => UUID.fromString(idStr))
```

You can also MappedEncoding to define instances that convert to/from the target type.

```scala

// import io.getquill.MappedEncoding - (or import MappedEncoding directly)

implicit val encodeUUID = MappedEncoding[UUID, String](_.toString)
implicit val decodeUUID = MappedEncoding[String, UUID](UUID.fromString(_))
```
Note that can it be also used to provide mapping for element types of collection (SQL Arrays or Cassandra Collections).

### Raw Encoding

If the database type is not supported by Quill, it is possible to provide "raw" encoders and decoders:

```scala
trait UUIDEncodingExample {
  val jdbcContext: PostgresJdbcContext[Literal] // your context should go here

  implicit val uuidDecoder: Decoder[UUID] =
    decoder((index, row) =>
      UUID.fromString(row.getObject(index).toString)) // database-specific implementation

  implicit val uuidEncoder: Encoder[UUID] =
    encoder(java.sql.Types.OTHER, (index, value, row) =>
        row.setObject(index, value, java.sql.Types.OTHER)) // database-specific implementation

  // Only for postgres
  implicit def arrayUUIDEncoder[Col <: Seq[UUID]]: Encoder[Col] = arrayRawEncoder[UUID, Col]("uuid")
  implicit def arrayUUIDDecoder[Col <: Seq[UUID]](implicit bf: CBF[UUID, Col]): Decoder[Col] =
    arrayRawDecoder[UUID, Col]
}
```

## `AnyVal`

Quill automatically encodes `AnyVal`s (value classes):

```scala
final case class UserId(value: Int) extends AnyVal
final case class User(id: UserId, name: String)

val q = quote {
  for {
    u <- query[User] if u.id == lift(UserId(1))
  } yield u
}

ctx.run(q)
// SELECT u.id, u.name FROM User u WHERE (u.id = 1)
```

## Table/Column Customizations

The meta DSL allows the user to customize how Quill handles column/table naming and behavior.

### Changing Table and Column Names

You can change how Quill queries handle table and column names for a record case class.

```scala
def example = {
  implicit val personSchemaMeta = schemaMeta[Person]("people", _.id -> "person_id")

  ctx.run(query[Person])
  // SELECT x.person_id, x.name, x.age FROM people x
}
```

By default, quill expands `query[Person]` to `querySchema[Person]("Person")`. It's possible to customize this behavior using an implicit instance of `SchemaMeta`:

### Excluding Columns from Insert

You can exclude columns (e.g. Auto-Generated ones) from insertion in `q.insertValue(...)` by using an `InsertMeta`.

```scala
implicit val personInsertMeta = insertMeta[Person](_.id)

ctx.run(query[Person].insertValue(lift(Person(-1, "John", 22))))
// INSERT INTO Person (name,age) VALUES (?, ?)
```

Note that the parameter of `insertMeta` is called `exclude`, but it isn't possible to use named parameters for macro invocations.

### Excluding Columns from Update

You can exclude columns (e.g. Auto-Generated ones) from updates in `q.insertValue(...)` by using an `UpdateMeta`.

```scala
implicit val personUpdateMeta = updateMeta[Person](_.id)

ctx.run(query[Person].filter(_.id == 1).updateValue(lift(Person(1, "John", 22))))
// UPDATE Person SET name = ?, age = ? WHERE id = 1
```

Note that the parameter of `updateMeta` is called `exclude`, but it isn't possible to use named parameters for macro invocations.

## Mapped Records

The QueryMeta customizes the expansion of query types and extraction of the final value. For instance, it's possible to use this feature to normalize values before reading them from the database:

```scala
implicit val personQueryMeta =
  queryMeta(
    (q: Query[Person]) =>
      q.map(p => (p.id, sql"CONVERT(${p.name} USING utf8)".as[String], p.age))
  ) {
    case (id, name, age) =>
      Person(id, name, age)
  }
```

The query meta definition is open and allows the user to even join values from other tables before reading the final value. This kind of usage is not encouraged.

---

## Getting Started with Quill

> ### [Scastie](https://scastie.scala-lang.org/) is a great tool to try out Quill without having to prepare a local environment. It works with [mirror contexts](contexts.md#mirror-context), see [this](https://scastie.scala-lang.org/QwOewNEiR3mFlKIM7v900A) snippet as an example.

Quill has integrations with many libraries. If you are using a regular RDBMS e.g. PostgreSQL
and want to use Quill to query it with an asynchronous, non-blocking, reactive application, the easiest way to get
started is by using ZIO.

A simple ZIO + Quill application looks like this:
```scala
case class Person(name: String, age: Int)

class DataService(quill: Quill.Postgres[SnakeCase]) {

  def getPeople: ZIO[Any, SQLException, List[Person]] = run(query[Person])
}
object DataService {
  def getPeople: ZIO[DataService, SQLException, List[Person]] =
    ZIO.serviceWithZIO[DataService](_.getPeople)

  val live = ZLayer.fromFunction(new DataService(_))
}
object Main extends ZIOAppDefault {
  override def run = {
    DataService.getPeople
      .provide(
        DataService.live,
        Quill.Postgres.fromNamingStrategy(SnakeCase),
        Quill.DataSource.fromPrefix("myDatabaseConfig")
      )
      .debug("Results")
      .exitCode
  }
}
```

Add the following to build.sbt:

```scala
libraryDependencies ++= Seq(
  "io.getquill"          %% "quill-jdbc-zio" % "4.8.6",
  "org.postgresql"       %  "postgresql"     % "42.3.1"
)
```

You can find this code (with some more examples) complete with a docker-provided Postgres database [here](https://github.com/deusaquilus/zio-quill-gettingstarted).
A variety of other examples using Quill with ZIO are available in the [examples](https://github.com/zio/zio-quill/tree/master/quill-jdbc-zio/src/test/scala/io/getquill/examples) folder.

## Choosing a Module

Choose the quill module that works for you!

* If you are starting from scratch with a regular RDBMS try using the `quill-jdbc-zio` module as shown above.
* If you are developing a legacy Java project and don't want/need reactive, use `quill-jdbc`.
* If you are using Cassandra, Spark, or OrientDB, try the corresponding modules for each of them.

---

## Introduction to ZIO Quill

Quill provides a Quoted Domain Specific Language ([QDSL](https://homepages.inf.ed.ac.uk/wadler/papers/qdsl/qdsl.pdf)) to express queries in Scala and execute them in a target language. 

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-quill/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/io.getquill/quill-core_2.12.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/io/getquill/quill-core_2.12/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/io.getquill/quill-core_2.12.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/io/getquill/quill-core_2.12/) [![javadoc](https://javadoc.io/badge2/io.getquill/zio-quill-docs_2.12/javadoc.svg)](https://javadoc.io/doc/io.getquill/zio-quill-docs_2.12) [![ZIO Quill](https://img.shields.io/github/stars/zio/zio-quill?style=social)](https://github.com/zio/zio-quill)

## Introduction

The library's core is designed to support multiple target languages, currently featuring specializations for Structured Query Language ([SQL](https://en.wikipedia.org/wiki/SQL)) and Cassandra Query Language ([CQL](https://cassandra.apache.org/doc/latest/cql/)).

1. **Boilerplate-free mapping**: The database schema is mapped using simple case classes.
2. **Quoted DSL**: Queries are defined inside a `quote` block. Quill parses each quoted block of code (quotation) at compile time and translates them to an internal Abstract Syntax Tree (AST)
3. **Compile-time query generation**: The `ctx.run` call reads the quotation's AST and translates it to the target language at compile time, emitting the query string as a compilation message. As the query string is known at compile time, the runtime overhead is very low and similar to using the database driver directly.
4. **Compile-time query validation**: If configured, the query is verified against the database at compile time and the compilation fails if it is not valid. The query validation **does not** alter the database state.

> ### Scala 3 Support
> [ProtoQuill](https://github.com/zio/zio-protoquill) provides Scala 3 support for Quill rebuilding on top of new metaprogramming capabilities from the ground > up! It is published to maven-central as the `quill-<module>_3` line of artifacts.

> ### Doobie Support
> See [here](contexts.md#quill-doobie) for Doobie integration instructions.

## Example

![example](https://raw.githubusercontent.com/getquill/quill/master/example.gif)

Note: The GIF example uses Eclipse, which shows compilation messages to the user.

---

## Logging

## Logging to a File at Compile-Time

To write compile-time queries to a log, use the `-Dquill.log.file=queries.sql` and specify
the file to be written (e.g. `queries.sql`). The path is based on the build root (i.e. the current-working-directory of the Java build).

When using SBT, this parameter can be set either in your SBT_OPTS, the project-specific .sbtopts file or directly passed to the SBT command.
In IntelliJ this can be set under settings -> sbt -> VM Parameters.

(Also make sure that `use for: "Builds"` is selected otherwise IntelliJ will not use SBT for the build in the first place.)

![Screenshot from 2022-04-14 23-28-47](https://user-images.githubusercontent.com/1369480/163513653-b5266cd6-1bff-4792-b0d2-936d24b7e0f1.png)

Also note that the `-Dquill.macro.log.pretty=true` parameter works together with `-Dquill.log.file` and will output pretty-printed
queries to the specified file.

For a file that looks like this:
```
// Example.scala
package io.getquill

object Example {
  case class Person(id: Int, name: String, age: Int)
  case class Address(owner:Int, street: String)
  val ctx = new SqlMirrorContext(PostgresDialect, Literal)

  val people = run(query[Person])
  val addresses = run(query[Person])
}
```

The following log will be produced:
```sql

-- file: /home/me/quill-example/src/main/scala/io/getquill/Example.scala:9:19
-- time: 2022-04-14T23:18:19.533

 SELECT
   x.id,
   x.name,
   x.age
 FROM
   Person x

;

-- file: /home/me/quill-example/src/main/scala/io/getquill/Example.scala:10:22
-- time: 2022-04-14T23:18:19.9

 SELECT
   x.id,
   x.name,
   x.age
 FROM
   Person x

;
```

## Disable Compile-Time Console Logging

To disable the console logging of queries during compilation use `quill.macro.log` option:
```
sbt -Dquill.macro.log=false
```
## Runtime

Quill uses SLF4J for logging. Each context logs queries which are currently executed.
It also logs the list of parameters that are bound into a prepared statement if any.
To enable that use `quill.binds.log` option:
```
java -Dquill.binds.log=true -jar myapp.jar
```
The option `quill.query.tooLong` restricts the maximum length of logged SQL statements:
```
java -Dquill.binds.log=true -Dquill.query.tooLong=10000 -jar myapp.jar
```

## Pretty Printing

Quill can pretty print compile-time produced queries by leveraging a great library
produced by [@vertical-blank](https://github.com/vertical-blank) which is compatible
with both Scala and ScalaJS. To enable this feature use the `quill.macro.log.pretty` option:
```
sbt -Dquill.macro.log.pretty=true
```

Before:
```
[info] /home/me/project/src/main/scala/io/getquill/MySqlTestPerson.scala:20:18: SELECT p.id, p.name, p.age, a.ownerFk, a.street, a.state, a.zip FROM Person p INNER JOIN Address a ON a.ownerFk = p.id
```

After:
```
[info] /home/me/project/src/main/scala/io/getquill/MySqlTestPerson.scala:20:18:
[info]   | SELECT
[info]   |   p.id,
[info]   |   p.name,
[info]   |   p.age,
[info]   |   a.ownerFk,
[info]   |   a.street,
[info]   |   a.state,
[info]   |   a.zip
[info]   | FROM
[info]   |   Person p
[info]   |   INNER JOIN Address a ON a.ownerFk = p.id
```

---

## Quill vs. Cassandra

This document compares Quill to the [Datastax Java](https://github.com/datastax/java-driver) driver and the [Phantom](https://web.archive.org/web/20141229171448/http://websudos.github.io:80/phantom) library. This is an incomplete comparison, additions and corrections are welcome.

All examples have been properly tested, and they should work out of the box.

## Index ##

* [Prerequisites](#prerequisites)
* [Abstraction level](#abstraction-level)
* [Simple queries](#simple-queries)
* [Composable queries](#composable-queries)
* [Extensibility](#extensibility)
* [Custom data types](#custom-data-types)
* [Non-blocking IO](#non-blocking-io)
* [Other considerations](#other-considerations)

## Prerequisites ##

The keyspace and column family needed for all examples are defined in this CQL script:

```
CREATE KEYSPACE IF NOT EXISTS db
WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

CREATE TABLE IF NOT EXISTS db.weather_station (
  country TEXT,
  city TEXT,
  station_id TEXT,
  entry INT,
  value INT,
  PRIMARY KEY (country, city, station_id)
);
```

In order to make all Quill examples to work you will need to create this config file, named `application.conf` in your resources folder:

```
db.keyspace=db
db.preparedStatementCacheSize=100
db.session.contactPoint=127.0.0.1
db.session.queryOptions.consistencyLevel=ONE
```

Add your project's `build.sbt` dependencies as described in [Cassandra Contexts](https://github.com/getquill/quill#cassandra-contexts)

## Abstraction level ##

The Datastax Java driver provides simple abstractions that let you either write your queries as plain strings or use a declarative Query Builder. It also provides a higher level [Object Mapper](https://github.com/datastax/java-driver/tree/2.1/manual/object_mapper). For this comparison we will only use the Query Builder.

Although both Quill and Phantom represent column family rows as flat immutable structures (case classes without nested data) and provide a type-safe composable query DSL, they work at a different abstraction level. 

Phantom provides an embedded DSL that help you write CQL queries in a type-safe manner. Quill is referred as a Language Integrated Query library to match the available publications on the subject. The paper ["Language-integrated query using comprehension syntax: state of the art, open problems, and work in progress"](http://research.microsoft.com/en-us/events/dcp2014/cheney.pdf) has an overview with some of the available implementations of language integrated queries.

## Simple queries ##

This section compares how the different libraries let the user query a column family to obtain some elements.

**Java Driver (v3.0.0)**
```

object JavaDriver extends App {

  val nrOfCacheEntries: Int = 100

  val cluster: Cluster = Cluster.builder().addContactPoints("localhost").build()

  val session: Session = cluster.newSession()

  val cache: LoadingCache[String, PreparedStatement] =
    CacheBuilder.newBuilder().
      maximumSize(nrOfCacheEntries).
      build(
        new CacheLoader[String, PreparedStatement]() {
          def load(key: String): PreparedStatement = session.prepare(key.toString)
        }
      )

  case class WeatherStation(country: String, city: String, stationId: String, entry: Int, value: Int)

  object WeatherStation {

    def getAllByCountry(cache: LoadingCache[String, PreparedStatement], session: Session)(country: String): List[WeatherStation] = {
      val query: Statement =
        QueryBuilder.select().
          all().
          from("db", "weather_station").
          where(QueryBuilder.eq("country", QueryBuilder.bindMarker()))

      session.execute(cache.get(query.toString).bind(country)).all().asScala.map(
        row => WeatherStation(row.getString("country"), row.getString("city"), row.getString("station_id"), row.getInt("entry"), row.getInt("value"))
      ).to[List]
    }
  }

  val getAllByCountry: String => List[WeatherStation] = WeatherStation.getAllByCountry(cache, session)_

  getAllByCountry("UK")

  session.close()
  cluster.close()
}
```

The Java driver requires explicit handling of a `PreparedStatement`s cache to avoid preparing the same statement more that once, that could affect performance.

**Phantom (v1.22.0)**
```

object Phantom extends App {

  case class WeatherStation(country: String, city: String, stationId: String, entry: Int, value: Int)

  abstract class WeatherStationCF(override val tableName: String) extends CassandraTable[WeatherStationCF, WeatherStation] with RootConnector {

    object country extends StringColumn(this) with PartitionKey[String]
    object city extends StringColumn(this) with PrimaryKey[String]
    object stationId extends StringColumn(this) with PrimaryKey[String] {
      override lazy val name: String = "station_id"
    }
    object entry extends IntColumn(this) with PrimaryKey[Int]
    object value extends IntColumn(this)

    override def fromRow(r: Row): WeatherStation =
      WeatherStation(country(r), city(r), stationId(r), entry(r), value(r))
  }

  abstract class WeatherStationQueries extends WeatherStationCF("weather_station") {

    def getAllByCountry(country: String): Future[List[WeatherStation]] =
      select.where(_.country eqs country).fetch()
  }

  class DB(ks: KeySpaceDef) extends DatabaseImpl(ks) {

    object stations extends WeatherStationQueries with connector.Connector
  }

  val db = new DB(ContactPoint.local.keySpace("db"))

  val result = db.stations.getAllByCountry("UK")

  result.onComplete(_ => db.shutdown())
}
```

Phantom requires mapping classes to lift the database model to DSL types. The query definition also requires special equality operators.

**Quill**
```scala

object Quill extends App {

  val db = new CassandraAsyncContext(SnakeCase, "db")

  case class WeatherStation(country: String, city: String, stationId: String, entry: Int, value: Int)

  object WeatherStation {

    val getAllByCountry = quote {
      (country: String) =>
        query[WeatherStation].filter(_.country == country)
    }
  }

  val result = db.run(WeatherStation.getAllByCountry(lift("UK")))

  result.onComplete(_ => db.close())
}
```

During the compilation of this example, as the quotation is known statically, Quill will emit the CQL string as an info message, e.g. `SELECT country, city, station_id, entry, value FROM weather_station WHERE country = ?`.

## Composable queries ##

This section compares how the different libraries let the user compose queries.

**Java Driver (v3.0.0)** 

The Query Builder allows the user to partially construct queries and add filters later:

```

object JavaDriver extends App {

  val nrOfCacheEntries: Int = 100

  val cluster: Cluster = Cluster.builder().addContactPoints("localhost").build()

  val session: Session = cluster.newSession()

  val cache: LoadingCache[String, PreparedStatement] =
    CacheBuilder.newBuilder().
      maximumSize(nrOfCacheEntries).
      build(
        new CacheLoader[String, PreparedStatement]() {
          def load(key: String): PreparedStatement = session.prepare(key.toString)
        }
      )

  case class WeatherStation(country: String, city: String, stationId: String, entry: Int, value: Int)

  object WeatherStation {

    def getAll: Select =
      QueryBuilder.select().
        all().
        from("db", "weather_station")

    def getAllByCountry(cache: LoadingCache[String, PreparedStatement], session: Session)(country: String): List[WeatherStation] = {
      val query: Statement =
        getAll.
          where(QueryBuilder.eq("country", QueryBuilder.bindMarker()))

      session.execute(cache.get(query.toString).bind(country)).all().asScala.map(
        row => WeatherStation(row.getString("country"), row.getString("city"), row.getString("station_id"), row.getInt("entry"), row.getInt("value"))
      ).to[List]
    }
  }

  val getAllByCountry: String => List[WeatherStation] = WeatherStation.getAllByCountry(cache, session)_

  getAllByCountry("UK")

  session.close()
  cluster.close()
}
```

The DSL has limited composition compatibility.

**Phantom (v1.22.0)**
```

object Phantom extends App {

  case class WeatherStation(country: String, city: String, stationId: String, entry: Int, value: Int)

  abstract class WeatherStationCF(override val tableName: String) extends CassandraTable[WeatherStationCF, WeatherStation] with RootConnector {

    object country extends StringColumn(this) with PartitionKey[String]
    object city extends StringColumn(this) with PrimaryKey[String]
    object stationId extends StringColumn(this) with PrimaryKey[String]
    object entry extends IntColumn(this) with PrimaryKey[Int]
    object value extends IntColumn(this)

    override def fromRow(r: Row): WeatherStation =
      WeatherStation(country(r), city(r), stationId(r), entry(r), value(r))
  }

  abstract class WeatherStationQueries extends WeatherStationCF("weather_station") {

    def getAllByCountry(country: String): Future[List[WeatherStation]] =
      findAllByCountry(country).fetch()

    def getAllByCountryAndCity(country: String, city: String): Future[List[WeatherStation]] =
      findAllByCountryAndCity(country, city).fetch()

    def getAllByCountryCityAndId(country: String, city: String, stationId: String): Future[List[WeatherStation]] =
      findAllByCountryCityAndId(country, city, stationId).fetch()

    private def findAllByCountry(country: String) =
      select.where(_.country eqs country)

    private def findAllByCountryAndCity(country: String, city: String) =
      findAllByCountry(country).and(_.city eqs city)

    private def findAllByCountryCityAndId(country: String, city: String, stationId: String) =
      findAllByCountryAndCity(country, city).and(_.stationId eqs stationId)
  }

  class DB(ks: KeySpaceDef) extends DatabaseImpl(ks) {

    object stations extends WeatherStationQueries with connector.Connector
  }

  val db = new DB(ContactPoint.local.keySpace("db"))

  val result = db.stations.getAllByCountryCityAndId("UK", "London", "SW27")

  result.onComplete(_ => db.shutdown())
}
```

Phantom allows the user certain level of composability, but it gets a bit verbose due to the nature of the DSL. 

**Quill**
```scala

object Quill extends App {

  val db = new CassandraAsyncContext(SnakeCase, "db")

  case class WeatherStation(country: String, city: String, stationId: String, entry: Int, value: Int)

  object WeatherStation {

    val getAllByCountry = quote {
      (country: String) =>
        query[WeatherStation].filter(_.country == country)
    }

    val getAllByCountryAndCity = quote {
      (country: String, city: String) =>
        getAllByCountry(country).filter(_.city == city)
    }

    val getAllByCountryCityAndId = quote {
      (country: String, city: String, stationId: String) =>
        getAllByCountryAndCity(country, city).filter(_.stationId == stationId)
    }
  }

  val result = db.run(WeatherStation.getAllByCountryCityAndId("UK", "London", "SW2"))

  result.onComplete(_ => db.close())
}
```

Quill offers more advanced composability, but CQL being a much simpler query language than SQL can't benefit much from it. During the compilation of this example, as the quotation is known statically, Quill will emit the CQL string as an info message, e.g. `SELECT country, city, station_id, entry, value FROM weather_station WHERE country = ? AND city = ? AND station_id = ?`.

## Extensibility ##

This section explores the extensibility capabilities of each library .

**Java Driver (v3.0.0)**

There is no much offered by the driver to extend the Query Builder, e.g. add a missing CQL feature.

**Phantom (v1.22.0)**

You could extend Phantom by extending the DSL to add new features, although it might not be a straightforward process.

**Quill**

Quill provides an easy mechanism to add non-supported features through [infix](https://github.com/getquill/quill#infix). In fact, most of the [CQL specific features](https://github.com/getquill/quill/blob/master/quill-cassandra/src/main/scala/io/getquill/sources/cassandra/ops/package.scala) are added using infix.

## Custom data types ##

This section would allow us to compare how the different libraries let us read custom data types in a seamless way.

**Java Driver (v3.0.0)**
```

object JavaDriver extends App {

  case class Country(code: String)

  object Country {

    class Codec(charset: Charset) extends TypeCodec[Country](DataType.text(), classOf[Country]) {

      override def serialize(country: Country, protocolVersion: ProtocolVersion): ByteBuffer =
        if (country == null) null else ByteBuffer.wrap(country.code.getBytes(charset))

      override def parse(value: String): Country =
        if (value == null || value.isEmpty || value.equalsIgnoreCase("NULL"))
          null
        else if (!ParseUtils.isQuoted(value))
          throw new InvalidTypeException("text or varchar values must be enclosed by single quotes")
        else Country(ParseUtils.unquote(value))

      override def format(country: Country): String =
        if (country == null) "NULL"
        else ParseUtils.quote(country.code)

      override def deserialize(bytes: ByteBuffer, protocolVersion: ProtocolVersion): Country =
        if (bytes == null) null
        else if (bytes.remaining == 0) Country("")
        else Country(new String(Bytes.getArray(bytes), charset))
    }
  }

  val nrOfCacheEntries: Int = 100

  val cluster: Cluster = Cluster.builder().addContactPoints("localhost").build()

  cluster.getConfiguration.getCodecRegistry.register(new Country.Codec(Charset.forName("UTF-8")))

  val session: Session = cluster.newSession()

  val cache: LoadingCache[String, PreparedStatement] =
    CacheBuilder.newBuilder().
      maximumSize(nrOfCacheEntries).
      build(
        new CacheLoader[String, PreparedStatement]() {
          def load(key: String): PreparedStatement = session.prepare(key.toString)
        }
      )

  case class WeatherStation(country: Country, city: String, stationId: String, entry: Int, value: Int)

  object WeatherStation {

    def getAllByCountry(cache: LoadingCache[String, PreparedStatement], session: Session)(country: Country): List[WeatherStation] = {
      val query: Statement =
        QueryBuilder.select().
          all().
          from("db", "weather_station").
          where(QueryBuilder.eq("country", QueryBuilder.bindMarker()))

      session.execute(cache.get(query.toString).bind(country)).all().asScala.map(
        row => WeatherStation(row.get("country", classOf[Country]), row.getString("city"), row.getString("station_id"), row.getInt("entry"), row.getInt("value"))
      ).to[List]
    }
  }

  val getAllByCountry: Country => List[WeatherStation] = WeatherStation.getAllByCountry(cache, session)_

  getAllByCountry(Country("UK"))

  session.close()
  cluster.close()
}
```

It is necessary to create a new `TypeCodec` and register it in the `CodecRegistry`.

**Phantom (v1.22.0)**
```

object Phantom extends App {

  case class Country(code: String) extends AnyVal

  object Country {

    implicit object CountryIsPrimitive extends Primitive[Country] {

      override type PrimitiveType = Country

      override def fromRow(column: String, row: Row): Try[Country] =
        nullCheck(column, row) {
          r => Country(r.getString(column))
        }

      override val cassandraType: String = CQLSyntax.Types.Text

      override def fromString(code: String): Country = Country(code)

      override def asCql(country: Country): String = CQLQuery.empty.singleQuote(country.code)

      override val clz: Class[CountryIsPrimitive.PrimitiveType] = classOf[Country]
    }

    type Column[Owner <: CassandraTable[Owner, Record], Record] = PrimitiveColumn[Owner, Record, Country]
  }

  case class WeatherStation(country: Country, city: String, stationId: String, entry: Int, value: Int)

  case class People(name: String, age: Int)

  abstract class WeatherStationCF(override val tableName: String) extends CassandraTable[WeatherStationCF, WeatherStation] with RootConnector {

    object country extends Country.Column(this) with PartitionKey[Country]
    object city extends StringColumn(this) with PrimaryKey[String]
    object stationId extends StringColumn(this) with PrimaryKey[String] {
      override lazy val name: String = "station_id"
    }
    object entry extends IntColumn(this) with PrimaryKey[Int]
    object value extends IntColumn(this)

    override def fromRow(r: Row): WeatherStation =
      WeatherStation(country(r), city(r), stationId(r), entry(r), value(r))
  }

  abstract class WeatherStationQueries extends WeatherStationCF("weather_station") {

    def getAllByCountry(country: Country): Future[List[WeatherStation]] =
      select.where(_.country eqs country).fetch()
  }

  class DB(ks: KeySpaceDef) extends DatabaseImpl(ks) {

    object stations extends WeatherStationQueries with connector.Connector
  }

  val db = new DB(ContactPoint.local.keySpace("db"))

  val result = db.stations.getAllByCountry(Country("UK"))

  result.onComplete(_ => db.shutdown())
}
```

It is necessary to define a new `Column` type to be used when defining the data model.

**Quill**
```scala

object Quill extends App {

  val db = new CassandraAsyncContext(SnakeCase, "db")

  case class Country(code: String) extends AnyVal

  object Country {

    implicit val decode: MappedEncoding[String, Country] = MappedEncoding[String, Country](Country(_))
    implicit val encode: MappedEncoding[Country, String] = MappedEncoding[Country, String](_.code)
  }
  case class WeatherStation(country: Country, city: String, stationId: String, entry: Int, value: Int)

  object WeatherStation {

    val getAllByCountry = quote {
      (country: Country) =>
        query[WeatherStation].filter(_.country == country)
    }
  }

  val result = db.run(WeatherStation.getAllByCountry(lift(Country("UK"))))

  result.onComplete(_ => db.close())
}
```

Quill only requires definition of implicit encodings from/to `String`.

## Non-blocking IO ##

This section compares the different options the libraries offer to do non-blocking IO.

**Java Driver (v3.0.0)**

The Datastax driver allows the user to execute queries [asynchronously](https://github.com/datastax/java-driver/tree/2.1/manual/async), returning `ListenableFuture`s.
   
**Phantom (v1.22.0)**

Phantom is asynchronous by default and all operations return `Future`s. It also allows users to process the data coming from Cassandra in a streaming fashion using [`play-iteratees`](https://www.playframework.com/documentation/2.4.x/Iteratees) or [`play-streams-experimental`](https://www.playframework.com/documentation/2.4.x/ReactiveStreamsIntegration), that make it possible to integrate with other software that support [reactive-streams](https://github.com/reactive-streams/reactive-streams-jvm).

**Quill (v0.4.0)**

Quill provides blocking, asynchronous and streaming sources for Cassandra. The asynchronous source returns `Future`s on all operations. The streaming source uses [Monix](https://github.com/monixio/monix) to return an `Observable`. Monix is a reactive library compatible with the [reactive-streams](https://github.com/reactive-streams/reactive-streams-jvm) protocol.

## Other considerations ##

There other aspects the user might want to take into account like 3rd party dependencies. As both Phantom and Quill depend on the Datastax Java Driver, we are going to pay attention to which additional dependencies each of them add.

Phantom is composed by several modules, each of them with their 3rd party dependencies. Overall it adds more 3rd party dependencies than Quill and it has dependencies on libraries like shapeless, play-iteratees, play-streams-experimental or akka-actor. Quill, on the other hand, only adds dependencies on monix and scalamacros resetallattrs.

---

## Quill vs. Slick

This document compares Quill to the [Typesafe Slick](https://scala-slick.org) library. This is an incomplete comparison, additions and corrections are welcome.

## Abstraction level ##

Quill and Slick have similar abstraction levels. They represent database rows as flat immutable structures (case classes without nested data) and provide a type-safe composable query DSL.

Slick's documentation refers to this abstraction level as a [new paradigm called functional-relational mapping (FRM)](https://github.com/slick/slick/blob/3b3bd36c93c6d9c63b0471ff4d8409f913954b2b/slick/src/sphinx/introduction.rst#functional-relational-mapping). In fact, the approach is not new and was introduced in the late '90s by ["Kleisli􏰂, a Functional Query System"](https://www.comp.nus.edu.sg/~wongls/psZ/wls-jfp98-3.ps). It was also used by the [Links programming language](https://web.archive.org/web/20120127183323/https://groups.inf.ed.ac.uk/links/papers/links-fmco06.pdf), and later on was popularized by [Microsoft LINQ](https://msdn.microsoft.com/en-us/library/bb425822.aspx) in a less functional manner.

Quill is referred as a Language Integrated Query library to match the available publications on the subject. The paper ["Language-integrated query using comprehension syntax: state of the art, open problems, and work in progress"](http://research.microsoft.com/en-us/events/dcp2014/cheney.pdf) has an overview with some of the available implementations of language integrated queries.

## QDSL versus EDSL ##

Quill's DSL is a macro-based quotation mechanism, allowing usage of Scala types and operators directly. Please refer to the paper ["Everything old is new again: Quoted Domain Specific Languages"](https://homepages.inf.ed.ac.uk/wadler/papers/qdsl/qdsl.pdf) for more details. On the other hand, Slick provides a DSL that requires lifting of types and operations to the DSL counterparts at runtime. Example:

**quill**
```scala
case class Person(name: String, age: Int)

val range = quote {
  (a: Int, b: Int) =>
    for {
      u <- query[Person] if (a <= u.age && u.age < b)
    } yield {
      u
    }
}
val ageFromName = quote {
  (s: String) =>
    for {
      u <- query[Person] if (s == u.name)
    } yield {
      u.age
    }
}
val q = quote {
  (s: String, t: String) =>
    for {
      a <- ageFromName(s)
      b <- ageFromName(t)
      r <- range(a, b)
    } yield {
      r
    }
}
```

**slick**
```scala
case class PersonRow(name: String, age: Int)

class Person(_tableTag: Tag) extends Table[PersonRow](_tableTag, "Person") {
  def * = (name, age) <> (PersonRow.tupled, PersonRow.unapply)
  def ? = (Rep.Some(name), Rep.Some(age)).shaped.<>({r=>import r._; _1.map(_=> PersonRow.tupled((_1.get, _2.get)))}, (_:Any) =>  throw new Exception("Inserting into ? projection not supported."))

  val name: Rep[String] = column[String]("name", O.Length(255,varying=true))
  val age: Rep[Int] = column[Int]("age")
}

lazy val Person = new TableQuery(tag => new Person(tag))

val range =
  (a: Rep[Int], b: Rep[Int]) =>
    for {
      u <- Person if (a <= u.age && u.age < b)
    } yield {
      u
    }
val ageFromName =
  (s: Rep[String]) =>
    for {
      u <- Person if (s === u.name)
    } yield {
      u.age
    }
val q = 
  (s: String, t: String) =>
    for {
      a <- ageFromName(s)
      b <- ageFromName(t)
      r <- range(a, b)
    } yield {
      r
    }
```

Slick requires explicit type definition to map the database model to lifted values, which can be automatically generated and maintained by the [`slick-codegen`](https://scala-slick.org/doc/3.1.0/code-generation.html) tool. The query definition also requires special equality operators and usage of `Rep` for composable queries.

## Compile-time versus Runtime ##

Quill's quoted DSL opens a new path to query generation. For the quotations that are known statically, the query normalization and translation to SQL happen at compile-time. The user receives feedback during compilation, knows the SQL string that will be used and if it will succeed when executed against the database.

The feedback cycle using Slick is typically longer. Some factors like normalization bugs and unsupported operations can make the query fail, but only at runtime it is possible to know whether they will affect the query or not.

## Non-blocking IO ##

Slick provides an asynchronous wrapper on top of jdbc's blocking interface, making it harder to scale applications using it. On the other hand, quill provides fully asynchronous non-blocking database access through quill-zio.

## Extensibility ##

It is common to have to write plain SQL statements when a feature is not supported by Slick. Quill's [`infix` mechanism](https://github.com/getquill/quill#infix) solves this problem by allowing the user to insert arbitrary SQL within quotations at any position.

## Normalization ##

Quill's normalization engine is based on the rules introduced by the paper ["A practical theory of language-integrated query"](https://www.infoq.com/presentations/theory-language-integrated-query). They ensure that, given some fulfilled requirements, the normalization will always succeed. Quill verifies these requirements at compile-time.

Unfortunately, the paper doesn't cover all SQL features supported by Quill. Some additional transformations were added to the normalization engine for this reason.

Slick's normalization is based on an multi-phase compilation engine. The code complexity is very high, probably due to the lack of principled normalization rules.

The last stable version (3.1) features a major rewrite of the query compiler. Before it, even simple compositions used to produce highly nested queries with bad performance characteristics when executed against MySQL.

The reader is invited to compare the libraries' normalization code:

https://github.com/getquill/quill/tree/master/quill-core/src/main/scala/io/getquill/norm
https://github.com/slick/slick/tree/master/slick/src/main/scala/slick/compiler

---

## Writing Queries

## Introduction

The QDSL allows the user to write plain Scala code, leveraging Scala's syntax and type system. Quotations are created using the `quote` method and can contain any excerpt of code that uses supported operations. To create quotations, first create a context instance. Please see the [context](contexts.md) section for more details on the different context available.

For this documentation, a special type of context that acts as a [mirror](contexts.md#mirror-context) is used:

```scala

val ctx = new SqlMirrorContext(MirrorSqlDialect, Literal)
```

The context instance provides all the types, methods, and encoders/decoders needed for quotations:

```scala

```

A quotation can be a simple value:

```scala
val pi = quote(3.14159)
```

And be used within another quotation:

```scala
final case class Circle(radius: Float)

val areas = quote {
  query[Circle].map(c => pi * c.radius * c.radius)
}
```

Quotations can also contain high-order functions and inline values:

```scala
val area = quote {
  (c: Circle) => {
    val r2 = c.radius * c.radius
    pi * r2
  }
}
```

```scala
val areas = quote {
  query[Circle].map(c => area(c))
}
```

Quill's normalization engine applies reduction steps before translating the quotation to the target language. The correspondent normalized quotation for both versions of the `areas` query is:

```scala
val areas = quote {
  query[Circle].map(c => 3.14159 * c.radius * c.radius)
}
```

Scala doesn't have support for high-order functions with type parameters. It's possible to use a method type parameter for this purpose:

```scala
def existsAny[T] = quote {
  (xs: Query[T]) => (p: T => Boolean) =>
    	xs.filter(p(_)).nonEmpty
}

val q = quote {
  query[Circle].filter { c1 =>
    existsAny(query[Circle])(c2 => c2.radius > c1.radius)
  }
}
```

You can also use implicit classes to extend things in quotations.

```scala
implicit final class Ext(private val q: Query[Person]) extends AnyVal {
  def olderThan(age: Int) = quote {
    query[Person].filter(p => p.age > lift(age))
  }
}

run(query[Person].olderThan(44))
```
(see [implicit-extensions](extending-quill.md#implicit-extensions) for additional information.)

## Compile-time quotations

Quotations are both compile-time and runtime values. Quill uses a type refinement to store the quotation's AST as an annotation available at compile-time and the `q.ast` method exposes the AST as runtime value.

It is important to avoid giving explicit types to quotations when possible. For instance, this quotation can't be read at compile-time as the type refinement is lost:

```scala
// Avoid type widening (Quoted[Query[Circle]]), or else the quotation will be dynamic.
val q: Quoted[Query[Circle]] = quote {
  query[Circle].filter(c => c.radius > 10)
}

ctx.run(q) // Dynamic query
```

Quill falls back to runtime normalization and query generation if the quotation's AST can't be read at compile-time. Please refer to [dynamic queries](#dynamic-queries) for more information.

#### Inline queries

Quoting is implicit when writing a query in a `run` statement.

```scala
ctx.run(query[Circle].map(_.radius))
// SELECT r.radius FROM Circle r
```

## Bindings

Quotations are designed to be self-contained, without references to runtime values outside their scope. There are two mechanisms to explicitly bind runtime values to a quotation execution.

### Lifted values

A runtime value can be lifted to a quotation through the method `lift`:

```scala
def biggerThan(i: Float) = quote {
  query[Circle].filter(r => r.radius > lift(i))
}
ctx.run(biggerThan(10)) // SELECT r.radius FROM Circle r WHERE r.radius > ?
```

Note that literal-constants do not need to be lifted, they can be used in queries directly. Literal constants are supported starting Scala 2.12.
```scala
final val minAge = 21  // This is the same as: final val minAge: 21 = 21
ctx.run(query[Person].filter(p => p.age > minAge)) // SELECT p.name, p.age FROM Person p WHERE p.name > 21
```

### Lifted queries

A `Iterable` instance can be lifted as a `Query`. There are two main usages for lifted queries:

#### contains

```scala
def find(radiusList: List[Float]) = quote {
  query[Circle].filter(r => liftQuery(radiusList).contains(r.radius))
}
ctx.run(find(List(1.1F, 1.2F)))
// SELECT r.radius FROM Circle r WHERE r.radius IN (?)
```

#### batch action
```scala
def insertValues(circles: List[Circle]) = quote {
  liftQuery(circles).foreach(c => query[Circle].insertValue(c))
}
ctx.run(insertValues(List(Circle(1.1F), Circle(1.2F))))
// INSERT INTO Circle (radius) VALUES (?)
```

## Schema

The database schema is represented by case classes. By default, quill uses the class and field names as the database identifiers:

```scala
final case class Circle(radius: Float)

val q = quote {
  query[Circle].filter(c => c.radius > 1)
}

ctx.run(q) // SELECT c.radius FROM Circle c WHERE c.radius > 1
```

### Schema customization

Alternatively, the identifiers can be customized:

```scala
val circles = quote {
  querySchema[Circle]("circle_table", _.radius -> "radius_column")
}

val q = quote {
  circles.filter(c => c.radius > 1)
}

ctx.run(q)
// SELECT c.radius_column FROM circle_table c WHERE c.radius_column > 1
```

If multiple tables require custom identifiers, it is good practice to define a `schema` object with all table queries to be reused across multiple queries:

```scala
final case class Circle(radius: Int)
final case class Rectangle(length: Int, width: Int)
object schema {
  val circles = quote {
    querySchema[Circle](
        "circle_table",
        _.radius -> "radius_column")
  }
  val rectangles = quote {
    querySchema[Rectangle](
        "rectangle_table",
        _.length -> "length_column",
        _.width -> "width_column")
  }
}
```

### Database-generated values

#### returningGenerated

Database generated values can be returned from an insert query by using `.returningGenerated`. These properties
will also be excluded from the insertion since they are database generated.

```scala
final case class Product(id: Int, description: String, sku: Long)

val q = quote {
  query[Product].insertValue(lift(Product(0, "My Product", 1011L))).returningGenerated(_.id)
}

val returnedIds = ctx.run(q) //: List[Int]
// INSERT INTO Product (description,sku) VALUES (?, ?) -- NOTE that 'id' is not being inserted.
```

Multiple properties can be returned in a Tuple or Case Class and all of them will be excluded from insertion.

> NOTE: Using multiple properties is currently supported by Postgres, Oracle and SQL Server

```scala
// Assuming sku is generated by the database.
val q = quote {
  query[Product].insertValue(lift(Product(0, "My Product", 1011L))).returningGenerated(r => (id, sku))
}

val returnedIds = ctx.run(q) //: List[(Int, Long)]
// INSERT INTO Product (description) VALUES (?) RETURNING id, sku -- NOTE that 'id' and 'sku' are not being inserted.
```

#### returning

In UPDATE and DELETE queries we frequently want to return the records that were modified/deleted.
The `returning` method is used for that.

> Note that most of these operations are only supported in Postgres and SQL Server

For example when we want to return information from records that are being updated:

```scala
val desc = "Update Product"
val sku = 2002L
val q = quote {
  query[Product].filter(p => p.id == 42).update(_.description -> lift(desc), _.sku -> lift(sku)).returning(r => (r.id, r.description))
}
val updated = ctx.run(q) //: (Int, String)
// Postgres
// UPDATE Product AS p SET description = ?, sku = ? WHERE p.id = 42 RETURNING p.id, p.description
// SQL Server
// UPDATE Product SET description = ?, sku = ? OUTPUT id, description WHERE id = 42
```
> When multiple records are updated using `update.returning` a warning will be issued and only the first result will be returned.
> Use [`returningMany`](writing-queries.md#returningmany) to return all the updated records in this case.

You can do the same thing with `updateValue`.

```scala
// (use an UpdateMeta to exclude generated id columns)
implicit val productUpdateMeta = updateMeta[Product](_.id)
val q = quote {
  query[Product].filter(p => p.id == 42).updateValue(lift(Product(42, "Updated Product", 2022L))).returning(r => (r.id, r.description))
}
val updated = ctx.run(q) //: (Int, String)
// Postgres
// UPDATE Product AS p SET description = ?, sku = ? WHERE p.id = 42 RETURNING p.id, p.description
// SQL Server
// UPDATE Product SET description = ?, sku = ? OUTPUT INSERTED.id, INSERTED.description WHERE id = 42
```

You can also return information that is being deleted in a DELETE query. Or even the entire deleted record!

```scala
val q = quote {
  query[Product].filter(p => p.id == 42).delete.returning(r => r)
}

val deleted = ctx.run(q) //: Product
// Postgres
// DELETE FROM Product AS p WHERE p.id = 42 RETURNING p.id, p.description, p.sku 
// SQL Server
// DELETE FROM Product OUTPUT DELETED.id, DELETED.description, DELETED.sku WHERE id = 42
```
> When multiple records are deleted using `delete.returning` a warning will be issued and only the first result will be returned.
> Use [`returningMany`](writing-queries.md#returningmany) to return all the deleted records in this case.

#### returningMany

Similar to insert/update.returning, the returningMany function can be used to return all the values that were
updated/deleted from a query. Not just one.

Return *all* the records that were updated.

```scala
val desc = "Update Product"
val sku = 2002L
val q = quote {
  query[Product].filter(p => p.id == 42).update(_.description -> lift(desc), _.sku -> lift(sku)).returning(r => (r.id, r.description))
}
val updated = ctx.run(q) //: List[(Int, String)]
// Postgres
// UPDATE Product AS p SET description = ?, sku = ? WHERE p.id = 42 RETURNING p.id, p.description
// SQL Server
// UPDATE Product SET description = ?, sku = ? OUTPUT id, description WHERE id = 42
```

Return *all* the records that were deleted.

```scala
val q = quote {
  query[Product].filter(p => p.id == 42).delete.returning(r => r)
}

val deleted = ctx.run(q) //: List[Product]
// Postgres
// DELETE FROM Product AS p WHERE p.id = 42 RETURNING p.id, p.description, p.sku 
// SQL Server
// DELETE FROM Product OUTPUT DELETED.id, DELETED.description, DELETED.sku WHERE id = 42
```

#### Postgres Customized returning

Returning values returned can be further customized in some databases.

In Postgres, the `returning` and `returningGenerated` methods also support arithmetic operations, SQL UDFs and
even entire queries for INSERT, UPDATE, and DELETE actions. These are inserted directly into the SQL `RETURNING` clause.

For example, assuming this basic query:
```scala
val q = quote {
  query[Product].filter(p => p.id == 42).update(_.description -> "My Product", _.sku -> 1011L)
}
```

Add 100 to the value of `id`:
```scala
ctx.run(q.returning(r => r.id + 100)) //: List[Int]
// UPDATE Product AS p SET description = 'My Product', sku = 1011L WHERE p.id = 42 RETURNING p.id + 100
```

Pass the value of `id` into a UDF:
```scala
val udf = quote { (i: Long) => sql"myUdf($i)".as[Int] }
ctx.run(q.returning(r => udf(r.id))) //: List[Int]
// UPDATE Product AS p SET description = 'My Product', sku = 1011L WHERE p.id = 42 RETURNING myUdf(p.id)
```

Use the return value of `sku` to issue a query:
```scala
final case class Supplier(id: Int, clientSku: Long)
ctx.run {
  q.returning(r => query[Supplier].filter(s => s.sku == r.sku).map(_.id).max)
} //: List[Option[Long]]
// UPDATE Product AS p SET description = 'My Product', sku = 1011L WHERE p.id = 42 RETURNING (SELECT MAX(s.id) FROM Supplier s WHERE s.sku = clientSku)
```

As is typically the case with Quill, you can use all of these features together.
```scala
ctx.run {
  q.returning(r =>
    (r.id + 100, udf(r.id), query[Supplier].filter(s => s.sku == r.sku).map(_.id).max)
  )
} // List[(Int, Int, Option[Long])]
// UPDATE Product AS p SET description = 'My Product', sku = 1011L WHERE p.id = 42
// RETURNING id + 100, myUdf(id), (SELECT MAX(s.id) FROM Supplier s WHERE s.sku = sku)
```

> NOTE: Queries used inside of return clauses can only return a single row per insert.
Otherwise, Postgres will throw:
`ERROR: more than one row returned by a subquery used as an expression`. This is why is it strongly
recommended that you use aggregators such as `max` or `min`inside of quill returning-clause queries.
In the case that this is impossible (e.g. when using Postgres booleans), you can use the `.value` method:
`q.returning(r => query[Supplier].filter(s => s.sku == r.sku).map(_.id).value)`.

#### insert.returning

In certain situations we also may want to return information from inserted records.

```scala
val q = quote {
  query[Product].insertValue(lift(Product(0, "My Product", 1011L))).returning(r => (r.id, r.description))
}

val returnedIds = ctx.run(q) //: List[(Int, String)]
// INSERT INTO Product (id, description, sku) VALUES (?, ?, ?) RETURNING id, description
```

Wait a second! Why did we just insert `id` into the database? That is because `returning` does not exclude values
from the insertion! We can fix this situation by manually specifying the columns to insert:

```scala
val q = quote {
  query[Product].insert(_.description -> "My Product", _.sku -> 1011L))).returning(r => (r.id, r.description))
}

val returnedIds = ctx.run(q) //: List[(Int, String)]
// INSERT INTO Product (description, sku) VALUES (?, ?) RETURNING id, description
```

We can also fix this situation by using an insert-meta.

```scala
implicit val productInsertMeta = insertMeta[Product](_.id)
val q = quote {
  query[Product].insertValue(lift(Product(0L, "My Product", 1011L))).returning(r => (r.id, r.description))
}

val returnedIds = ctx.run(q) //: List[(Int, String)]
// INSERT INTO Product (description, sku) VALUES (?, ?) RETURNING id, description
```

### Embedded case classes

Quill supports nested embedded case classes.
> In previous iterations of Quill would need to extend the `Embedded` trait but this is no longer necessary.

```scala
final case class Contact(phone: String, address: String) /* The embedded class */
final case class Person(id: Int, name: String, contact: Contact)

ctx.run(query[Person])
// SELECT x.id, x.name, x.phone, x.address FROM Person x
```

Note that default naming behavior uses the name of the nested case class properties. It's possible to override this default behavior using a custom `schema`:

```scala
final case class Contact(phone: String, address: String) extends Embedded
final case class Person(id: Int, name: String, homeContact: Contact, workContact: Option[Contact])

val q = quote {
  querySchema[Person](
    "Person",
    _.homeContact.phone          -> "homePhone",
    _.homeContact.address        -> "homeAddress",
    _.workContact.map(_.phone)   -> "workPhone",
    _.workContact.map(_.address) -> "workAddress"
  )
}

ctx.run(q)
// SELECT x.id, x.name, x.homePhone, x.homeAddress, x.workPhone, x.workAddress FROM Person x
```

## Queries

The overall abstraction of quill queries uses database tables as if they were in-memory collections. Scala for-comprehensions provide syntactic sugar to deal with these kinds of monadic operations:

```scala
final case class Person(id: Int, name: String, age: Int)
final case class Contact(personId: Int, phone: String)

val q = quote {
  for {
    p <- query[Person] if(p.id == 999)
    c <- query[Contact] if(c.personId == p.id)
  } yield {
    (p.name, c.phone)
  }
}

ctx.run(q)
// SELECT p.name, c.phone FROM Person p, Contact c WHERE (p.id = 999) AND (c.personId = p.id)
```

Quill normalizes the quotation and translates the monadic joins to applicative joins, generating a database-friendly query that avoids nested queries.

Any of the following features can be used together with the others and/or within a for-comprehension:

### filter
```scala
val q = quote {
  query[Person].filter(p => p.age > 18)
}

ctx.run(q)
// SELECT p.id, p.name, p.age FROM Person p WHERE p.age > 18
```

### map
```scala
val q = quote {
  query[Person].map(p => p.name)
}

ctx.run(q)
// SELECT p.name FROM Person p
```

### flatMap
```scala
val q = quote {
  query[Person].filter(p => p.age > 18).flatMap(p => query[Contact].filter(c => c.personId == p.id))
}

ctx.run(q)
// SELECT c.personId, c.phone FROM Person p, Contact c WHERE (p.age > 18) AND (c.personId = p.id)
```

### sortBy
```scala
val q1 = quote {
  query[Person].sortBy(p => p.age)
}

ctx.run(q1)
// SELECT p.id, p.name, p.age FROM Person p ORDER BY p.age ASC NULLS FIRST

val q2 = quote {
  query[Person].sortBy(p => p.age)(Ord.descNullsLast)
}

ctx.run(q2)
// SELECT p.id, p.name, p.age FROM Person p ORDER BY p.age DESC NULLS LAST

val q3 = quote {
  query[Person].sortBy(p => (p.name, p.age))(Ord(Ord.asc, Ord.desc))
}

ctx.run(q3)
// SELECT p.id, p.name, p.age FROM Person p ORDER BY p.name ASC, p.age DESC
```

### aggregation

You can use aggregators inside of map-clauses. Multiple aggregators can be used as needed.
Available aggregators are `max`, `min`, `count`, `avg` and `sum`.
```scala
val q = quote {
  query[Person].map(p => (min(p.age), max(p.age)))
}
// SELECT MIN(p.age), MAX(p.age) FROM Person p
```

### groupByMap

The `groupByMap` method is the preferred way to do grouping in Quill. It provides a simple aggregation-syntax similar to SQL.
Available aggregators are `max`, `min`, `count`, `avg` and `sum`.

```scala
val q = quote {
  query[Person].groupByMap(p => p.name)(p => (p.name, max(p.age)))
}
ctx.run(q)
// SELECT p.name, MAX(p.age) FROM Person p GROUP BY p.name
```

You can use as many aggregators as needed and group by multiple fields (using a Tuple).

```scala
val q = quote {
  query[Person].groupByMap(p => (p.name, p.otherField))(p => (p.name, p.otherField, max(p.age)))
}
ctx.run(q)
// SELECT p.name, p.otherField, MAX(p.age) FROM Person p GROUP BY p.name, p.otherField
```

Writing a custom aggregator using `infix` with the `groupByMax` syntax is also very simple.
For example, in Postgres the `STRING_AGG` function is used to concatenate all the encountered strings.
```scala
val stringAgg = quote {
  (str: String, separator: String) => sql"STRING_AGG($str, $separator)".pure.as[String]
}
val q = quote {
  query[Person].groupByMap(p => p.age)(p => (p.age, stringAgg(p.name, ";")))
}
run(q)
// SELECT p.age, STRING_AGG(p.name, ';') FROM Person p GROUP BY p.age
```

You can also map to a case class instead of a tuple. This will give you a `Query[YourCaseClass]` that you can further compose.
```scala
final case class NameAge(name: String, age: Int)
// Will return Query[NameAge]
val q = quote {
  query[Person].groupByMap(p => p.name)(p => NameAge(p.name, max(p.age)))
}
ctx.run(q)
// SELECT p.name, MAX(p.age) FROM Person p GROUP BY p.name
```

> Note that it is a requirement in SQL for every column in the selection (without an aggregator) to be in the `GROUP BY` clause.
> If it is not, an exception will be thrown by the database. Quill does not (yet!) protect the user in this situation.
> ```scala
> run( query[Person].groupByMap(p => p.name)(p => (p.name, p.otherField, max(p.age))) )
> // > SELECT p.name, p.otherField, MAX(p.age) FROM Person p GROUP BY p.name
> // ERROR: column "person.otherField" must appear in the GROUP BY clause or be used in an aggregate function
> ```

### groupBy

Quill also provides a way to do `groupBy/map` in a more scala-idiomatic way. In this case (below), the `groupBy`
produces a `Query[(Int,Query[Person])]` where the inner Query can be mapped to an expression with an aggregator
(as would be the Scala `List[Person]` in `Map[Int,List[Person]]` resulting from a `(people:List[Person]).groupBy(_.name)`.

```scala
val q = quote {
  query[Person].groupBy(p => p.age).map {
    case (age, people) =>
      (age, people.size)
  }
}

ctx.run(q)
// SELECT p.age, COUNT(*) FROM Person p GROUP BY p.age
```

### drop/take

```scala
val q = quote {
  query[Person].drop(2).take(1)
}

ctx.run(q)
// SELECT x.id, x.name, x.age FROM Person x LIMIT 1 OFFSET 2
```

### concatMap (i.e. `UNNEST`)
```scala
// similar to `flatMap` but for transformations that return a traversable instead of `Query`

val q = quote {
  query[Person].concatMap(p => p.name.split(" "))
}

ctx.run(q)
// SELECT UNNEST(SPLIT(p.name, " ")) FROM Person p
```

### union
```scala
val q = quote {
  query[Person].filter(p => p.age > 18).union(query[Person].filter(p => p.age > 60))
}

ctx.run(q)
// SELECT x.id, x.name, x.age FROM (SELECT id, name, age FROM Person p WHERE p.age > 18
// UNION SELECT id, name, age FROM Person p1 WHERE p1.age > 60) x
```

### unionAll/++
```scala
val q = quote {
  query[Person].filter(p => p.age > 18).unionAll(query[Person].filter(p => p.age > 60))
}

ctx.run(q)
// SELECT x.id, x.name, x.age FROM (SELECT id, name, age FROM Person p WHERE p.age > 18
// UNION ALL SELECT id, name, age FROM Person p1 WHERE p1.age > 60) x

val q2 = quote {
  query[Person].filter(p => p.age > 18) ++ query[Person].filter(p => p.age > 60)
}

ctx.run(q2)
// SELECT x.id, x.name, x.age FROM (SELECT id, name, age FROM Person p WHERE p.age > 18
// UNION ALL SELECT id, name, age FROM Person p1 WHERE p1.age > 60) x
```

### aggregation
```scala
val r = quote {
  query[Person].map(p => p.age)
}

ctx.run(r.min) // SELECT MIN(p.age) FROM Person p
ctx.run(r.max) // SELECT MAX(p.age) FROM Person p
ctx.run(r.avg) // SELECT AVG(p.age) FROM Person p
ctx.run(r.sum) // SELECT SUM(p.age) FROM Person p
ctx.run(r.size) // SELECT COUNT(p.age) FROM Person p
```

### isEmpty/nonEmpty
```scala
val q = quote {
  query[Person].filter{ p1 =>
    query[Person].filter(p2 => p2.id != p1.id && p2.age == p1.age).isEmpty
  }
}

ctx.run(q)
// SELECT p1.id, p1.name, p1.age FROM Person p1 WHERE
// NOT EXISTS (SELECT * FROM Person p2 WHERE (p2.id <> p1.id) AND (p2.age = p1.age))

val q2 = quote {
  query[Person].filter{ p1 =>
    query[Person].filter(p2 => p2.id != p1.id && p2.age == p1.age).nonEmpty
  }
}

ctx.run(q2)
// SELECT p1.id, p1.name, p1.age FROM Person p1 WHERE
// EXISTS (SELECT * FROM Person p2 WHERE (p2.id <> p1.id) AND (p2.age = p1.age))
```

### contains
```scala
val q = quote {
  query[Person].filter(p => liftQuery(Set(1, 2)).contains(p.id))
}

ctx.run(q)
// SELECT p.id, p.name, p.age FROM Person p WHERE p.id IN (?, ?)

val q1 = quote { (ids: Query[Int]) =>
  query[Person].filter(p => ids.contains(p.id))
}

ctx.run(q1(liftQuery(List(1, 2))))
// SELECT p.id, p.name, p.age FROM Person p WHERE p.id IN (?, ?)

val peopleWithContacts = quote {
  query[Person].filter(p => query[Contact].filter(c => c.personId == p.id).nonEmpty)
}
val q2 = quote {
  query[Person].filter(p => peopleWithContacts.contains(p.id))
}

ctx.run(q2)
// SELECT p.id, p.name, p.age FROM Person p WHERE p.id IN (SELECT p1.* FROM Person p1 WHERE EXISTS (SELECT c.* FROM Contact c WHERE c.personId = p1.id))
```

### distinct
```scala
val q = quote {
  query[Person].map(p => p.age).distinct
}

ctx.run(q)
// SELECT DISTINCT p.age FROM Person p
```

### distinct on

> Note that `DISTINCT ON` is currently only supported in Postgres and H2.

```scala
val q = quote {
  query[Person].distinctOn(p => p.name)
}

ctx.run(q)
// SELECT DISTINCT ON (p.name) p.name, p.age FROM Person
```

Typically, `DISTINCT ON` is used with `SORT BY`.
```scala
val q = quote {
  query[Person].distinctOn(p => p.name).sortBy(p => p.age)
}

ctx.run(q)
// SELECT DISTINCT ON (p.name) p.name, p.age FROM Person ORDER BY p.age ASC NULLS FIRST
```
You can also use multiple fields in the `DISTINCT ON` criteria:
```scala
// final case class Person(firstName: String, lastName: String, age: Int)
val q = quote {
  query[Person].distinctOn(p => (p.firstName, p.lastName))
}

ctx.run(q)
// SELECT DISTINCT ON (p.firstName, p.lastName) p.firstName, p.lastName, p.age FROM Person p
```

### nested
```scala
val q = quote {
  query[Person].filter(p => p.name == "John").nested.map(p => p.age)
}

ctx.run(q)
// SELECT p.age FROM (SELECT p.age FROM Person p WHERE p.name = 'John') p
```

### joins
Joins are arguably the largest source of complexity in most SQL queries.
Quill offers a few different syntaxes so you can choose the right one for your use-case!

```scala
final case class A(id: Int)
final case class B(fk: Int)

// Applicative Joins:
quote {
  query[A].join(query[B]).on(_.id == _.fk)
}

// Implicit Joins:
quote {
  for {
    a <- query[A]
    b <- query[B] if (a.id == b.fk)
  } yield (a, b)
}

// Flat Joins:
quote {
  for {
    a <- query[A]
    b <- query[B].join(_.fk == a.id)
  } yield (a, b)
}
```

Let's see them one by one assuming the following schema:
```scala
final case class Person(id: Int, name: String)
final case class Address(street: String, zip: Int, fk: Int)
```
(Note: If your use case involves lots and lots of joins, both inner and outer. Skip right to the flat-joins section!)

#### applicative joins

Applicative joins are useful for joining two tables together,
they are straightforward to understand, and typically look good on one line.
Quill supports inner, left-outer, right-outer, and full-outer (i.e. cross) applicative joins.

```scala
// Inner Join
val q = quote {
  query[Person].join(query[Address]).on(_.id == _.fk)
}

ctx.run(q) //: List[(Person, Address)]
// SELECT x1.id, x1.name, x2.street, x2.zip, x2.fk
// FROM Person x1 INNER JOIN Address x2 ON x1.id = x2.fk

// Left (Outer) Join
val q = quote {
  query[Person].leftJoin(query[Address]).on((p, a) => p.id == a.fk)
}

ctx.run(q) //: List[(Person, Option[Address])]
// Note that when you use named-variables in your comprehension, Quill does its best to honor them in the query.
// SELECT p.id, p.name, a.street, a.zip, a.fk
// FROM Person p LEFT JOIN Address a ON p.id = a.fk

// Right (Outer) Join
val q = quote {
  query[Person].rightJoin(query[Address]).on((p, a) => p.id == a.fk)
}

ctx.run(q) //: List[(Option[Person], Address)]
// SELECT p.id, p.name, a.street, a.zip, a.fk
// FROM Person p RIGHT JOIN Address a ON p.id = a.fk

// Full (Outer) Join
val q = quote {
  query[Person].fullJoin(query[Address]).on((p, a) => p.id == a.fk)
}

ctx.run(q) //: List[(Option[Person], Option[Address])]
// SELECT p.id, p.name, a.street, a.zip, a.fk
// FROM Person p FULL JOIN Address a ON p.id = a.fk
```

What about joining more than two tables with the applicative syntax?
Here's how to do that:
```scala
final case class Company(zip: Int)

// All is well for two tables but for three or more, the nesting mess begins:
val q = quote {
  query[Person]
    .join(query[Address]).on({case (p, a) => p.id == a.fk}) // Let's use `case` here to stay consistent
    .join(query[Company]).on({case ((p, a), c) => a.zip == c.zip})
}

ctx.run(q) //: List[((Person, Address), Company)]
// (Unfortunately when you use `case` statements, Quill can't help you with the variables names either!)
// SELECT x01.id, x01.name, x11.street, x11.zip, x11.fk, x12.name, x12.zip
// FROM Person x01 INNER JOIN Address x11 ON x01.id = x11.fk INNER JOIN Company x12 ON x11.zip = x12.zip
```
No worries though, implicit joins and flat joins have your other use-cases covered!

#### implicit joins

Quill's implicit joins use a monadic syntax making them pleasant to use for joining many tables together.
They look a lot like Scala collections when used in for-comprehensions
making them familiar to a typical Scala developer.
What's the catch? They can only do inner-joins.

```scala
val q = quote {
  for {
    p <- query[Person]
    a <- query[Address] if (p.id == a.fk)
  } yield (p, a)
}

run(q) //: List[(Person, Address)]
// SELECT p.id, p.name, a.street, a.zip, a.fk
// FROM Person p, Address a WHERE p.id = a.fk
```

Now, this is great because you can keep adding more and more joins
without having to do any pesky nesting.
```scala
val q = quote {
  for {
    p <- query[Person]
    a <- query[Address] if (p.id == a.fk)
    c <- query[Company] if (c.zip == a.zip)
  } yield (p, a, c)
}

run(q) //: List[(Person, Address, Company)]
// SELECT p.id, p.name, a.street, a.zip, a.fk, c.name, c.zip
// FROM Person p, Address a, Company c WHERE p.id = a.fk AND c.zip = a.zip
```
Well that looks nice but wait! What If I need to inner, **and** outer join lots of tables nicely?
No worries, flat-joins are here to help!

### flat joins

Flat Joins give you the best of both worlds! In the monadic syntax, you can use both inner joins,
and left-outer joins together without any of that pesky nesting.

```scala
// Inner Join
val q = quote {
  for {
    p <- query[Person]
    a <- query[Address].join(a => a.fk == p.id)
  } yield (p,a)
}

ctx.run(q) //: List[(Person, Address)]
// SELECT p.id, p.name, a.street, a.zip, a.fk
// FROM Person p INNER JOIN Address a ON a.fk = p.id

// Left (Outer) Join
val q = quote {
  for {
    p <- query[Person]
    a <- query[Address].leftJoin(a => a.fk == p.id)
  } yield (p,a)
}

ctx.run(q) //: List[(Person, Option[Address])]
// SELECT p.id, p.name, a.street, a.zip, a.fk
// FROM Person p LEFT JOIN Address a ON a.fk = p.id
```

Now you can keep adding both right and left joins without nesting!
```scala
val q = quote {
  for {
    p <- query[Person]
    a <- query[Address].join(a => a.fk == p.id)
    c <- query[Company].leftJoin(c => c.zip == a.zip)
  } yield (p,a,c)
}

ctx.run(q) //: List[(Person, Address, Option[Company])]
// SELECT p.id, p.name, a.street, a.zip, a.fk, c.name, c.zip
// FROM Person p
// INNER JOIN Address a ON a.fk = p.id
// LEFT JOIN Company c ON c.zip = a.zip
```

Can't figure out what kind of join you want to use? Who says you have to choose?

With Quill the following multi-join queries are equivalent, use them according to preference:

```scala

final case class Employer(id: Int, personId: Int, name: String)

val qFlat = quote {
  for{
    (p,e) <- query[Person].join(query[Employer]).on(_.id == _.personId)
       c  <- query[Contact].leftJoin(_.personId == p.id)
  } yield(p, e, c)
}

val qNested = quote {
  for{
    ((p,e),c) <-
      query[Person].join(query[Employer]).on(_.id == _.personId)
      .leftJoin(query[Contact]).on(
        _._1.id == _.personId
      )
  } yield(p, e, c)
}

ctx.run(qFlat)
ctx.run(qNested)
// SELECT p.id, p.name, p.age, e.id, e.personId, e.name, c.id, c.phone
// FROM Person p INNER JOIN Employer e ON p.id = e.personId LEFT JOIN Contact c ON c.personId = p.id
```

Note that in some cases implicit and flat joins cannot be used together, for example, the following
query will fail.
```scala
val q = quote {
  for {
    p <- query[Person]
    p1 <- query[Person] if (p1.name == p.name)
    c <- query[Contact].leftJoin(_.personId == p.id)
  } yield (p, c)
}

// ctx.run(q)
// java.lang.IllegalArgumentException: requirement failed: Found an `ON` table reference of a table that is
// not available: Set(p). The `ON` condition can only use tables defined through explicit joins.
```
This happens because an explicit join typically cannot be done after an implicit join in the same query.

A good guideline is in any query or subquery, choose one of the following:
* Use flat-joins + applicative joins or
* Use implicit joins

Also, note that not all Option operations are available on outer-joined tables (i.e. tables wrapped in an `Option` object),
only a specific subset. This is mostly due to the inherent limitations of SQL itself. For more information, see the
'Optional Tables' section.

### Optionals / Nullable Fields

> Note that the behavior of Optionals has recently changed to include stricter null-checks. See the [orNull / getOrNull](#ornull--getornull) section for more details.

Option objects are used to encode nullable fields.
Say you have the following schema:
```sql
CREATE TABLE Person(
  id INT NOT NULL PRIMARY KEY,
  name VARCHAR(255) -- This is nullable!
);
CREATE TABLE Address(
  fk INT, -- This is nullable!
  street VARCHAR(255) NOT NULL,
  zip INT NOT NULL,
  CONSTRAINT a_to_p FOREIGN KEY (fk) REFERENCES Person(id)
);
CREATE TABLE Company(
  name VARCHAR(255) NOT NULL,
  zip INT NOT NULL
)
```
This would encode to the following:
```scala
final case class Person(id:Int, name:Option[String])
final case class Address(fk:Option[Int], street:String, zip:Int)
final case class Company(name:String, zip:Int)
```

Some important notes regarding Optionals and nullable fields.

> In many cases, Quill tries to rely on the null-fallthrough behavior that is ANSI standard:
>  * `null == null := false`
>  * `null == [true | false] := false`
>
> This allows the generated SQL for most optional operations to be simple. For example, the expression
> `Option[String].map(v => v + "foo")` can be expressed as the SQL `v || 'foo'` as opposed to
> `CASE IF (v is not null) v || 'foo' ELSE null END` so long as the concatenation operator `||`
> "falls-through" and returns `null` when the input is null. This is not true of all databases (e.g. [Oracle](https://community.oracle.com/ideas/19866)),
> forcing Quill to return the longer expression with explicit null-checking. Also, if there are conditionals inside
> of an Option operation (e.g. `o.map(v => if (v == "x") "y" else "z")`) this creates SQL with case statements,
> which will never fall-through when the input value is null. This forces Quill to explicitly null-check such statements in every
> SQL dialect.

Let's go through the typical operations of optionals.

#### isDefined / isEmpty

The `isDefined` method is generally a good way to null-check a nullable field:
```scala
val q = quote {
  query[Address].filter(a => a.fk.isDefined)
}
ctx.run(q)
// SELECT a.fk, a.street, a.zip FROM Address a WHERE a.fk IS NOT NULL
```
The `isEmpty` method works the same way:
```scala
val q = quote {
  query[Address].filter(a => a.fk.isEmpty)
}
ctx.run(q)
// SELECT a.fk, a.street, a.zip FROM Address a WHERE a.fk IS NULL
```

#### exists

This method is typically used for inspecting nullable fields inside of boolean conditions, most notably joining!
```scala
val q = quote {
  query[Person].join(query[Address]).on((p, a)=> a.fk.exists(_ == p.id))
}
ctx.run(q)
// SELECT p.id, p.name, a.fk, a.street, a.zip FROM Person p INNER JOIN Address a ON a.fk = p.id
```

Note that in the example above, the `exists` method does not cause the generated
SQL to do an explicit null-check in order to express the `False` case. This is because Quill relies on the
typical database behavior of immediately falsifying a statement that has `null` on one side of the equation.

#### forall

Use this method in boolean conditions that should succeed in the null case.
```scala
val q = quote {
  query[Person].join(query[Address]).on((p, a) => a.fk.forall(_ == p.id))
}
ctx.run(q)
// SELECT p.id, p.name, a.fk, a.street, a.zip FROM Person p INNER JOIN Address a ON a.fk IS NULL OR a.fk = p.id
```
Typically this is useful when doing negative conditions, e.g. when a field is **not** some specified value (e.g. `"Joe"`).
Being `null` in this case is typically a matching result.
```scala
val q = quote {
  query[Person].filter(p => p.name.forall(_ != "Joe"))
}

ctx.run(q)
// SELECT p.id, p.name FROM Person p WHERE p.name IS NULL OR p.name <> 'Joe'
```

#### filterIfDefined
Use this to filter by a optional field that you want to ignore when None.
This is useful when you want to filter by a map-key that may or may not exist.

```scala
val fieldFilters: Map[String, String] = Map("name" -> "Joe", "age" -> "123")
val q = quote {
  query[Person].filter(p => lift(fieldFilters.get("name)).filterIfDefined(_ == p.name))
}
 
ctx.run(q)
// SELECT p.id, p.name, p.title FROM Person p WHERE p.title IS NULL OR p.title = 'The Honorable'
```

It also works for regular fields.
```scala
// final case class Person(name: String, age: Int, title: Option[String])
val q = quote {
  query[Person].filter(p => p.title.filterIfDefined(_ == "The Honorable"))
}
 
ctx.run(q)
// SELECT p.id, p.name, p.title FROM Person p WHERE p.title IS NULL OR p.title = 'The Honorable'
```

#### map
As in regular Scala code, performing any operation on an optional value typically requires using the `map` function.
```scala
val q = quote {
 for {
    p <- query[Person]
  } yield (p.id, p.name.map("Dear " + _))
}

ctx.run(q)
// SELECT p.id, 'Dear ' || p.name FROM Person p
// * In Dialects where `||` does not fall-through for nulls (e.g. Oracle):
// * SELECT p.id, CASE WHEN p.name IS NOT NULL THEN 'Dear ' || p.name ELSE null END FROM Person p
```

Additionally, this method is useful when you want to get a non-optional field out of an outer-joined table
(i.e. a table wrapped in an `Option` object).

```scala
val q = quote {
  query[Company].leftJoin(query[Address])
    .on((c, a) => c.zip == a.zip)
    .map {case(c,a) =>                          // Row type is (Company, Option[Address])
      (c.name, a.map(_.street), a.map(_.zip))   // Use `Option.map` to get `street` and `zip` fields
    }
}

run(q)
// SELECT c.name, a.street, a.zip FROM Company c LEFT JOIN Address a ON c.zip = a.zip
```

For more details about this operation (and some caveats), see the 'Optional Tables' section.

#### flatMap and flatten

Use these when the `Option.map` functionality is not sufficient. This typically happens when you need to manipulate
multiple nullable fields in a way which would otherwise result in `Option[Option[T]]`.
```scala
val q = quote {
  for {
    a <- query[Person]
    b <- query[Person] if (a.id > b.id)
  } yield (
    // If this was `a.name.map`, resulting record type would be Option[Option[String]]
    a.name.flatMap(an =>
      b.name.map(bn =>
        an+" comes after "+bn)))
}

ctx.run(q) //: List[Option[String]]
// SELECT (a.name || ' comes after ') || b.name FROM Person a, Person b WHERE a.id > b.id
// * In Dialects where `||` does not fall-through for nulls (e.g. Oracle):
// * SELECT CASE WHEN a.name IS NOT NULL AND b.name IS NOT NULL THEN (a.name || ' comes after ') || b.name ELSE null END FROM Person a, Person b WHERE a.id > b.id

// Alternatively, you can use `flatten`
val q = quote {
  for {
    a <- query[Person]
    b <- query[Person] if (a.id > b.id)
  } yield (
    a.name.map(an =>
      b.name.map(bn =>
        an + " comes after " + bn)).flatten)
}

ctx.run(q) //: List[Option[String]]
// SELECT (a.name || ' comes after ') || b.name FROM Person a, Person b WHERE a.id > b.id
```
This is also very useful when selecting from outer-joined tables i.e. where the entire table
is inside of an `Option` object. Note how below we get the `fk` field from `Option[Address]`.

```scala
val q = quote {
  query[Person].leftJoin(query[Address])
    .on((p, a) => a.fk.exists(_ == p.id))
    .map {case (p /*Person*/, a /*Option[Address]*/) => (p.name, a.flatMap(_.fk))}
}

ctx.run(q) //: List[(Option[String], Option[Int])]
// SELECT p.name, a.fk FROM Person p LEFT JOIN Address a ON a.fk = p.id
```

#### orNull / getOrNull

The `orNull` method can be used to convert an Option-enclosed row back into a regular row.
Since `Option[T].orNull` does not work for primitive types (e.g. `Int`, `Double`, etc...),
you can use the `getOrNull` method inside of quoted blocks to do the same thing.

> Note that since the presence of null columns can cause queries to break in some data sources (e.g. Spark), so use this operation very carefully.

```scala
val q = quote {
  query[Person].join(query[Address])
    .on((p, a) => a.fk.exists(_ == p.id))
    .filter {case (p /*Person*/, a /*Option[Address]*/) =>
      a.fk.getOrNull != 123 } // Exclude a particular value from the query.
                              // Since we already did an inner-join on this value, we know it is not null.
}

ctx.run(q) //: List[(Address, Person)]
// SELECT p.id, p.name, a.fk, a.street, a.zip FROM Person p INNER JOIN Address a ON a.fk IS NOT NULL AND a.fk = p.id WHERE a.fk <> 123
```

In certain situations, you may wish to pretend that a nullable-field is not actually nullable and perform regular operations
(e.g. arithmetic, concatenation, etc...) on the field. You can use a combination of `Option.apply` and `orNull` (or `getOrNull` where needed)
in order to do this.

```scala
val q = quote {
  query[Person].map(p => Option(p.name.orNull + " suffix"))
}

ctx.run(q)
// SELECT p.name || ' suffix' FROM Person p
// i.e. same as the previous behavior
```

In all other situations, since Quill strictly checks nullable values, and `case.. if` conditionals will work correctly in all Optional constructs.
However, since they may introduce behavior changes in your codebase, the following warning has been introduced:

> Conditionals inside of Option.[map | flatMap | exists | forall] will create a `CASE` statement in order to properly null-check the sub-query (...)

```
val q = quote {
  query[Person].map(p => p.name.map(n => if (n == "Joe") "foo" else "bar").getOrElse("baz"))
}
// Information:(16, 15) Conditionals inside of Option.map will create a `CASE` statement in order to properly null-check the sub-query: `p.name.map((n) => if(n == "Joe") "foo" else "bar")`.
// Expressions like Option(if (v == "foo") else "bar").getOrElse("baz") will now work correctly, but expressions that relied on the broken behavior (where "bar" would be returned instead) need to be modified  (see the "orNull / getOrNull" section of the documentation of more detail).

ctx.run(a)
// Used to be this:
// SELECT CASE WHEN CASE WHEN p.name = 'Joe' THEN 'foo' ELSE 'bar' END IS NOT NULL THEN CASE WHEN p.name = 'Joe' THEN 'foo' ELSE 'bar' END ELSE 'baz' END FROM Person p
// Now is this:
// SELECT CASE WHEN p.name IS NOT NULL AND CASE WHEN p.name = 'Joe' THEN 'foo' ELSE 'bar' END IS NOT NULL THEN CASE WHEN p.name = 'Joe' THEN 'foo' ELSE 'bar' END ELSE 'baz' END FROM Person p
```

### equals

The `==`, `!=`, and `.equals` methods can be used to compare regular types as well Option types in a scala-idiomatic way.
That is to say, either `T == T` or `Option[T] == Option[T]` is supported and the following "truth-table" is observed:

Left         | Right        | Equality   | Result
-------------|--------------|------------|----------
`a`          | `b`          | `==`       | `a == b`
`Some[T](a)` | `Some[T](b)` | `==`       | `a == b`
`Some[T](a)` | `None`       | `==`       | `false`
`None      ` | `Some[T](b)` | `==`       | `false`
`None      ` | `None`       | `==`       | `true`
`Some[T]   ` | `Some[R]   ` | `==`       | Exception thrown.
`a`          | `b`          | `!=`       | `a != b`
`Some[T](a)` | `Some[T](b)` | `!=`       | `a != b`
`Some[T](a)` | `None`       | `!=`       | `true`
`None      ` | `Some[T](b)` | `!=`       | `true`
`Some[T]   ` | `Some[R]   ` | `!=`       | Exception thrown.
`None      ` | `None`       | `!=`       | `false`

```scala
final case class Node(id:Int, status:Option[String], otherStatus:Option[String])

val q = quote { query[Node].filter(n => n.id == 123) }
ctx.run(q)
// SELECT n.id, n.status, n.otherStatus FROM Node n WHERE p.id = 123

val q = quote { query[Node].filter(r => r.status == r.otherStatus) }
ctx.run(q)
// SELECT r.id, r.status, r.otherStatus FROM Node r WHERE r.status IS NULL AND r.otherStatus IS NULL OR r.status = r.otherStatus

val q = quote { query[Node].filter(n => n.status == Option("RUNNING")) }
ctx.run(q)
// SELECT n.id, n.status, n.otherStatus FROM node n WHERE n.status IS NOT NULL AND n.status = 'RUNNING'

val q = quote { query[Node].filter(n => n.status != Option("RUNNING")) }
ctx.run(q)
// SELECT n.id, n.status, n.otherStatus FROM node n WHERE n.status IS NULL OR n.status <> 'RUNNING'
```

If you would like to use an equality operator that follows that ansi-idiomatic approach, failing
the comparison if either side is null as well as the principle that `null = null := false`, you can import `===` (and `=!=`)
from `Context.extras`. These operators work across `T` and `Option[T]` allowing comparisons like `T === Option[T]`,
`Option[T] == T` etc... to be made. You can use also `===`
directly in Scala code and it will have the same behavior, returning `false` when other the left-hand
or right-hand side is `None`. This is particularity useful in paradigms like Spark where
you will typically transition inside and outside of Quill code.

> When using `a === b` or `a =!= b` sometimes you will see the extra `a IS NOT NULL AND b IS NOT NULL` comparisons
> and sometimes you will not. This depends on `equalityBehavior` in `SqlIdiom` which determines whether the given SQL
> dialect already does ansi-idiomatic comparison to `a`, and `b` when an `=` operator is used,
> this allows us to omit the extra `a IS NOT NULL AND b IS NOT NULL`.

```scala

// === works the same way inside of a quotation
val q = run( query[Node].filter(n => n.status === "RUNNING") )
// SELECT n.id, n.status FROM node n WHERE n.status IS NOT NULL AND n.status = 'RUNNING'

// as well as outside
(nodes:List[Node]).filter(n => n.status === "RUNNING")
```

#### Optional Tables

As we have seen in the examples above, only the `map` and `flatMap` methods are available on outer-joined tables
(i.e. tables wrapped in an `Option` object).

Since you cannot use `Option[Table].isDefined`, if you want to null-check a whole table
(e.g. if a left-join was not matched), you have to `map` to a specific field on which you can do the null-check.

```scala
val q = quote {
  query[Company].leftJoin(query[Address])
    .on((c, a) => c.zip == a.zip)         // Row type is (Company, Option[Address])
    .filter({case(c,a) => a.isDefined})   // You cannot null-check a whole table!
}
```

Instead, map the row-variable to a specific field and then check that field.
```scala
val q = quote {
  query[Company].leftJoin(query[Address])
    .on((c, a) => c.zip == a.zip)                     // Row type is (Company, Option[Address])
    .filter({case(c,a) => a.map(_.street).isDefined}) // Null-check a non-nullable field instead
}
ctx.run(q)
// SELECT c.name, c.zip, a.fk, a.street, a.zip
// FROM Company c
// LEFT JOIN Address a ON c.zip = a.zip
// WHERE a.street IS NOT NULL
```

Finally, it is worth noting that a whole table can be wrapped into an `Option` object. This is particularly
useful when doing a union on table-sets that are both right-joined and left-joined together.
```scala
val aCompanies = quote {
  for {
    c <- query[Company] if (c.name like "A%")
    a <- query[Address].join(_.zip == c.zip)
  } yield (c, Option(a))  // change (Company, Address) to (Company, Option[Address])
}
val bCompanies = quote {
  for {
    c <- query[Company] if (c.name like "A%")
    a <- query[Address].leftJoin(_.zip == c.zip)
  } yield (c, a) // (Company, Option[Address])
}
val union = quote {
  aCompanies union bCompanies
}
ctx.run(union)
// SELECT x.name, x.zip, x.fk, x.street, x.zip FROM (
// (SELECT c.name name, c.zip zip, x1.zip zip, x1.fk fk, x1.street street
// FROM Company c INNER JOIN Address x1 ON x1.zip = c.zip WHERE c.name like 'A%')
// UNION
// (SELECT c1.name name, c1.zip zip, x2.zip zip, x2.fk fk, x2.street street
// FROM Company c1 LEFT JOIN Address x2 ON x2.zip = c1.zip WHERE c1.name like 'A%')
// ) x
```

### Ad-Hoc Case Classes

Case Classes can also be used inside quotations as output values:

```scala
final case class Person(id: Int, name: String, age: Int)
final case class Contact(personId: Int, phone: String)
final case class ReachablePerson(name:String, phone: String)

val q = quote {
  for {
    p <- query[Person] if(p.id == 999)
    c <- query[Contact] if(c.personId == p.id)
  } yield {
    ReachablePerson(p.name, c.phone)
  }
}

ctx.run(q)
// SELECT p.name, c.phone FROM Person p, Contact c WHERE (p.id = 999) AND (c.personId = p.id)
```

As well as in general:

```scala
final case class IdFilter(id:Int)

val q = quote {
  val idFilter = new IdFilter(999)
  for {
    p <- query[Person] if(p.id == idFilter.id)
    c <- query[Contact] if(c.personId == p.id)
  } yield {
    ReachablePerson(p.name, c.phone)
  }
}

ctx.run(q)
// SELECT p.name, c.phone FROM Person p, Contact c WHERE (p.id = 999) AND (c.personId = p.id)
```
***Note*** however that this functionality has the following restrictions:
1. The Ad-Hoc Case Class can only have one constructor with one set of parameters.
2. The Ad-Hoc Case Class must be constructed inside the quotation using one of the following methods:
    1. Using the `new` keyword: `new Person("Joe", "Bloggs")`
    2. Using a companion object's apply method:  `Person("Joe", "Bloggs")`
    3. Using a companion object's apply method explicitly: `Person.apply("Joe", "Bloggs")`
4. Any custom logic in a constructor/apply-method of an Ad-Hoc case class will not be invoked when it is 'constructed' inside a quotation. To construct an Ad-Hoc case class with custom logic inside a quotation, you can use a quoted method.

## Query probing

Query probing validates queries against the database at compile time, failing the compilation if it is not valid. The query validation does not alter the database state.

This feature is disabled by default. To enable it, mix the `QueryProbing` trait to the database configuration:

```
object myContext extends YourContextType with QueryProbing
```

The context must be created in a separate compilation unit in order to be loaded at compile time. Please use [this guide](https://www.scala-sbt.org/0.13/docs/Macro-Projects.html) that explains how to create a separate compilation unit for macros, that also serves to the purpose of defining a query-probing-capable context. `context` could be used instead of `macros` as the name of the separate compilation unit.

The configurations correspondent to the config key must be available at compile time. You can achieve it by adding this line to your project settings:

```
unmanagedClasspath in Compile += baseDirectory.value / "src" / "main" / "resources"
```

If your project doesn't have a standard layout, e.g. a play project, you should configure the path to point to the folder that contains your config file.

## Actions

Database actions are defined using quotations as well. These actions don't have a collection-like API but rather a custom DSL to express inserts, deletes, and updates.

### insertValue / insert

```scala
val a = quote(query[Contact].insertValue(lift(Contact(999, "+1510488988"))))

ctx.run(a) // = 1 if the row was inserted 0 otherwise
// INSERT INTO Contact (personId,phone) VALUES (?, ?)
```

#### It is also possible to insert specific columns (via insert):

```scala
val a = quote {
  query[Contact].insert(_.personId -> lift(999), _.phone -> lift("+1510488988"))
}

ctx.run(a)
// INSERT INTO Contact (personId,phone) VALUES (?, ?)
```

### batch insert
```scala
val a = quote {
  liftQuery(List(Person(0, "John", 31),Person(2, "name2", 32))).foreach(e => query[Person].insertValue(e))
}

ctx.run(a) //: List[Long] size = 2. Contains 1 @ positions, where row was inserted E.g List(1,1)
// INSERT INTO Person (id,name,age) VALUES (?, ?, ?)
```
> In addition to regular JDBC batching, Quill can optimize batch queries by using multiple VALUES-clauses e.g:
> ```scala
> ctx.run(a, 2)
> // INSERT INTO Person (id,name,age) VALUES (?, ?, ?), (?, ?, ?) // Note, the extract (?, ?, ?) will not be visible in the compiler output. 
> ```
> In situations with high network latency this can improve performance by 20-40x! See the [Batch Optimization](#batch-optimization) below for more info.

Just as in regular queries use the extended insert/update syntaxes to achieve finer-grained control of the data being created/modified modified.
For example, if the ID is a generated value you can skip ID insertion like this:
(This can also be accomplished with an insert-meta).
```scala
// final case class Person(id: Int, name: String, age: Int)
val a = quote {
  liftQuery(List(Person(0, "John", 31),Person(0, "name2", 32))).foreach(e => query[Person].insert(_.name -> p.name, _.age -> p.age))
}

ctx.run(a)
// INSERT INTO Person (name,age) VALUES (?, ?)
```

Batch queries can also have a returning/returningGenerated clause:
```scala
// final case class Person(id: Int, name: String, age: Int)
val a = quote {
  liftQuery(List(Person(0, "John", 31),Person(0, "name2", 32))).foreach(e => query[Person].insert(_.name -> p.name, _.age -> p.age)).returning(_.id)
}

ctx.run(a)
// INSERT INTO Person (name,age) VALUES (?, ?) RETURNING id
```

Note that the `liftQuery[Something]` and the query[Something]` values do not necessarily need to be the same object-type.
(In fact the liftQuery value can even be a constant!)
For example:
```scala
// final case class Person(name: String, age: Int)
// final case class Vip(first: String, last: String, age: Int)
// val vips: List[Vip] = ...
val q = quote {
  liftQuery(vips).foreach(v => query[Person].insertValue(Person(v.first + v.last, v.age)))
}

ctx.run(q)
// INSERT INTO Person (name,age) VALUES ((? || ?), ?)
```

Note that UPDATE queries can also be done in batches (as well as DELETE queries).
```scala
val q = quote {
  liftQuery(vips).foreach(v => query[Person].filter(p => p.age > 22).updateValue(Person(v.first + v.last, v.age)))
}

ctx.run(q)
// UPDATE Person SET name = (? || ?), age = ? WHERE age > 22
```

### updateValue / update
```scala
val a = quote {
  query[Person].filter(_.id == 999).updateValue(lift(Person(999, "John", 22)))
}

ctx.run(a) // = Long number of rows updated
// UPDATE Person SET id = ?, name = ?, age = ? WHERE id = 999
```

#### Using specific columns (via update):

```scala
val a = quote {
  query[Person].filter(p => p.id == lift(999)).update(_.age -> lift(18))
}

ctx.run(a)
// UPDATE Person SET age = ? WHERE id = ?
```

#### Using columns as part of the update:

```scala
val a = quote {
  query[Person].filter(p => p.id == lift(999)).update(p => p.age -> (p.age + 1))
}

ctx.run(a)
// UPDATE Person SET age = (age + 1) WHERE id = ?
```

### batch update

```scala
val a = quote {
  liftQuery(List(Person(1, "name", 31),Person(2, "name2", 32))).foreach { person =>
     query[Person].filter(_.id == person.id).update(_.name -> person.name, _.age -> person.age)
  }
}

ctx.run(a) // : List[Long] size = 2. Contains 1 @ positions, where row was inserted E.g List(1,0)
// UPDATE Person SET name = ?, age = ? WHERE id = ?
```

### delete
```scala
val a = quote {
  query[Person].filter(p => p.name == "").delete
}

ctx.run(a) // = Long the number of rows deleted
// DELETE FROM Person WHERE name = ''
```

### insert or update (upsert, conflict)

Upsert is supported by Postgres, SQLite, MySQL and H2 `onConflictIgnore` only (since v1.4.200 in PostgreSQL compatibility mode)

#### Postgres and SQLite

##### Ignore conflict
```scala
val a = quote {
  query[Product].insert(_.id -> 1, _.sku -> 10).onConflictIgnore
}

// INSERT INTO Product AS t (id,sku) VALUES (1, 10) ON CONFLICT DO NOTHING
```

Ignore conflict by explicitly setting conflict target
```scala
val a = quote {
  query[Product].insert(_.id -> 1, _.sku -> 10).onConflictIgnore(_.id)
}

// INSERT INTO Product AS t (id,sku) VALUES (1, 10) ON CONFLICT (id) DO NOTHING
```

Multiple properties can be used as well.
```scala
val a = quote {
  query[Product].insert(_.id -> 1, _.sku -> 10).onConflictIgnore(_.id, _.description)
}

// INSERT INTO Product (id,sku) VALUES (1, 10) ON CONFLICT (id,description) DO NOTHING
```

##### Update on Conflict

Resolve conflict by updating existing row if needed. In `onConflictUpdate(target)((t, e) => assignment)`: `target` refers to
conflict target, `t` - to existing row and `e` - to excluded, e.g. row proposed for insert.
```scala
val a = quote {
  query[Product]
    .insert(_.id -> 1, _.sku -> 10)
    .onConflictUpdate(_.id)((t, e) => t.sku -> (t.sku + e.sku))
}

// INSERT INTO Product AS t (id,sku) VALUES (1, 10) ON CONFLICT (id) DO UPDATE SET sku = (t.sku + EXCLUDED.sku)
```
Multiple properties can be used with `onConflictUpdate` as well.
```scala
val a = quote {
  query[Product]
    .insert(_.id -> 1, _.sku -> 10)
    .onConflictUpdate(_.id, _.description)((t, e) => t.sku -> (t.sku + e.sku))
}

INSERT INTO Product AS t (id,sku) VALUES (1, 10) ON CONFLICT (id,description) DO UPDATE SET sku = (t.sku + EXCLUDED.sku)
```

#### MySQL

Ignore any conflict, e.g. `insert ignore`
```scala
val a = quote {
  query[Product].insert(_.id -> 1, _.sku -> 10).onConflictIgnore
}

// INSERT IGNORE INTO Product (id,sku) VALUES (1, 10)
```

Ignore duplicate key conflict by explicitly setting it
```scala
val a = quote {
  query[Product].insert(_.id -> 1, _.sku -> 10).onConflictIgnore(_.id)
}

// INSERT INTO Product (id,sku) VALUES (1, 10) ON DUPLICATE KEY UPDATE id=id
```

Resolve duplicate key by updating existing row if needed. In `onConflictUpdate((t, e) => assignment)`: `t` refers to
existing row and `e` - to values, e.g. values proposed for insert.
```scala
val a = quote {
  query[Product]
    .insert(_.id -> 1, _.sku -> 10)
    .onConflictUpdate((t, e) => t.sku -> (t.sku + e.sku))
}

// INSERT INTO Product (id,sku) VALUES (1, 10) ON DUPLICATE KEY UPDATE sku = (sku + VALUES(sku))
```

## Batch Optimization

When doing batch INSERT queries (as well as UPDATE, and DELETE), Quill mostly delegates the functionality to standard JDBC batching.
This functionality works roughly in the following way.
```scala
val ps: PreparedStatement = connection.prepareStatement("INSERT ... VALUES ...")
// 1. Iterate over the rows
for (row <- rowsToInsert) {
  // 2. For each row, add the columns to the prepared statement
  for ((column, columnIndex) <- row)
    row.setColumn(column, columnIndex)
  // 3. Add the row to the list of things being added in the batch
  ps.addBatch()
}
// 4. Write everything in the batch to the Database
ps.executeBatch()
```
Reasonably speaking, we would expect each call in Stage #3 to locally stage the value of the row and then submit
all of the rows to the database in Stage #4 but that basically every database that is not what happens. In Stage #3,
a network call is actually made to the Database to remotely stage the row. Practically this means that the performance of
addBatch/executeBatch degrades per-row, per-millisecond-network-latency. Even at 50 milliseconds of network latency
the impact of this is highly significant:

|Network Latency | Rows Inserted | Total Time
|-------|-----------|---------|
| 0ms   | 10k rows  | 0.486   |
| 50ms  | 10k rows  | 3.226   |
| 100ms | 10k rows  | 5.266   |
| 0ms   | 100k rows | 1.416   |
| 50ms  | 100k rows | 23.248  |
| 100ms | 100k rows | 43.077  |
| 0ms   | 1m rows   | 13.616  |
| 50ms  | 1m rows   | 234.452 |
| 100ms | 1m rows   | 406.101 |

In order to alleviate this problem Quill can take advantage of the ability of most database dialects to use multiple
VALUES-clauses to batch-insert rows. Conceptually, this works in the following way:
```scala
final case class Person(name: String, age: Int)
val people = List(Person("Joe", 22), Person("Jack", 33), Person("Jill", 44))
val q = quote { liftQuery(people).foreach(p => query[Person].insertValue(p)) }
run(q, 2) // i.e. insert rows from the `people` list in batches of 2
//
// Query1) INSERT INTO Person (name, age) VALUES ([Joe] , [22]), ([Jack], [33])
//         INSERT INTO Person (name, age) VALUES (  ?   ,  ?  ), (   ?  ,  ?  ) <- actual query
// Query2) INSERT INTO Person (name, age) VALUES ([Jill], [44])
//         INSERT INTO Person (name, age) VALUES (  ?   ,  ?  )                 <- actual query
```
> Note that only `INSERT INTO Person (name, age) VALUES (?, ?)` will appear in the compiler-output for this query!

Using a batch-count of about 1000-5000 rows (i.e. `run(q, 1000)`) can significantly improve query performance:

| Network Latency | Rows Inserted | Total Time |
|-----------------|---------------|------------|
| 0ms             | 10k rows      | 3.772      |
| 50ms            | 10k rows      | 3.899      |
| 100ms           | 10k rows      | 4.63       |
| 0ms             | 100k rows     | 2.902      |
| 50ms            | 100k rows     | 3.225      |
| 100ms           | 100k rows     | 3.554      |
| 0ms             | 1m rows       | 9.923      |
| 50ms            | 1m rows       | 10.035     |
| 100ms           | 1m rows       | 10.328     |

One thing to take note of is that each one of the `?` placeholders above is a prepared-statement variable. This means
that in batch-sizes of 1000, there will be 1000 `?` variables in each query. In many databases this has a strict limit.
For example, in Postgres this is restricted to 32767. This means that when using batches of 1000 rows, each row can have
up to 32 columns or the following error will occur:
```
IOException: Tried to send an out-of-range integer as a 2-byte value
```
In other database e.g. SQL Server, unfortunately this limit is much smaller. For example in SQL Server it is just 2100 variables
or the following error will occur.
```
The server supports a maximum of 2100 parameters. Reduce the number of parameters and resend the request
```
This means that in SQL Server, for a batch-size of 100, you can only insert into a table of up to 21 columns.

In the future, we hope to alleviate this issue by directly substituting variables into `?` variables before the query is executed
however such functionality could potentially come at the risk of SQL-injection vulnerabilities.

## Printing Queries

The `translate` method is used to convert a Quill query into a string which can then be printed.

```scala
val str = ctx.translate(query[Person])
println(str)
// SELECT x.id, x.name, x.age FROM Person x
```

Insert queries can also be printed:

```scala
val str = ctx.translate(query[Person].insertValue(lift(Person(0, "Joe", 45))))
println(str)
// INSERT INTO Person (id,name,age) VALUES (0, 'Joe', 45)
```

As well as batch insertions:

```scala
val q = quote {
  liftQuery(List(Person(0, "Joe",44), Person(1, "Jack",45)))
    .foreach(e => query[Person].insertValue(e))
}
val strs: List[String] = ctx.translate(q)
strs.map(println)
// INSERT INTO Person (id, name,age) VALUES (0, 'Joe', 44)
// INSERT INTO Person (id, name,age) VALUES (1, 'Jack', 45)
```

The `translate` method is available in every Quill context as well as the Cassandra and OrientDB contexts,
the latter two, however, do not support Insert and Batch Insert query printing.

## IO Monad

Quill provides an IO monad that allows the user to express multiple computations and execute them separately. This mechanism is also known as a free monad, which provides a way of expressing computations as referentially-transparent values and isolates the unsafe IO operations into a single operation. For instance:

```scala
// this code using Future

final case class Person(id: Int, name: String, age: Int)

val p = Person(0, "John", 22)
ctx.run(query[Person].insertValue(lift(p))).flatMap { _ =>
  ctx.run(query[Person])
}

// isn't referentially transparent because if you refactor the second database
// interaction into a value, the result will be different:

val allPeople = ctx.run(query[Person])
ctx.run(query[Person].insertValue(lift(p))).flatMap { _ =>
  allPeople
}

// this happens because `ctx.run` executes the side-effect (database IO) immediately
```

```scala
// The IO monad doesn't perform IO immediately, so both computations:

val p = Person(0, "John", 22)

val a =
  ctx.runIO(query[Person].insertValue(lift(p))).flatMap { _ =>
    ctx.runIO(query[Person])
  }

val allPeople = ctx.runIO(query[Person])

val b =
  ctx.runIO(query[Person].insertValue(lift(p))).flatMap { _ =>
    allPeople
  }

// produce the same result when executed

performIO(a) == performIO(b)
```

The IO monad has an interface similar to `Future`; please refer to [the class](https://github.com/getquill/quill/blob/master/quill-core/src/main/scala/io/getquill/monad/IOMonad.scala#L39) for more information regarding the available operations.

The return type of `performIO` varies according to the context. For instance, async contexts return `Future`s while JDBC returns values synchronously.

***NOTE***: Avoid using the variable name `io` since it conflicts with Quill's package `io.getquill`, otherwise you will get the following error.
```
recursive value io needs type
```

### IO Monad and transactions

`IO` also provides the `transactional` method that delimits a transaction:

```scala
val a =
  ctx.runIO(query[Person].insertValue(lift(p))).flatMap { _ =>
    ctx.runIO(query[Person])
  }

performIO(a.transactional) // note: transactional can be used outside of `performIO`
```

### Getting a ResultSet

Quill JDBC Contexts allow you to use `prepare` in order to get a low-level `ResultSet` that is useful
for interacting with legacy APIs. This function  returns a `f: (Connection) => (PreparedStatement)`
closure as opposed to a `PreparedStatement` in order to guarantee that JDBC Exceptions are not
thrown until you can wrap them into the appropriate Exception-handling mechanism (e.g.
`try`/`catch`, `Try` etc...).

```scala
val q = quote {
  query[Product].filter(_.id == 1)
}
val preparer: (Connection) => (PreparedStatement)  = ctx.prepare(q)
// SELECT x1.id, x1.description, x1.sku FROM Product x1 WHERE x1.id = 1

// Use ugly stateful code, bracketed effects, or try-with-resources here:
var preparedStatement: PreparedStatement = _
var resultSet: ResultSet = _

try {
  preparedStatement = preparer(myCustomDataSource.getConnection)
  resultSet = preparedStatement.executeQuery()
} catch {
  case e: Exception =>
    // Close the preparedStatement and catch possible exceptions
    // Close the resultSet and catch possible exceptions
}
```

The `prepare` function can also be used with `insertValue`, and `updateValue` actions.

```scala
val q = quote {
  query[Product].insertValue(lift(Product(1, "Desc", 123))
}
val preparer: (Connection) => (PreparedStatement)  = ctx.prepare(q)
// INSERT INTO Product (id,description,sku) VALUES (?, ?, ?)
```

As well as with batch queries.
> Make sure to first quote your batch query and then pass the result into the `prepare` function
(as is done in the example below) or the Scala compiler may not type the output correctly
[#1518](https://github.com/getquill/quill/issues/1518).

```scala
val q = quote {
  liftQuery(products).foreach(e => query[Product].insertValue(e))
}
val preparers: Connection => List[PreparedStatement] = ctx.prepare(q)
val preparedStatement: List[PreparedStatement] = preparers(jdbcConf.dataSource.getConnection)
```

### Effect tracking

The IO monad tracks the effects that a computation performs in its second type parameter:

```scala
val a: IO[ctx.RunQueryResult[Person], Effect.Write with Effect.Read] =
  ctx.runIO(query[Person].insertValue(lift(p))).flatMap { _ =>
    ctx.runIO(query[Person])
  }
```

This mechanism is useful to limit the kind of operations that can be performed. See this [blog post](https://danielwestheide.com/blog/2015/06/28/put-your-writes-where-your-master-is-compile-time-restriction-of-slick-effect-types.html) as an example.

## Implicit query

Quill provides implicit conversions from case class companion objects to `query[T]` through an additional trait:

```scala
val ctx = new SqlMirrorContext(MirrorSqlDialect, Literal) with ImplicitQuery

val q = quote {
  for {
    p <- Person if(p.id == 999)
    c <- Contact if(c.personId == p.id)
  } yield {
    (p.name, c.phone)
  }
}

ctx.run(q)
// SELECT p.name, c.phone FROM Person p, Contact c WHERE (p.id = 999) AND (c.personId = p.id)
```

Note the usage of `Person` and `Contact` instead of `query[Person]` and `query[Contact]`.

## SQL-specific operations

Some operations are SQL-specific and not provided with the generic quotation mechanism. The SQL contexts provide implicit classes for this kind of operation:

```scala
val ctx = new SqlMirrorContext(MirrorSqlDialect, Literal)

```

### like

```scala
val q = quote {
  query[Person].filter(p => p.name like "%John%")
}
ctx.run(q)
// SELECT p.id, p.name, p.age FROM Person p WHERE p.name like '%John%'
```

### forUpdate

```scala
val q = quote {
  query[Person].filter(p => p.name == "Mary").forUpdate()
}
ctx.run(q)
// SELECT p.id, p.name, p.age FROM Person p WHERE p.name = 'Mary' FOR UPDATE
```

## SQL-specific encoding

### Arrays

Quill provides SQL Arrays support. In Scala we represent them as any collection that implements `Seq`:
```scala

final case class Book(id: Int, notes: List[String], pages: Vector[Int], history: Seq[Date])

ctx.run(query[Book])
// SELECT x.id, x.notes, x.pages, x.history FROM Book x
```
Note that not all drivers/databases provides such feature hence only `PostgresJdbcContext` support SQL Arrays.

## Cassandra-specific encoding

```scala
val ctx = new CassandraMirrorContext(Literal)

```

### Collections

The Cassandra context provides List, Set, and Map encoding:

```scala

final case class Book(id: Int, notes: Set[String], pages: List[Int], history: Map[Int, Boolean])

ctx.run(query[Book])
// SELECT id, notes, pages, history FROM Book
```

### User-Defined Types

The cassandra context provides encoding of UDT (user-defined types).
```scala

final case class Name(firstName: String, lastName: String) extends Udt
```

To encode the UDT and bind it into the query (insert/update queries), the context needs to retrieve UDT metadata from
the cluster object. By default, the context looks for UDT metadata within the currently logged keyspace, but it's also possible to specify a
concrete keyspace with `udtMeta`:

```scala
implicit val nameMeta = udtMeta[Name]("keyspace2.my_name")
```
When a keyspace is not set in `udtMeta` then the currently logged one is used.

Since it's possible to create a context without
specifying a keyspace, (e.g. the keyspace parameter is null and the session is not bound to any keyspace), the UDT metadata will be
resolved throughout the entire cluster.

It is also possible to rename UDT columns with `udtMeta`:

```scala
implicit val nameMeta = udtMeta[Name]("name", _.firstName -> "first", _.lastName -> "last")
```

## Cassandra-specific operations

The cassandra context also provides a few additional operations:

### allowFiltering
```scala
val q = quote {
  query[Person].filter(p => p.age > 10).allowFiltering
}
ctx.run(q)
// SELECT id, name, age FROM Person WHERE age > 10 ALLOW FILTERING
```

### ifNotExists
```scala
val q = quote {
  query[Person].insert(_.age -> 10, _.name -> "John").ifNotExists
}
ctx.run(q)
// INSERT INTO Person (age,name) VALUES (10, 'John') IF NOT EXISTS
```

### ifExists
```scala
val q = quote {
  query[Person].filter(p => p.name == "John").delete.ifExists
}
ctx.run(q)
// DELETE FROM Person WHERE name = 'John' IF EXISTS
```

### usingTimestamp
```scala
val q1 = quote {
  query[Person].insert(_.age -> 10, _.name -> "John").usingTimestamp(99)
}
ctx.run(q1)
// INSERT INTO Person (age,name) VALUES (10, 'John') USING TIMESTAMP 99

val q2 = quote {
  query[Person].usingTimestamp(99).update(_.age -> 10)
}
ctx.run(q2)
// UPDATE Person USING TIMESTAMP 99 SET age = 10
```

### usingTtl
```scala
val q1 = quote {
  query[Person].insert(_.age -> 10, _.name -> "John").usingTtl(11)
}
ctx.run(q1)
// INSERT INTO Person (age,name) VALUES (10, 'John') USING TTL 11

val q2 = quote {
  query[Person].usingTtl(11).update(_.age -> 10)
}
ctx.run(q2)
// UPDATE Person USING TTL 11 SET age = 10

val q3 = quote {
  query[Person].usingTtl(11).filter(_.name == "John").delete
}
ctx.run(q3)
// DELETE FROM Person USING TTL 11 WHERE name = 'John'
```

### using
```scala
val q1 = quote {
  query[Person].insert(_.age -> 10, _.name -> "John").using(ts = 99, ttl = 11)
}
ctx.run(q1)
// INSERT INTO Person (age,name) VALUES (10, 'John') USING TIMESTAMP 99 AND TTL 11

val q2 = quote {
  query[Person].using(ts = 99, ttl = 11).update(_.age -> 10)
}
ctx.run(q2)
// UPDATE Person USING TIMESTAMP 99 AND TTL 11 SET age = 10

val q3 = quote {
  query[Person].using(ts = 99, ttl = 11).filter(_.name == "John").delete
}
ctx.run(q3)
// DELETE FROM Person USING TIMESTAMP 99 AND TTL 11 WHERE name = 'John'
```

### ifCond
```scala
val q1 = quote {
  query[Person].update(_.age -> 10).ifCond(_.name == "John")
}
ctx.run(q1)
// UPDATE Person SET age = 10 IF name = 'John'

val q2 = quote {
  query[Person].filter(_.name == "John").delete.ifCond(_.age == 10)
}
ctx.run(q2)
// DELETE FROM Person WHERE name = 'John' IF age = 10
```

### delete column
```scala
val q = quote {
  query[Person].map(p => p.age).delete
}
ctx.run(q)
// DELETE p.age FROM Person
```

### list.contains / set.contains
requires `allowFiltering`
```scala
val q = quote {
  query[Book].filter(p => p.pages.contains(25)).allowFiltering
}
ctx.run(q)
// SELECT id, notes, pages, history FROM Book WHERE pages CONTAINS 25 ALLOW FILTERING
```

### map.contains
requires `allowFiltering`
```scala
val q = quote {
  query[Book].filter(p => p.history.contains(12)).allowFiltering
}
ctx.run(q)
// SELECT id, notes, pages, history FROM book WHERE history CONTAINS 12 ALLOW FILTERING
```

### map.containsValue
requires `allowFiltering`
```scala
val q = quote {
  query[Book].filter(p => p.history.containsValue(true)).allowFiltering
}
ctx.run(q)
// SELECT id, notes, pages, history FROM book WHERE history CONTAINS true ALLOW FILTERING
```

## Dynamic queries

Quill's default operation mode is compile-time, but there are queries that have their structure defined only at runtime. Quill automatically falls back to runtime normalization and query generation if the query's structure is not static. Example:

```scala
val ctx = new SqlMirrorContext(MirrorSqlDialect, Literal)

sealed trait QueryType
case object Minor extends QueryType
case object Senior extends QueryType

def people(t: QueryType): Quoted[Query[Person]] =
  t match {
    case Minor => quote {
      query[Person].filter(p => p.age < 18)
    }
    case Senior => quote {
      query[Person].filter(p => p.age > 65)
    }
  }

ctx.run(people(Minor))
// SELECT p.id, p.name, p.age FROM Person p WHERE p.age < 18

ctx.run(people(Senior))
// SELECT p.id, p.name, p.age FROM Person p WHERE p.age > 65
```

### Dynamic query API

Additionally, Quill provides a separate query API to facilitate the creation of dynamic queries. This API allows users to easily manipulate quoted values instead of working only with quoted transformations.

**Important**: A few of the dynamic query methods accept runtime string values. It's important to keep in mind that these methods could be a vector for SQL injection.

Let's use the `filter` transformation as an example. In the regular API, this method has no implementation since it's an abstract member of a trait:

```
def filter(f: T => Boolean): EntityQuery[T]
```

In the dynamic API, `filter` is has a different signature and a body that is executed at runtime:

```
def filter(f: Quoted[T] => Quoted[Boolean]): DynamicQuery[T] =
  transform(f, Filter)
```

It takes a `Quoted[T]` as input and produces a `Quoted[Boolean]`. The user is free to use regular scala code within the transformation:

```scala
def people(onlyMinors: Boolean) =
  dynamicQuery[Person].filter(p => if(onlyMinors) quote(p.age < 18) else quote(true))
```

In order to create a dynamic query, use one of the following methods:

```scala
dynamicQuery[Person]
dynamicQuerySchema[Person]("people", alias(_.name, "pname"))
```

It's also possible to transform a `Quoted` into a dynamic query:

```scala
val q = quote {
  query[Person]
}
q.dynamic.filter(p => quote(p.name == "John"))
```

The dynamic query API is very similar to the regular API but has a few differences:

**Queries**
```scala
// schema queries use `alias` instead of tuples
dynamicQuerySchema[Person]("people", alias(_.name, "pname"))

// this allows users to use a dynamic list of aliases
val aliases = List(alias[Person](_.name, "pname"), alias[Person](_.age, "page"))
dynamicQuerySchema[Person]("people", aliases:_*)

// a few methods have an overload with the `Opt` suffix,
// which apply the transformation only if the option is defined:

def people(minAge: Option[Int]) =
  dynamicQuery[Person].filterOpt(minAge)((person, minAge) => quote(person.age >= minAge))

def people(maxRecords: Option[Int]) =
  dynamicQuery[Person].takeOpt(maxRecords)

def people(dropFirst: Option[Int]) =
  dynamicQuery[Person].dropOpt(dropFirst)

// method with `If` suffix, for better chaining
def people(userIds: Seq[Int]) =
  dynamicQuery[Person].filterIf(userIds.nonEmpty)(person => quote(liftQuery(userIds).contains(person.id)))
```

**Actions**
```scala
// actions use `set`
dynamicQuery[Person].filter(_.id == 1).update(set(_.name, quote("John")))

// or `setValue` if the value is not quoted
dynamicQuery[Person].insert(setValue(_.name, "John"))

// or `setOpt` that will be applied only the option is defined
dynamicQuery[Person].insert(setOpt(_.name, Some("John")))

// it's also possible to use a runtime string value as the column name
dynamicQuery[Person].filter(_.id == 1).update(set("name", quote("John")))

// to insert or update a case class instance, use `insertValue`/`updateValue`
val p = Person(0, "John", 21)
dynamicQuery[Person].insertValue(p)
dynamicQuery[Person].filter(_.id == 1).updateValue(p)
```

### Dynamic query normalization cache

Quill is super fast for static queries (almost zero runtime overhead compared to directly sql executing).

But there is significant impact for dynamic queries.

Normalization caching was introduced to improve the situation, which will speedup dynamic queries significantly. It is enabled by default.

To disable dynamic normalization caching, pass following property to sbt during compile time

```
sbt -Dquill.query.cacheDynamic=false
```

---

## Introduction to ZIO Redis

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-redis/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-redis_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-redis_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-redis_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-redis_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-redis-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-redis-docs_2.13) [![ZIO Redis](https://img.shields.io/github/stars/zio/zio-redis?style=social)](https://github.com/zio/zio-redis)

## Introduction

[ZIO Redis](https://github.com/zio/zio-redis) is a ZIO-native Redis client.
It aims to provide a **type-safe** and **performant** API for accessing Redis
instances.

## Installation

To use ZIO Redis, add the following line to your `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-redis" % "1.1.10"
```

## Example

To execute our ZIO Redis effect, we should provide the `RedisExecutor` layer to that effect. To create this layer we
should also provide the following layers:

- **RedisConfig** — Using default one, will connect to the `localhost:6379` Redis instance.
- **BinaryCodec** — In this example, we are going to use the built-in `ProtobufCodec` codec from zio-schema project.

To run this example we should put following dependencies in our `build.sbt` file:

```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-redis" % "1.1.10",
  "dev.zio" %% "zio-schema-protobuf" % "1.6.1"
)
```

```scala

object ZIORedisExample extends ZIOAppDefault {
  
  object ProtobufCodecSupplier extends CodecSupplier {
    def get[A: Schema]: BinaryCodec[A] = ProtobufCodec.protobufCodec
  }
  
  val myApp: ZIO[Redis, RedisError, Unit] = 
    for {
      redis <- ZIO.service[Redis]
      _     <- redis.set("myKey", 8L, Some(1.minutes))
      v     <- redis.get("myKey").returning[Long]
      _     <- Console.printLine(s"Value of myKey: $v").orDie
      _     <- redis.hSet("myHash", ("k1", 6), ("k2", 2))
      _     <- redis.rPush("myList", 1, 2, 3, 4)
      _     <- redis.sAdd("mySet", "a", "b", "a", "c")
    } yield ()

  override def run = 
    myApp.provide(Redis.local, ZLayer.succeed[CodecSupplier](ProtobufCodecSupplier))
}
```

## Testing

To test you can use the embedded redis instance by adding to your build:

```scala
libraryDependencies := "dev.zio" %% "zio-redis-embedded" % "1.1.10"
```

Then you can supply `EmbeddedRedis.layer.orDie` as your `RedisConfig` and you're good to go!

```scala

object EmbeddedRedisSpec extends ZIOSpecDefault {
  object ProtobufCodecSupplier extends CodecSupplier {
    def get[A: Schema]: BinaryCodec[A] = ProtobufCodec.protobufCodec
  }
  
  final case class Item private (id: UUID, name: String, quantity: Int)
  object Item {
    implicit val itemSchema: Schema[Item] = DeriveSchema.gen[Item]
  }
  
  def spec = suite("EmbeddedRedis should")(
    test("set and get values") {
      for {
        redis <- ZIO.service[Redis]
        item   = Item(UUID.randomUUID, "foo", 2)
        _     <- redis.set(s"item.${item.id.toString}", item)
        found <- redis.get(s"item.${item.id.toString}").returning[Item]
      } yield assert(found)(isSome(equalTo(item)))
    }
  ).provideShared(
    EmbeddedRedis.layer,
    ZLayer.succeed[CodecSupplier](ProtobufCodecSupplier),
    Redis.singleNode
  ) @@ TestAspect.silentLogging
}
```

## Resources

- [ZIO Redis](https://www.youtube.com/watch?v=yqFt3b3RBkI) by Dejan Mijic — Redis is one of the most commonly used
  in-memory data structure stores. In this talk, Dejan will introduce ZIO Redis, a purely functional, strongly typed
  client library backed by ZIO, with excellent performance and extensive support for nearly all of Redis' features. He
  will explain the library design using the bottom-up approach - from communication protocol to public APIs. Finally, he
  will wrap the talk by demonstrating the client's usage and discussing its performance characteristics.

---

## Introduction to ZIO RocksDB

A ZIO-based interface to RocksDB.

## Installation

Add the following dependencies to your `build.sbt` file:
```scala
libraryDependencies ++= Seq(
  "dev.zio" %% "zio-rocksdb" % "0.4.2"
)
```

## Using RocksDB

Use the provided `RocksDB` wrapper:

```scala

val UTF_8 = StandardCharsets.UTF_8
val key   = "key".getBytes(UTF_8)
val value = "value".getBytes(UTF_8)

val database  = RocksDB.live("/data/state")
val readWrite = RocksDB.put(key, value) *> RocksDB.get(key)
val result    = readWrite.provideCustomLayer(database)
```

## Using TransactionDB

`zio-rocksdb` provides transactional capabilities using RocksDB [Transaction API].

```scala

val key0   = "key0".getBytes(UTF_8)
val key1   = "key1".getBytes(UTF_8)
val value0 = "value0".getBytes(UTF_8)
val value1 = "value1".getBytes(UTF_8)

val database      = TransactionDB.live("/data/state")
val write0        = Transaction.put(key0, value0)
val write1        = Transaction.put(key1, value1)
val writeTogether = TransactionDB.atomically {
  write0 <&> write1
}
val result        = readWrite.provideCustomLayer(database)
```

[Transaction API]: https://github.com/facebook/rocksdb/wiki/Transactions#transactiondb

---

## Introduction to ZIO S3

Thin wrapper over S3 async client for ZIO

Setup
-----

```
//support scala 2.12 / 2.13

libraryDependencies += "dev.zio" %% "zio-s3" % "0.4.2.1"
```

How to use it ?
---------------

ZIO-S3 is a thin wrapper over the s3 async java client. It exposes the main operations of the s3 java client.

```scala

  // list all buckets available  
  listBuckets.provideLayer(
     live("us-east-1", AwsBasicCredentials.create("accessKeyId", "secretAccessKey"))
  )
  
  // list all objects of all buckets
  val l2: ZStream[S3, S3Exception, String] = (for {
     bucket <- ZStream.fromIterableZIO(listBuckets) 
     obj <- listAllObjects(bucket.name)
  } yield obj.bucketName + "/" + obj.key).provideLayer(
     live("us-east-1", AwsBasicCredentials.create("accessKeyId", "secretAccessKey"))
  )  
```

All available s3 combinators and operations are available in the package object `zio.s3`, you only need to `import zio.s3._`

Credentials
-----------

zio-s3 expose credentials providers from aws https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/credentials.html
If credentials cannot be found in one or multiple providers selected the operation will fail with `InvalidCredentials`

```scala

// build S3 Layer from basic credentials
val s3: Layer[S3Exception, S3] =
  live(Region.AF_SOUTH_1, AwsBasicCredentials.create("key", "secret"))

// build S3 Layer from System properties or Environment variables
val s3: Layer[S3Exception, S3] =
  liveZIO(Region.AF_SOUTH_1, system <> env)

// build S3 Layer  from Instance profile credentials
val s3: Layer[S3Exception, S3] =
  liveZIO(Region.AF_SOUTH_1, instanceProfile)

// build S3 Layer from web identity token credentials with STS. awssdk sts module required to be on classpath
val s3: Layer[S3Exception, S3] = liveZIO(Region.AF_SOUTH_1, webIdentity)

// build S3 Layer from default available credentials providers
val s3: Layer[S3Exception, S3] = liveZIO(Region.AF_SOUTH_1, default)

// use custom logic to fetch aws credentials
val zcredentials: ZIO[R, S3Exception, AwsCredentials] = ??? // specific implementation to fetch credentials
val s3: ZLayer[Any, S3Exception, S3] = settings(Region.AF_SOUTH_1, zcredentials) >>> live

```

Test / Stub
-----------

a stub implementation of s3 storage is provided for testing purpose and use internally a filesystem to simulate s3 storage

```scala

// build s3 Layer
val stubS3: ZLayer[Any, Nothing, S3] = stub(ZPath("/tmp/s3-data"))

// list all buckets available by using S3 Stub Layer 
// will list all directories of `/tmp/s3-data`
listBuckets.provideLayer(stubS3) 
```

More information here on how to use [ZLayer https://zio.dev/docs/howto/howto_use_layers](https://zio.dev/docs/howto/howto_use_layers)

Examples
--------

```scala

// upload
val json: Chunk[Byte] = Chunk.fromArray("""{  "id" : 1 , "name" : "A1" }""".getBytes)
val up: ZIO[S3, S3Exception, Unit] = putObject(
  "bucket-1",
  "user.json",
  json.length,
  ZStream.fromChunk(json),
  UploadOptions.fromContentType("application/json")
)

// multipartUpload 

val is = ZStream.fromInputStream(new FileInputStream(Paths.get("/my/path/to/myfile.zip").toFile))
val proc2: ZIO[S3, S3Exception, Unit] =
  multipartUpload(
    "bucket-1",
    "upload/myfile.zip",
    is,
    MultipartUploadOptions.fromUploadOptions(UploadOptions.fromContentType("application/zip"))
  )(4)

// download

val os: OutputStream = ???
val proc3: ZIO[S3, Exception, Long] = getObject("bucket-1", "upload/myfile.zip").run(ZSink.fromOutputStream(os))
```

Support any commands ?
---

If you need a method which is not wrapped by the library, you can have access to underlying S3 client in a safe manner by using

```scala

def execute[T](f: S3AsyncClient => CompletableFuture[T]) 
```

---

## Developer Notes

## Todo List

### ZIO SBT Website

- Add a command to make a version history page in documentation
- Check if the committed workflow is valid or not.
- To generate README.md file, it doesn't need to compile all docs.
- Add env variable for zio version (ZIO_VERSION).
- Support for various scala versions
- Support for auto update PRs
- Add workflow for header checker
- Add prepare task to perform preliminary tasks before PR
- Parametrize "docs" project instead of hardcoded value

---

## ZIO SBT

_ZIO SBT_ contains multiple sbt plugins that are useful for ZIO projects. It provides high-level SBT utilities that simplify the development of ZIO applications.

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-sbt/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-sbt-website_2.12.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-sbt-website_2.12/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-sbt-website_2.12.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-sbt-website_2.12/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-sbt-docs_2.12/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-sbt-docs_2.12) [![ZIO SBT](https://img.shields.io/github/stars/zio/zio-sbt?style=social)](https://github.com/zio/zio-sbt)

## Installation

Add the following lines to your `project/plugins.sbt` file:

```scala
addSbtPlugin("dev.zio" % "zio-sbt-ecosystem" % "0.5.3")
addSbtPlugin("dev.zio" % "zio-sbt-ci"        % "0.5.3")
addSbtPlugin("dev.zio" % "zio-sbt-website"   % "0.5.3")
```

Then you can enable them by using the following code in your `build.sbt` file:

```scala
enablePlugins(
  ZioSbtWebsitePlugin,
  ZioSbtEcosystemPlugin,
  ZioSbtCiPlugin
)
```

:::note
Always try to keep the SBT version specified in the `project/build.properties` file up to date to ensure compatibility with the ZIO SBT plugins.
:::

## ZIO SBT Ecosystem

ZIO SBT Ecosystem plugin is an sbt plugin that provides a set of sbt settings and tasks that are very common and useful for configuring and managing ZIO projects. It is designed help developers to quickly set up a new ZIO project with a minimal amount of effort.

This pluging provides the following settings with default values:

- scala212
- scala213
- scala3

The default values are the latest stable versions of Scala 2.12, 2.13, and Scala 3. All of these settings are of type `String` and can be overridden by the user.

By having these settings, then we can use them in other sbt settings. For example, we can use them to define the `crossScalaVersions` setting:

```scala
crossScalaVersions := Seq(scala212.value, scala213.value, scala3.value)
```

There are also some other settings that are useful for configuring the projects:

- `stdSettings`— a set of standard settings which are common for every ZIO project, which includes configuring:
  - silencer plugin
  - kind projector plugin
  - cross project plugin
  - scalafix plugin
  - java target platform
- `enableZIO`- a set of ZIO related settings such as enabling zio streams and ZIO test framework.
- `jsSettings`, `nativeSettings`- common platform specific settings for Scala.js and Scala Native.

It also provides some helper methods that are useful for configuring a compiler option for a specific Scala version:

- `optionsOn`
- `optionsOnExcept`
- `optionsOnOrElse`
- `addOptionsOn`
- `addOptionsOnOrElse`
- `addOptionsOnExcept`

And the same for adding a dependency for a specific Scala version:

- `dependenciesOn`
- `dependenciesOnExcept`
- `dependenciesOnOrElse`
- `addDependenciesOn`
- `addDependenciesOnExcept`
- `addDependenciesOnOrElse`

## ZIO SBT Website

ZIO SBT Website is an SBT plugin that has the following tasks:

- `sbt compileDocs`— compile documentation inside `docs` directory. The compilation result will be inside `website/docs` directory.
- `sbt installWebsite`— creates a website for the project inside the `website` directory.
- `sbt previewWebsite`— runs a local webserver that serves documentation locally on http://localhost:3000. By changing the documentation inside the `docs` directory, the website will be reloaded with new content.
- `sbt publishToNpm`— publishes documentation inside the `docs` directory to the npm registry.
- `sbt generateReadme`— generate README.md file from `docs/index.md` and sbt setting keys.

## ZIO SBT CI Plugin

ZIO SBT CI is an sbt plugin which generates a GitHub workflow for a project, making it easier to set up continuous integration (CI) pipelines for Scala projects. With this plugin, developers can streamline their development workflow by automating the testing and deployment process, reducing manual effort and errors. The plugin is designed to work seamlessly with sbt, the popular build tool for Scala projects, and integrates smoothly with GitHub Actions, the CI/CD platform provided by GitHub.

ZIO SBT CI provides a simple and efficient way to configure, manage, and run CI pipelines, helping teams to deliver high-quality software faster and with greater confidence.

ZIO SBT CI plugin generates a default GitHub workflow that includes common CI tasks such as building, testing, and publishing artifacts. However, users can also manually customize the workflow. This plugin is designed to be flexible and extensible, making it easy for users to tailor the workflow to their specific needs. Additionally, the plugin also provides tons of optional sbt settings that users can modify to change various aspects of the generated workflow. Overall, ZIO SBT CI plugin strikes a balance between automation and flexibility, allowing users to automate their CI process while still giving them control over how the workflow is generated.

### Getting Started

To use ZIO SBT CI plugin, add the following lines to your `plugins.sbt` file:

```scala
addSbtPlugin("dev.zio" % "zio-sbt-ci" % "0.5.3")

resolvers ++= Resolver.sonatypeOssRepos("public")
```

Then in your `build.sbt` file, enable the plugin by adding the following line:

```scala
enablePlugins(ZioSbtCiPlugin)
```

Now you can generate a Github workflow by running the following command:

```bash
sbt ciGenerateGithubWorkflow
```

This will generate a GitHub workflow file inside the `.github/workflows` directory, named `ci.yml`. The workflow file contains following default Jobs:

- Build
- Lint
- Test
- Update Readme

> **Note:**
> 
> To use this plugin, we also need to install [ZIO Assistant](https://github.com/apps/zio-assistant) bot.

## ZIO SBT GitHub Query Plugin

ZIO SBT GitHub Query is an sbt plugin for fetching GitHub issues/PRs and building a searchable SQLite database with full-text search.

### Installation

Add to `plugins.sbt`:

```scala
addSbtPlugin("dev.zio" % "zio-sbt-gh-query" % "0.5.3")
```

The plugin is auto-enabled. Configure in `build.sbt`:

```scala
// Required: specify your GitHub repository
ghRepo := "your-org/your-repo"

// Optional: override the default data directory (defaults to .zio-sbt)
ghDir := file(".zio-sbt")
```

### Commands

| Command | Description |
|---------|-------------|
| `gh-sync` | Fetch data from GitHub and build/update the search database. On first run (or with `--force`), does a full fetch and rebuild. On subsequent runs, fetches only new/updated items incrementally. |
| `gh-query <query>` | Full-text search across issues and PRs. Supports `--verbose` flag to include body text. |
| `gh-status` | Show database statistics (issue/PR/comment counts, last fetch time). |

### Usage

```bash
# Fetch all issues/PRs and build the database
sbt gh-sync

# Incrementally fetch new/updated items
sbt gh-sync

# Re-fetch everything and rebuild from scratch
sbt "gh-sync --force"

# Basic search query
sbt "gh-query codec"

# Search with full body content
sbt "gh-query --verbose codec"

# Check database statistics
sbt gh-status
```

### Dependencies

The plugin checks for all required dependencies before running any command and reports clear
error messages with install instructions if anything is missing.

| Dependency | Required by | Install |
|---|---|---|
| `bash` | `gh-sync` | https://www.gnu.org/software/bash/ |
| `gh` (GitHub CLI) | `gh-sync` | https://cli.github.com |
| `jq` | `gh-sync` | https://jqlang.github.io/jq/download/ |
| `python3` | all commands | https://www.python.org/downloads/ |
| `sqlite3` with [FTS5](https://www.sqlite.org/fts5.html) | `gh-sync`, `gh-query` | Ensure your Python's sqlite3 is built with FTS5 support |

Before running `gh-sync` for the first time, authenticate the GitHub CLI:

```bash
gh auth login
```

The authenticated account must have read access to the target repository.

### Database Schema

The plugin creates a SQLite database with:

- `issues` table - stores issues and PRs
- `comments` table - stores issue and PR comments
- `search_index` - FTS5 full-text search index (requires SQLite built with FTS5 enabled)

## ZIO SBT Source

ZIO SBT Source is a Scala 2.13 + Scala 3 cross-compiled library that provides utilities for self-documenting example code. It includes the `ExprEval` macro, which captures the source text of expressions at compile time and prints them alongside their evaluated results at runtime.

### Installation

Add the following line to your `libraryDependencies` in `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-sbt-source" % "0.5.3"
```

### Features

The `ExprEval.show` macro provides an intuitive way to write self-documenting example code:

```scala

// Print expression source and result
show(
  42 + 8,
  "Hello, " + "World!"
)
```

Output:
```
42 + 8
// 50
"Hello, " + "World!"
// Hello, World!
```

### Comment Labels

Add `//` comments above your `show` call to add context labels to the output:

```scala
// Calculate the answer to life, the universe, and everything
show(6 * 7)
```

Output:
```
// Calculate the answer to life, the universe, and everything
6 * 7
// 42
```

### Block Form

For multiple expressions, use the block form:

```scala
show {
  1 + 2
  3 * 4
  "hello"
}
```

### Implementation Details

- **Scala 3**: Uses `scala.quoted.*` with `inline def` for compile-time source capture
- **Scala 2.13**: Uses `scala.reflect.macros.whitebox` to achieve the same behavior
- **Runtime Helper**: `SourceReader` utility reads comment lines from source files

## Testing Strategies

### Default Testing Strategy

The default testing strategy for ZIO SBT CI plugin is to run `sbt +test` on Corretto Java 11, 17 and 21. So this will generate the following job:

```yaml
test:
  name: Test
  runs-on: ubuntu-latest
  continue-on-error: false
  strategy:
    fail-fast: false
    matrix:
      java: ['11', '17', '21']
  steps:
  - name: Install libuv
    run: sudo apt-get update && sudo apt-get install -y libuv1-dev
  - name: Setup Scala
    uses: actions/setup-java@v3.13.0
    with:
      distribution: corretto
      java-version: ${{ matrix.java }}
      check-latest: true
  - name: Cache Dependencies
    uses: coursier/cache-action@v6
  - name: Git Checkout
    uses: actions/checkout@v6
    with:
      fetch-depth: '0'
  - name: Test
    run: sbt +test
```

The `sbt +test` command will run the `test` task for all submodules in the project against all Scala versions defined in the `crossScalaVersions` setting.

### Concurrent Testing Strategy

In some cases, we may have multiple submodules in our project and we want to test them concurrently using GitHub Actions matrix strategy.

The `ciTargetScalaVersions` setting key is used to define a mapping of project names to the Scala versions that should be used for testing phase of continuous integration (CI).

For example, suppose we have a project with the name "submoduleA" and we want to test it against Scala `2.12.20`, and for the "submoduleB" we want to test it against Scala `2.12.20` and `2.13.18` and `3.3.7`, We can define the `ciTargetScalaVersions` setting as follows:

```scala
ThisBuild / ciTargetScalaVersions := Map(
    "submoduleA" -> Seq("2.12.20"),
    "submoduleB" -> Seq("2.12.20", "2.13.18", "3.3.7")
  )
```

In the example provided, `ciTargetScalaVersions` is defined at the `ThisBuild` level, meaning that the setting will apply to all projects within the build. The setting defines a Map where the key is the name of the current project, obtained by calling the `id` method on the `thisProject` setting, and the value is a sequence of Scala versions obtained from the `crossScalaVersions` of each submodule setting.

To simplify this process, we can populate the versions using each submodule's crossScalaVersions setting as follows:

```scala
ThisBuild / ciTargetScalaVersions := Map(
  (submoduleA / thisProject).value.id -> (submoduleA / crossScalaVersions).value,
  (submoduleB / thisProject).value.id -> (submoduleB / crossScalaVersions).value
)
```

The above code can be simplified further by using `targetScalaVersionsFor` helper method, it takes a list of submodules and returns a Map of project names to their `crossScalaVersions`:

```scala
ThisBuild / ciTargetScalaVersions := targetScalaVersionsFor(submoduleA, submoduleB).value
```

This will generate the following job:

```yaml
test:
  name: Test
  runs-on: ubuntu-latest
  continue-on-error: false
  strategy:
    fail-fast: false
    matrix:
      java: ['11', '17', '21']
      scala-project:
      - ++2.12.20 submoduleA
      - ++2.12.20 submoduleB
      - ++2.13.18 submoduleB
      - ++3.3.7 submoduleB
  steps:
  - name: Install libuv
    run: sudo apt-get update && sudo apt-get install -y libuv1-dev
  - name: Setup Scala
    uses: actions/setup-java@v3.10.0
    with:
      distribution: corretto
      java-version: ${{ matrix.java }}
      check-latest: true
  - name: Cache Dependencies
    uses: coursier/cache-action@v6
  - name: Git Checkout
    uses: actions/checkout@v6
    with:
      fetch-depth: '0'
  - name: Test
    run: sbt ${{ matrix.scala-project }}/test
```

---

## Automatic Schema Derivation

Automatic schema derivation is the process of generating schema definitions for data types automatically, without the need to manually write them. It allows us to generate the schema for a data type based on its structure and annotations.

Instead of manually specifying the schema for each data type, we can rely on automatic schema derivation to generate the schema for us. This approach can save time and reduce the potential for errors, especially when dealing with complex data models.

By leveraging reflection and type introspection using macros, automatic schema derivation analyzes the structure of the data type and its fields, including their names, types, and annotations. It then generates the corresponding schema definition based on this analysis.

ZIO streamlines schema derivation through its `zio-schema-derivation` package, which utilizes the capabilities of Scala macros to automatically derive schemas. In order to use automatic schema derivation, we neeed to add the following line to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-derivation" % 1.7.5
```

Once again, let's revisit our domain models:

```scala
final case class Person(name: String, age: Int)

sealed trait PaymentMethod

object PaymentMethod {
  final case class CreditCard(number: String, expirationMonth: Int, expirationYear: Int) extends PaymentMethod
  final case class WireTransfer(accountNumber: String, bankCode: String) extends PaymentMethod
}

final case class Customer(person: Person, paymentMethod: PaymentMethod)
```

We can easily use auto derivation to create schemas:

```scala

final case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
}

sealed trait PaymentMethod

object PaymentMethod {

  implicit val schema: Schema[PaymentMethod] =
    DeriveSchema.gen[PaymentMethod]

  final case class CreditCard(
      number: String,
      expirationMonth: Int,
      expirationYear: Int
  ) extends PaymentMethod

  final case class WireTransfer(accountNumber: String, bankCode: String)
      extends PaymentMethod
}

final case class Customer(person: Person, paymentMethod: PaymentMethod)

object Customer {
  implicit val schema: Schema[Customer] = DeriveSchema.gen[Customer]
}
```

Now we can write an example that demonstrates a roundtrip test for protobuf codecs:

```scala
// Create a customer instance
val customer =
  Customer(
    person = Person("John Doe", 42),
    paymentMethod = PaymentMethod.CreditCard("1000100010001000", 6, 2024)
  )

// Create binary codec from customer 
val customerCodec: BinaryCodec[Customer] =
  ProtobufCodec.protobufCodec[Customer]

// Encode the customer object
val encodedCustomer: Chunk[Byte] = customerCodec.encode(customer)

// Decode the byte array back to the person instance
val decodedCustomer: Either[DecodeError, Customer] =
  customerCodec.decode(encodedCustomer)

assert(Right(customer) == decodedCustomer)
```

---

## Basic Building Blocks

To get started, first we need to understand that a ZIO Schema is basically built-up from these three
sealed traits: `Record[R]`, `Enum[A]` and `Sequence[Col, Elem]`, along with the case class `Primitive[A]`. Every other type is just a specialisation of one of these (or not relevant to get you started).

The core data type of ZIO Schema is a `Schema[A]` which is **invariant in `A`** by necessity, because a Schema allows us to derive operations that produce an `A` but also operations that consume an `A` and that imposes limitations on the types of **transformation operators** and **composition operators** that we can provide based on a `Schema`.

It looks kind of like this (simplified):

```scala
sealed trait Schema[A] { self =>
  def zip[B](that: Schema[B]): Schema[(A, B)]

  def transform[B](f: A => B, g: B => A): Schema[B]
}
```

## Primitives

To describe scalar data type `A`, we use the `Primitive[A]` data type which basically is a wrapper around `StandardType`:

```scala
case class Primitive[A](standardType: StandardType[A]) extends Schema[A]
```

Primitive values are represented using the `Primitive[A]` type class and represent the elements that we cannot further define through other means. If we visualize our data structure as a tree, primitives are the leaves.

For a list of all standard types (and therefore primitive types) with built-in support, please see the [standard type reference](standard-type-reference.md)

Inside `Schema`'s companion object, we have an implicit conversion from `StandardType[A]` to `Schema[A]`:

```scala
object Schema {
   implicit def primitive[A](implicit standardType: StandardType[A]): Schema[A] = ???
}
```

So we can easily create a `Schema` for a primitive type `A` either by calling `Schema.primitive[A]` or by calling `Schema.apply[A]`:

```scala
val intSchema1: Schema[Int] = Schema[Int]
val intSchema2: Schema[Int] = Schema.primitive[Int] 
```

## Fail

To represents the absence of schema information for the given `A` type, we can use `Schema.fail` constructor, which creates the following schema:

```scala
object Schema {
  case class Fail[A](
    message: String,
    annotations: Chunk[Any] = Chunk.empty
  ) extends Schema[A]
}
```

## Collections

### Sequence

Often we have a type that is a collection of elements. For example, we might have a `List[User]`. This is called a `Sequence` and is represented using the `Sequence[Col, Elem, I]` type class:

```scala
object Schema {
  sealed trait Collection[Col, Elem] extends Schema[Col]
  
  final case class Sequence[Col, Elem, I](
      elementSchema: Schema[Elem],
      fromChunk: Chunk[Elem] => Col,
      toChunk: Col => Chunk[Elem],
      override val annotations: Chunk[Any] = Chunk.empty,
      identity: I
    ) extends Collection[Col, Elem]
}
```

The `Sequence` can be anything that can be isomorphic to a list. 

Here is an example schema for list of `Person`s:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
}

val personListSchema: Schema[List[Person]] =
  Sequence[List[Person], Person, String](
    elementSchema = Schema[Person],
    fromChunk = _.toList,
    toChunk = i => Chunk.fromIterable(i),
    annotations = Chunk.empty,
    identity = "List"
  )
```

ZIO Schema has `Schema.list[A]`, `Schema.chunk[A]` and `Schema.vector[A]` constructors that create `Schema[List[A]]`, `Schema[Chunk[A]]` and `Schema[Vector[A]]` for us:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
  
  implicit val listSchema:   Schema[List[Person]]   = Schema.list[Person]
  implicit val chunkSchema:  Schema[Chunk[Person]]  = Schema.chunk[Person]
  implicit val vectorSchema: Schema[Vector[Person]] = Schema.vector[Person]
}
```

### Map

Likewise, we can have a type that is a map of keys to values. ZIO Schema represents this using the following type class:

```scala
object Schema {
  sealed trait Collection[Col, Elem] extends Schema[Col]

  case class Map[K, V](
    keySchema: Schema[K],
    valueSchema: Schema[V],
    override val annotations: Chunk[Any] = Chunk.empty
  ) extends Collection[scala.collection.immutable.Map[K, V], (K, V)]
}
```

It stores the key and value schemas. Like `Sequence`, instead of using `Map` directly, we can use the `Schema.map[K, V]` constructor:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
  
  implicit val mapSchema: Schema[scala.collection.immutable.Map[String, Person]] = 
    Schema.map[String, Person]
}
```

### Set

The `Set` type class is similar to `Sequence` and `Map`. It is used to represent a schema for a set of elements:

```scala
object Schema {
  sealed trait Collection[Col, Elem] extends Schema[Col]

  case class Set[A](
    elementSchema: Schema[A],
    override val annotations: Chunk[Any] = Chunk.empty
  ) extends Collection[scala.collection.immutable.Set[A], A]
}
```

To create a `Schema` for a `Set[A]`, we can use the above type class directly or use the `Schema.set[A]` constructor:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
  
  implicit val setSchema: Schema[scala.collection.immutable.Set[Person]] = 
    Schema.set[Person]
}
```

## Records

Our data structures usually are composed of a lot of types. For example, we might have a `User` type that has a `name` field, an `age` field, an `address` field, and a `friends` field.

```scala
case class User(name: String, age: Int, address: Address, friends: List[User])
```

This is called a **product type** in functional programming. The equivalent of a product type in ZIO Schema is called a record.

In ZIO Schema such a record would be represented using the `Record[R]` typeclass:

```scala
object Schema {
  sealed trait Field[R, A] {
    type Field <: Singleton with String
    def name: Field
    def schema: Schema[A]
  }

  sealed trait Record[R] extends Schema[R] {
    def id: TypeId
    def fields: Chunk[Field[_]]
    def construct(fieldValues: Chunk[Any]): Either[String, R]
  }
}
```

ZIO Schema has specialized record types for case classes, called `CaseClass1[A, Z]`, `CaseClass2[A1, A2, Z]`, ..., `CaseClass22`. Here is the definition of `apply` method of `CaseClass1` and `CaseClass2`:

```scala
sealed trait CaseClass1[A, Z] extends Record[Z]

object CaseClass1 {
  def apply[A, Z](
    id0: TypeId,
    field0: Field[Z, A],
    defaultConstruct0: A => Z,
    annotations0: Chunk[Any] = Chunk.empty
  ): CaseClass1[A, Z] = ???
}

object CaseClass2 {
  def apply[A1, A2, Z](
    id0: TypeId,
    field01: Field[Z, A1],
    field02: Field[Z, A2],
    construct0: (A1, A2) => Z,
    annotations0: Chunk[Any] = Chunk.empty
  ): CaseClass2[A1, A2, Z] = ???
}
```

As we can see, they take a `TypeId`, a number of fields of type `Field`, and a construct function. The `TypeId` is used to uniquely identify the type. The `Field` is used to store the name of the field and the schema of the field. The `construct` is used to construct the type from the field values.

Here is an example of defining schema for `Person` data type:

```scala

final case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    Schema.CaseClass2[String, Int, Person](
      id0 = TypeId.fromTypeName("Person"),
      field01 = 
        Schema.Field(
          name0 = "name",
          schema0 = Schema[String],
          get0 = _.name,
          set0 = (p, x) => p.copy(name = x)
        ),
      field02 = 
        Schema.Field(
          name0 = "age",
          schema0 = Schema[Int],
          get0 = _.age,
          set0 = (person, age) => person.copy(age = age)
        ),
      construct0 = (name, age) => Person(name, age),
    )
}
```

There is also the `GenericRecord` which is used to either ad-hoc records or records that have more than 22 fields:

```scala
object Schema {
  sealed case class GenericRecord(
    id: TypeId,
    fieldSet: FieldSet,
    override val annotations: Chunk[Any] = Chunk.empty
  ) extends Record[ListMap[String, _]]
}
```

## Enumerations

Other times, you might have a type that represents a list of different types. For example, we might have a type, like this:

```scala
sealed trait PaymentMethod 

object PaymentMethod {
  final case class CreditCard(number: String, expirationMonth: Int, expirationYear: Int) extends PaymentMethod
  final case class WireTransfer(accountNumber: String, bankCode: String) extends PaymentMethod
}
```

In functional programming, this kind of type is called a **sum type**:
- In Scala 2, this is called a **sealed trait**.
- In Scala3, this is called an **enum**.

In ZIO Schema we call these types `enumeration` types, and they are represented using the `Enum[A]` type class.

```scala
object Schema {
  sealed trait Enum[Z] extends Schema[Z]
}
```

It has specialized types `Enum1[A, Z]`, `Enum2[A1, A2, Z]`, ..., `Enum22[A1, A2, ..., A22, Z]` for enumerations with 1, 2, ..., 22 cases. Here is the definition of `Enum1` and `Enum2`:

```scala 
  sealed case class Enum1[A, Z](
    id: TypeId,
    case1: Case[Z, A],
    annotations: Chunk[Any] = Chunk.empty
  ) extends Enum[Z]

  sealed case class Enum2[A1, A2, Z](
    id: TypeId,
    case1: Case[Z, A1],
    case2: Case[Z, A2],
    annotations: Chunk[Any] = Chunk.empty
  ) extends Enum[Z]

  // Enum3, Enum4, ..., Enum22
}
```

If the enumeration has more than 22 cases, we can use the `EnumN` type class:

```scala
object Schema {
  sealed case class EnumN[Z, C <: CaseSet.Aux[Z]](
    id: TypeId,
    caseSet: C,
    annotations: Chunk[Any] = Chunk.empty
  ) extends Enum[Z]
}
```

It has a simple constructor called `Schema.enumeration`:

```scala
object Schema {
  def enumeration[A, C <: CaseSet.Aux[A]](id: TypeId, caseSet: C): Schema[A] = ???
}
```

## Optionals

To create a `Schema` for optional values, we can use the `Optional` type class:

```scala
object Schema {
  case class Optional[A](
    schema: Schema[A],
    annotations: Chunk[Any] = Chunk.empty
  ) extends Schema[Option[A]]
}
```

Using the `Schema.option[A]` constructor, makes it easier to do so:

```scala
val option: Schema[Option[Person]] = Schema.option[Person]
```

## Either

Here is the same but for `Either`:

```scala
object Schema {
  case class Either[A, B](
    left: Schema[A],
    right: Schema[B],
    annotations: Chunk[Any] = Chunk.empty
  ) extends Schema[scala.util.Either[A, B]]
}
```

We can use `Schema.either[A, B]` to create a `Schema` for `scala.util.Either[A, B]`:

```scala
val eitherPersonSchema: Schema[scala.util.Either[String, Person]] = 
  Schema.either[String, Person]
```

## Tuple

Each schema has a `Schema#zip` operator that allows us to combine two schemas and create a schema for a tuple of the two types:

```scala
object Schema {
  def zip[B](that: Schema[B]): Schema[(A, B)] = 
    Schema.Tuple2(self, that)
}
```

It is implemented using the `Schema.Tuple2` type class:

```scala
object Schema {
  final case class Tuple2[A, B](
    left: Schema[A],
    right: Schema[B], 
    annotations: Chunk[Any] = Chunk.em
    pty
  ) extends Schema[(A, B)] 
}
```

ZIO Schema also provides implicit conversions for tuples of arity 2, 3, ..., 22:

```scala
object Schema {
  implicit def tuple2[A, B](implicit c1: Schema[A], c2: Schema[B]): Schema[(A, B)] =
    c1.zip(c2)

  implicit def tuple3[A, B, C](implicit c1: Schema[A], c2: Schema[B], c3: Schema[C]): Schema[(A, B, C)] =
    c1.zip(c2).zip(c3).transform({ case ((a, b), c) => (a, b, c) }, { case (a, b, c) => ((a, b), c) })

  // tuple3, tuple4, ..., tuple22
}
```

So we can easily create a `Schema` for a tuple of n elements, just by calling `Schema[(A1, A2, ..., An)]`:

```scala

val tuple2: Schema[(String, Int)]          = Schema[(String, Int)]
val tuple3: Schema[(String, Int, Boolean)] = Schema[(String, Int, Boolean)]
// ...
```

---

## Apache Avro Codecs

## Introduction

Apache Avro is a popular data serialization format used in distributed systems, particularly in the Apache Hadoop ecosystem. In this article, we will explore how to work with Apache Avro codecs in Scala using the ZIO Schema. Avro codecs allow us to easily serialize and deserialize data in Avro's binary and JSON formats.

## Installation

To use the Avro codecs, we need to add the following dependency to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-avro" % "1.7.5"
```

## Codecs

It has two codecs:

- An **AvroSchemaCodec** to serialize a `Schema[A]` to Avro JSON schema and deserialize an Avro JSON schema to a `Schema.GenericRecord`.
- An **AvroCodec** to serialize/deserialize the Avro binary serialization format.

### AvroSchemaCodec

The `AvroSchemaCodec` provides methods to encode a `Schema[_]` to Avro JSON schema and decode an Avro JSON schema to a `Schema[_]` ([`Schema.GenericRecord`](../../operations/dynamic-data-representation.md)):

```scala
trait AvroSchemaCodec {
  def encode(schema: Schema[_]): scala.util.Either[String, String]
  def decode(bytes: Chunk[Byte]): scala.util.Either[String, Schema[_]]
}
```

The `encode` method takes a `Schema[_]` and returns an `Either[String, String]` where the `Right` side contains the Avro schema in JSON‌ format.

The `decode` method takes a `Chunk[Byte]` which contains the Avro JSON Schema in binary format and returns an `Either[String, Schema[_]]` where the `Right` side contains the ZIO Schema in `GenericRecord` format.

Here is an example of how to use it:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
}

object Main extends ZIOAppDefault {
  def run =
    for {
      _          <- ZIO.debug("AvroSchemaCodec Example:")
      avroSchema <- ZIO.fromEither(AvroSchemaCodec.encode(Person.schema))
      _ <- ZIO.debug(s"The person schema in Avro Schema JSON format: $avroSchema")
      avroSchemaBinary = Chunk.fromArray(avroSchema.getBytes)
      zioSchema <- ZIO.fromEither(AvroSchemaCodec.decode(avroSchemaBinary))
      _ <- ZIO.debug(s"The person schema in ZIO Schema GenericRecord format: $zioSchema")
    } yield ()
}
```

The output:

```scala
AvroSchemaCodec Example:
The person schema in Avro Schema JSON format: {"type":"record","name":"Person","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}
The person schema in ZIO Schema GenericRecord format: GenericRecord(Nominal(Chunk(),Chunk(),Person),Field(name,Primitive(string,Chunk())) :*: Field(age,Primitive(int,Chunk())) :*: Empty,Chunk(name(Person)))
```

As we can see, we converted the `Schema[Person]` to Avro schema JSON format, and then we converted it back to the ZIO Schema `GenericRecord` format.

### AvroCodec

We can create a `BinaryCodec[A]` for any type `A` that has a `Schema[A]` instance using `AvroCodec.schemaBasedBinaryCodec`:

```scala
object AvroCodec {
  implicit def schemaBasedBinaryCodec[A](implicit schema: Schema[A]): BinaryCodec[A] = ???
}
```

Now, let's write an example and see how it works:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
  implicit val binaryCodec: BinaryCodec[Person] =
    AvroCodec.schemaBasedBinaryCodec[Person]
}

object Main extends ZIOAppDefault {
  def run =
    for {
      _ <- ZIO.debug("AvroCodec Example:")
      encodedPerson = Person.binaryCodec.encode(Person("John", 42))
      _ <- ZIO.debug(s"encoded person object: ${toHex(encodedPerson)}")
      decodedPerson <- ZIO.fromEither(
        Person.binaryCodec.decode(encodedPerson)
      )
      _ <- ZIO.debug(s"decoded person object: $decodedPerson")
    } yield ()

  def toHex(bytes: Chunk[Byte]): String =
    bytes.map("%02x".format(_)).mkString(" ")
}
```

The output:

```scala
AvroCodec Example:
encoded person object: 08 4a 6f 68 6e 54
decoded person object: Person(John,42)
```

## Annotations

The Apache Avro specification supports some attributes for describing the data which are not part of the default ZIO Schema. To support these extra metadata, we can use annotations defined in the `zio.schema.codec.AvroAnnotations` object.

There tons of annotations that we can use. Let's introduce some of them:

- `@AvroAnnotations.name(name: String)`: To change the name of a field or a record.
- `@AvroAnnotations.namespace(namespace: String)`: To add the namespace for a field or a record.
- `@AvroAnnotations.doc(doc: String)`: To add documentation to a field or a record.
- `@AvroAnnotations.aliases(aliases: Set[String])`: To add aliases to a field or a record.
- `@AvroAnnotations.avroEnum`: To treat a sealed trait as an Avro enum.
- `@AvroAnnotations.scale(scale: Int = 24)` and `@AvroAnnotations.precision(precision: Int = 48)`: To describe the scale and precision of a decimal field.
- `@AvroAnnotations.decimal(decimalType: DecimalType)`: Used to annotate a `BigInteger` or `BigDecimal` type to indicate the logical type encoding (avro bytes or avro fixed).
- `@AvroAnnotations.bytes(bytesType: BytesType)`: Used to annotate a Byte type to indicate the avro type encoding (avro bytes or avro fixed).
- `@AvroAnnotations.formatToString`: Used to annotate fields of type `LocalDate`, `LocalTime`, `LocalDateTime` or `Instant` in order to render them as a string using the given formatter instead of rendering them as avro logical types.
- `@AvroAnnotations.timeprecision(timeprecisionType: TimePrecisionType)`: Used to indicate the precision (millisecond precision or microsecond precision) of avro logical types `Time`, `Timestamp` and `Local timestamp`
- `@AvroAnnotations.error`: Used to annotate a record in order to render it as a avro error record
- `@AvroAnnotations.fieldOrder(fieldOrderType: FieldOrderType)`: Used to indicate the avro field order of a record

For example, to change the name of a field in the Avro schema, we can use the `AvroAnnotations.name` annotation:

```scala

@AvroAnnotations.name("User")
case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
}
```

Now, if we generate the Avro schema for the `Person` class, we will see that the name of the record is `User` instead of `Person`:

```scala

object Main extends ZIOAppDefault {
  def run =
    for {
      _          <- ZIO.debug("AvroSchemaCodec Example with annotations:")
      avroSchema <- ZIO.fromEither(AvroSchemaCodec.encode(Person.schema))
      _ <- ZIO.debug(s"The person schema in Avro Schema JSON format: $avroSchema")
    } yield ()
}
```

The output:

```scala
The person schema in Avro Schema JSON format: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":{"type":"bytes","logicalType":"decimal","precision":48,"scale":24}}]}
```

---

## Bson Codecs

## Introduction

BSON (Binary JSON) is a binary serialization format used to store and exchange data efficiently. In this article, we will explore how to derive BSON codecs from a ZIO Schema. The `zio-schema-bson` module, provides support for deriving codecs from ZIO Schema, and makes it easy to communicate data in BSON format.

## Installation

To use BSON codecs, you need to add the following dependency to your Scala project:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-bson" % 1.7.5
```

## BsonSchemaCodec

The `BsonSchemaCodec` object inside the `zio.schema.codec` package provides the `bsonCodec` operator which allows us to derive Protobuf codecs from a ZIO Schema:

```scala
object BsonSchemaCodec {
  def bsonCodec[A](schema: Schema[A]): BsonCodec[A]
}
```

## Example

Let's see an example of how to derive a BSON codec for a case class using ZIO Schema:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
  implicit val bsonCodec: BsonCodec[Person] =
    BsonSchemaCodec.bsonCodec(Person.schema)
}

object Main extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("Bson Example:")
    person: Person     = Person("John", 42)
    encoded: BsonValue = person.toBsonValue
    _       <- ZIO.debug(s"person object encoded to BsonValue: $encoded")
    decoded <- ZIO.fromEither(encoded.as[Person])
    _ <- ZIO.debug(s"BsonValue of person object decoded to Person: $decoded")
  } yield ()
}
```

In the example above, we defined a case class `Person` with fields `name` and `age`. We then derived a ZIO Schema for the `Person` case class using `DeriveSchema.gen`.

The `BsonSchemaCodec.bsonCodec` method allowed us to create a BSON codec for the `Person` case class by passing its corresponding ZIO Schema. Now, we can effortlessly encode `Person` objects to BSON and decode BSON values back to Person instances.

---

## Introduction to ZIO Schema Codecs

Once we generate a schema for a type, we can derive a codec for that type.

A codec is a utility that can encode/decode a value of some type `A` to/from some format (e.g. binary format, JSON, etc.)

## Codec

Unlike codecs in other libraries, a codec in ZIO Schema has no type parameter:

```scala
trait Codec {
  def encoder[A](schema: Schema[A]): ZTransducer[Any, Nothing, A, Byte]
  def decoder[A](schema: Schema[A]): ZTransducer[Any, String, Byte, A]

  def encode[A](schema: Schema[A]): A => Chunk[Byte]
  def decode[A](schema: Schema[A]): Chunk[Byte] => Either[String, A]
}
```

The `Codec` trait has two basic methods:

- `encode[A]`: Given a `Schema[A]` it is capable of generating an `Encoder[A]` ( `A => Chunk[Byte]`) for any Schema.
- `decode[A]`: Given a `Schema[A]` it is capable of generating a `Decoder[A]` ( `Chunk[Byte] => Either[String, A]`) for any Schema.

## Binary Codecs

The binary codecs are codecs that can encode/decode a value of some type `A` to/from binary format (e.g. `Chunk[Byte]`).  In ZIO Schema, by having a `BinaryCodec[A]` instance, other than being able to encode/decode a value of type `A` to/from binary format, we can also encode/decode a stream of values of type `A` to/from a stream of binary format.

```scala

trait Decoder[Whole, Element, +A] {
  def decode(whole: Whole): Either[DecodeError, A]
  def streamDecoder: ZPipeline[Any, DecodeError, Element, A]
}

trait Encoder[Whole, Element, -A] {
  def encode(value: A): Whole
  def streamEncoder: ZPipeline[Any, Nothing, A, Element]
}

trait Codec[Whole, Element, A] extends Encoder[Whole, Element, A] with Decoder[Whole, Element, A]

trait BinaryCodec[A] extends Codec[Chunk[Byte], Byte, A]
```

To make it simpler, we can think of a `BinaryCodec[A]` as the following trait:

```scala

trait BinaryCodec[A] {
  def encode(value: A): Chunk[Byte]
  def decode(whole: Chunk[Byte]): Either[DecodeError, A]

  def streamEncoder: ZPipeline[Any, Nothing, A, Byte]
  def streamDecoder: ZPipeline[Any, DecodeError, Byte, A]
}
```

Example of possible codecs are:

- CSV Codec
- JSON Codec (already available)
- Apache Avro Codec (in progress)
- Apache Thrift Codec (in progress)
- XML Codec
- YAML Codec
- Protobuf Codec (already available)
- QueryString Codec
- etc.

---

## JSON Codecs

## Introduction

JSON (JavaScript Object Notation) is a widely used data interchange format for transmitting and storing data. ZIO Schema provides `zio-schema-json` module which has functionality to derive JSON codecs from a ZIO Schema. JSON codecs allow us to easily serialize and deserialize data in JSON format. In this article, we will explore how derive JSON codecs using the ZIO Schema.

## Installation

To derive JSON codecs from a ZIO Schema, we need to add the following dependency to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-json" % 1.7.5
```

## JsonCodec

The `JsonCodec` object inside the `zio.schema.codec` package provides the `jsonCodec` operator which allows us to derive JSON codecs from a ZIO Schema:

```scala
object JsonCodec {
  def jsonCodec[A](schema: Schema[A]): zio.json.JsonCodec[A] = ???
}
```

Let's try an example to see how it works:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    DeriveSchema.gen
  implicit val jsonCodec: zio.json.JsonCodec[Person] =
    zio.schema.codec.JsonCodec.jsonCodec(schema)
}

object Main extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("JSON Codec Example:")
    person: Person  = Person("John", 42)
    encoded: String = person.toJson
    _       <- ZIO.debug(s"person object encoded to JSON string: $encoded")
    decoded <- ZIO.fromEither(Person.jsonCodec.decodeJson(encoded))
    _       <- ZIO.debug(s"JSON object decoded to Person class: $decoded")
  } yield ()
}
```

## BinaryCodec

We can also derive a binary codec from a ZIO Schema using the `schemaBasedBinaryCodec`:

```scala
object JsonCodec {
  implicit def schemaBasedBinaryCodec[A](implicit schema: Schema[A]): BinaryCodec[A] = ???
}
```

Let's try an example:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    DeriveSchema.gen
  implicit val jsonBinaryCodec: BinaryCodec[Person] =
    zio.schema.codec.JsonCodec.schemaBasedBinaryCodec(schema)
}

object Main extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("JSON Codec Example:")
    person: Person = Person("John", 42)
    encoded: Chunk[Byte] = Person.jsonBinaryCodec.encode(person)
    _ <- ZIO.debug(s"person object encoded to Binary JSON: ${toHex(encoded)}")
    decoded <- ZIO.fromEither(Person.jsonBinaryCodec.decode(encoded))
    _ <- ZIO.debug(s"JSON object decoded to Person class: $decoded")
  } yield ()

  def toHex(bytes: Chunk[Byte]): String =
    bytes.map("%02x".format(_)).mkString(" ")
}
```

The output of the above program is:

```scala
JSON Codec Example:
person object encoded to JSON string: 7b 22 6e 61 6d 65 22 3a 22 4a 6f 68 6e 22 2c 22 61 67 65 22 3a 34 32 7d
JSON object decoded to Person class: Person(John,42)
```

By utilizing JSON codecs derived from ZIO Schema, developers can easily serialize and deserialize data in JSON format without writing boilerplate code. This enhances productivity and simplifies data handling in Scala applications.

---

## MessagePack Codecs

## Introduction

MessagePack is a binary serialization format designed for efficient data exchange between different systems and languages. In this section, we will explore how to derive MessagePack codecs from a ZIO Schema. MessagePack codecs allow us to easily serialize and deserialize data in MessagePack format.

## Installation

To use MessagePack codecs, you need to add the following dependency to your build.sbt file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-msg-pack" % "1.7.5"
```

## BinaryCodec

The `MessagePackCodec` object inside the `zio.schema.codec` package provides the `messagePackCodec` operator which allows us to derive MessagePack codecs from a ZIO Schema:

```scala
object MessagePackCodec {
  implicit def messagePackCodec[A](implicit schema: Schema[A]): BinaryCodec[A] = ???
}
```

## Example

Let's try an example to see how it works:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    DeriveSchema.gen
  implicit val msgPackCodec: BinaryCodec[Person] =
    MessagePackCodec.messagePackCodec(schema)
}

object Main extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("MessagePack Codec Example:")
    person: Person = Person("John", 42)
    encoded: Chunk[Byte] = Person.msgPackCodec.encode(person)
    _ <- ZIO.debug(s"person object encoded to MessagePack's binary format: ${toHex(encoded)}")
    decoded <- ZIO.fromEither(Person.msgPackCodec.decode(encoded))
    _ <- ZIO.debug(s"MessagePack object decoded to Person class: $decoded")
  } yield ()

  def toHex(bytes: Chunk[Byte]): String =
    bytes.map("%02x".format(_)).mkString(" ")
}
```

The output of the above program is:

```scala
MessagePack Codec Example:
person object encoded to MessagePack's binary format: 82 a4 6e 61 6d 65 a4 4a 6f 68 6e a3 61 67 65 2a
MessagePack object decoded to Person class: Person(John,42)
```

---

## Protobuf Codecs

## Introduction

Protocol Buffers (protobuf) is a binary serialization format developed by Google. It is designed for efficient data exchange between different systems and languages. In this article, we will explore how to derive Protobuf codecs from a ZIO Schema. Protobuf codecs allow us to easily serialize and deserialize data in Protobuf format, making it simple to interact with APIs and data sources that use Protobuf as their data format.

## Installation

To start using Protobuf codecs in ZIO, you need to add the following dependency to your build.sbt file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-protobuf" % "1.7.5"
```

## BinaryCodec

The `ProtobufCodec` object inside the `zio.schema.codec` package provides the `protobufCodec` operator which allows us to derive Protobuf codecs from a ZIO Schema:

```scala
object ProtobufCodec {
  implicit def protobufCodec[A](implicit schema: Schema[A]): BinaryCodec[A] = ???
}
```

Optionally the `@fieldNumber(1)` annotation can be used on fields to specify the field number for a case class field. This together with default values
can be used to keep binary compatibility when evolving schemas. Default field numbers are indexed starting from 1.

For example considering the following three versions of a record:

```scala
final case class RecordV1(x: Int, y: Int)
final case class RecordV2(x: Int = 100, y: Int, z: Int)
final case class RecordV3(@fieldNumber(2) y: Int, @fieldNumber(4) extra: String = "unknown", @fieldNumber(3) z: Int)
```

The decoder of V1 can decode a binary encoded by V2, but cannot decode a binary encoded by V3 because it does not have a field number 1 (x).
The decoder of V2 can decode a binary encoded by V3 because it has a default value for field number 1 (x), 100. The decoder of V3 can read V2 but
cannot read V1 (as it does not have field number 3 (z)). As demonstrated, using explicit field numbers also allows reordering the fields without 
breaking the format. 

```scala

## Example: BinaryCodec

Let's try an example:

```scala mdoc:compile-only

case class Person(name: String, age: Int)

object Person {
  implicit val schema       : Schema[Person]      =
    DeriveSchema.gen
  implicit val protobufCodec: BinaryCodec[Person] =
    ProtobufCodec.protobufCodec(schema)
}

object Main extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("Protobuf Codec Example:")
    person: Person = Person("John", 42)
    encoded: Chunk[Byte] = Person.protobufCodec.encode(person)
    _ <- ZIO.debug(
      s"person object encoded to Protobuf's binary format: ${toHex(encoded)}"
    )
    decoded <- ZIO.fromEither(Person.protobufCodec.decode(encoded))
    _ <- ZIO.debug(s"Protobuf object decoded to Person class: $decoded")
  } yield ()

  def toHex(bytes: Chunk[Byte]): String =
    bytes.map("%02x".format(_)).mkString(" ")
}
```

Here is the output of running the above program:

```scala
Protobuf Codec Example:
person object encoded to Protobuf's binary format: 0a 04 4a 6f 68 6e 10 2a
Protobuf object decoded to Person class: Person(John,42)
```

## Example: Streaming Codecs

The following example shows how to use Protobuf codecs to encode and decode streams of data:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    DeriveSchema.gen
  implicit val protobufCodec: BinaryCodec[Person] =
    ProtobufCodec.protobufCodec(schema)
}

object Main extends ZIOAppDefault {

  def run = for {
    _ <- ZIO.debug("Protobuf Stream Codecs Example:")
    person = Person("John", 42)

    personToProto = Person.protobufCodec.streamEncoder
    protoToPerson = Person.protobufCodec.streamDecoder

    newPerson <- ZStream(person)
      .via(personToProto)
      .via(protoToPerson)
      .runHead
      .some
      .catchAll(error => ZIO.debug(error))
    _ <- ZIO.debug(
      "is old person the new person? " + (person == newPerson).toString
    )
    _ <- ZIO.debug("old person: " + person)
    _ <- ZIO.debug("new person: " + newPerson)
  } yield ()
}
```

The output of running the above program is:

```scala
Protobuf Stream Codecs Example:
is old person the new person? true
old person: Person(John,42)
new person: Person(John,42)
```

---

## Apache Thrift Codecs

## Introduction

Apache Thrift is an open-source framework that allows seamless communication and data sharing between different programming languages and platforms. In this section, we will explore how to derive Apache Thrift codecs from a ZIO Schema.

## Installation

To derive Apache Thrift codecs from a ZIO Schema, we need to add the following dependency to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-thrift" % "1.7.5"
```

## BinaryCodec

The `ThriftCodec` object inside the `zio.schema.codec` package provides the `thriftCodec` operator which allows us to derive Protobuf codecs from a ZIO Schema:

```scala
object ThriftCodec {
  implicit def thriftCodec[A](implicit schema: Schema[A]): BinaryCodec[A] = ???
}
```

## Example

Let's try an example:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    DeriveSchema.gen
    
  implicit val thriftCodec: BinaryCodec[Person] =
    ThriftCodec.thriftCodec(schema)
}

object Main extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("Apache Thrift Codec Example:")
    person: Person = Person("John", 42)
    encoded: Chunk[Byte] = Person.thriftCodec.encode(person)
    _ <- ZIO.debug(s"person object encoded to Thrift's binary format: ${toHex(encoded)}")
    decoded <- ZIO.fromEither(Person.thriftCodec.decode(encoded))
    _ <- ZIO.debug(s"Thrift object decoded to Person class: $decoded")
  } yield ()

  def toHex(bytes: Chunk[Byte]): String =
    bytes.map("%02x".format(_)).mkString(" ")
}
```

Here is the output of running the above program:

```scala
Apache Thrift Codec Example: 
person object encoded to Thrift's binary format: 0b 00 01 00 00 00 04 4a 6f 68 6e 08 00 02 00 00 00 2a 00
Thrift object decoded to Person class: Person(John,42)
```

---

## Optics Derivation

Optics are a way of accessing and manipulating data in a functional way. They can be used to get, set, and update values in data structures, as well as to traverse and explore data.

## Manual Derivation of Optics

Before we dive into auto-derivation of optics and how we can derive optics from a ZIO Schema, let's take a look at the pure optics and how we can create them manually using [ZIO Optics](https://zio.dev/zio-optics) library.

First, we should add `zio-optics` to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-optics" % "<version>"
```

Now let's define a simple data type called `User` and create two optics for its `name` and `age` fields:

```scala

case class User(name: String, age: Int)

val nameLens = Lens[User, String](
  user => Right(user.name),
  name => user => Right(user.copy(name = name))
)

val ageLens = Lens[User, Int](
  user => Right(user.age),
  age => user => Right(user.copy(age = age))
)

val ageAndNameLens = nameLens.zip(ageLens)
```

Now we can use these optics to get, set, and update values in the `Person` data structure:

```scala

object Main extends ZIOAppDefault {
  def run =
    for {
      _ <- ZIO.debug("Pure Optics")
      user = User("John", 34)

      updatedUser1 <- ZIO.fromEither(nameLens.setOptic("Jane")(user))
      _ <- ZIO.debug(s"Name of user updated: $updatedUser1")

      updatedUser2 <- ZIO.fromEither(ageLens.setOptic(32)(user))
      _ <- ZIO.debug(s"Age of user updated: $updatedUser2")

      updatedUser3 <- ZIO.fromEither(
        ageAndNameLens.set(("Jane", 32))(User("John", 34))
      )
      _ <- ZIO.debug(s"Name and age of the user updated: $updatedUser3")
    } yield ()
}
```

## Automatic Derivation of Optics

ZIO Schema has a module called `zio-schema-optics` which provides functionalities to derive various optics from a ZIO Schema.

By having a `Schema[A]`, we can derive optics automatically from a schema. This means that we don't have to write the optics manually, but instead, we can use the `Schema#makeAccessors` method which will derive the optics for us:

```scala
trait Schema[A] {
  def makeAccessors(b: AccessorBuilder): Accessors[b.Lens, b.Prism, b.Traversal]
}
```

It takes an `AccessorBuilder` which is an interface of the creation of optics:

```scala
trait AccessorBuilder {
  type Lens[F, S, A]
  type Prism[F, S, A]
  type Traversal[S, A]

  def makeLens[F, S, A](
    product: Schema.Record[S],
    term: Schema.Field[S, A]
  ): Lens[F, S, A]

  def makePrism[F, S, A](
    sum: Schema.Enum[S],
    term: Schema.Case[S, A]
  ): Prism[F, S, A]

  def makeTraversal[S, A](
    collection: Schema.Collection[S, A],
    element: Schema[A]
  ): Traversal[S, A]
}
```

It has three methods for creating three types of optics:

- **Lens** is an optic used to get and update values in a product type.
- **Prism** is an optic used to get and update values in a sum type.
- **Traversal** is an optic used to get and update values in a collection type.

Let's take a look at how we can derive optics using ZIO Schema Optics.

### Installation

To be able to derive optics from a ZIO Schema, we need to add the following line to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-optics" % 1.7.5
```

This package contains a `ZioOpticsBuilder` which is an implementation of the `AccessorBuilder` interface based on ZIO Optics library.

Now we are ready to try any of the following examples:

### Examples

#### Lens

Now we can derive the schema for our `User` data type in its companion object, and then derive optics using `Schema#makeAccessors` method:

```scala

case class User(name: String, age: Int)

object User {
  implicit val schema: CaseClass2[String, Int, User] =
    DeriveSchema.gen[User].asInstanceOf[CaseClass2[String, Int, User]]

  val (nameLens, ageLens) = schema.makeAccessors(ZioOpticsBuilder)
}
```

Based on the type of the schema, the `makeAccessors` method will derive the proper optics for us.

Now we can use these optics to update values in the `User` data structure:

```scala
object MainApp extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("Auto-derivation of Optics")
    user = User("John", 42)

    updatedUser1 = User.nameLens.set("Jane")(user)
    _ <- ZIO.debug(s"Name of user updated: $updatedUser1")

    updatedUser2 = User.ageLens.set(32)(user)
    _ <- ZIO.debug(s"Age of user updated: $updatedUser2")

    nameAndAgeLens = User.nameLens.zip(User.ageLens)
    updatedUser3   = nameAndAgeLens.set(("Jane", 32))(user)
    _ <- ZIO.debug(s"Name and age of the user updated: $updatedUser3")
  } yield ()
}
```

Output:

```scala
Auto-derivation of Lens Optics:
Name of user updated: Right(User(Jane,42))
Age of user updated: Right(User(John,32))
Name and age of the user updated: Right(User(Jane,32))
```

#### Prism

```scala

sealed trait Shape {
  def area: Double
}

case class Circle(radius: Double) extends Shape {
  val area: Double = Math.PI * radius * radius
}

case class Rectangle(width: Double, height: Double) extends Shape {
  val area: Double = width * height
}

object Shape {
  implicit val schema: Enum2[Circle, Rectangle, Shape] =
    DeriveSchema.gen[Shape].asInstanceOf[Enum2[Circle, Rectangle, Shape]]

  val (circlePrism, rectanglePrism) =
    schema.makeAccessors(ZioOpticsBuilder)
}

object MainApp extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("Auto-derivation of Prism Optics")
    shape = Circle(1.2)
    _ <- ZIO.debug(s"Original shape: $shape")
    updatedShape <- ZIO.fromEither(
      Shape.rectanglePrism.setOptic(Rectangle(2.0, 3.0))(shape)
    )
    _ <- ZIO.debug(s"Updated shape: $updatedShape")
  } yield ()

}
```

Output:

```scala
Auto-derivation of Prism Optics:
Original shape: Circle(1.2)
Updated shape: Rectangle(2.0,3.0)
```

#### Traversal

```scala

object IntList {
  implicit val listschema: Schema.Sequence[List[Int], Int, String] =
    Sequence[List[Int], Int, String](
      elementSchema = Schema[Int],
      fromChunk = _.toList,
      toChunk = i => Chunk.fromIterable(i),
      annotations = Chunk.empty,
      identity = "List"
    )

  val traversal: ZTraversal[List[Int], List[Int], Int, Int] =
    listschema.makeAccessors(ZioOpticsBuilder)
}

object MainApp extends ZIOAppDefault {
  def run = for {
    _ <- ZIO.debug("Auto-derivation of Traversal Optic:")
    list = List(1, 2, 3, 4, 5)
    _           <- ZIO.debug(s"Original list: $list")
    updatedList <- ZIO.fromEither(IntList.traversal.set(Chunk(1, 5, 7))(list))
    _           <- ZIO.debug(s"Updated list: $updatedList")
  } yield ()
}
```

Output:

```scala
Auto-derivation of Traversal Optic:
Original list: List(1, 2, 3, 4, 5)
Updated list: List(1, 5, 7, 4, 5)
```

---

## Ordering Derivation

Standard Scala library provides a type class called `Ordering[A]` that allows us to compare values of type `A`. ZIO Schema provides a method called `ordering` that generates an `Ordering[A]` instance for the underlying type described by the schema:

```scala
sealed trait Schema[A] {
  def ordering: Ordering[A]
}
```

Here is an example, where it helps us to sort the list of `Person`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
}

val sortedList: Seq[Person] =
  List(
    Person("John", 42),
    Person("Jane", 34)
  ).sorted(Person.schema.ordering)
```

---

## Derivation of ZIO Test Generators

## Introduction

ZIO Test supports property-based testing via the `Gen` type. `Gen[R, A]` is a random generator of values of type `A`. Such a generator can be used to produce test cases for a property, which can then be checked for validity. The `zio-schema-zio-test` module provides a way to derive a `Gen[R, A]` from a `Schema[A]`. In this section, we will see how this functionality works.

## Installation

In order to derive a generator from a ZIO Schema, we need to add the following dependency to our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-zio-test" % 1.7.5
```

## DriveGen

The `DriveGen` inside `zio.schema` package provides the `gen` operator which takes a `Schema[A]` implicitly and returns a `Gen[Sized, A]`:

```scala
object DeriveGen {
  def gen[A](implicit schema: Schema[A]): Gen[Sized, A] = ???
}
```

## Example

In the following example, we will derive a generator for the `Person` class using the `DeriveGen.gen` operator:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
  val gen: Gen[Sized, Person]         = DeriveGen.gen
}

object ExampleSpec extends ZIOSpecDefault {
  def spec =
    test("example test") {
      check(Person.gen) { p =>
        assertTrue(???)
      }
    }
}
```

---

## Combining Different Encoders

Let's take a look at a round-trip converting an object to JSON and back, then converting it to a protobuf and back. This is a simple example, but it shows how to combine different encoders to achieve a round-trip.

```scala
object CombiningExample extends zio.App {

  override def run(args: List[String]): UIO[ExitCode] = for {
    _ <- ZIO.unit
    _ <- ZIO.debug("combining roundtrip")
    person = Person("Michelle", 32)

    personToJson = JsonCodec.encoder[Person](schemaPerson)
    jsonToPerson = JsonCodec.decoder[Person](schemaPerson)

    personToProto = ProtobufCodec.encoder[Person](schemaPerson)
    protoToPerson = ProtobufCodec.decoder[Person](schemaPerson)

    newPerson <- ZStream(person)
      .tap(v => ZIO.debug("input object is: " + v))
      .transduce(personToJson)
      .transduce(jsonToPerson)
      .tap(v => ZIO.debug("object after json roundtrip: " + v))
      .transduce(personToProto)
      .transduce(protoToPerson)
      .tap(v => ZIO.debug("person after protobuf roundtrip: " + v))
      .runHead
      .some
      .catchAll(error => ZIO.debug(error))
    _ <- ZIO.debug("is old person the new person? " + (person == newPerson).toString)
    _ <- ZIO.debug("old person: " + person)
    _ <- ZIO.debug("new person: " + newPerson)
  } yield ExitCode.success
}
```

---

## Mapping DTO to Domain Object

When we write layered applications, where different layers are decoupled from each other, we need to transfer data between layers. For example, assume we have a layer that has `Person` data type and it receives JSON string of type `PersonDTO` from another layer. We need to convert `PersonDTO` to `Person` and maybe vice versa.

One way to do this is to write codec for `PersonDTO` and convert the JSON String to the `PersonDTO` and then convert `PersonDTO` to `Person`. This approach is not very convenient and we need to write some boilerplate code. With ZIO Schema we can simplify this process and write a codec for `Person` that uses a specialized schema for `Person`, i.e. `personDTOMapperSchema`, which describes `Person` data type in terms of transformation from `PersonDTO` to `Person` and vice versa. With this approach, we can directly convert the JSON string to `Person` in one step:

```scala

object MainApp extends ZIOAppDefault {

  case class PersonDTO(
                        firstName: String,
                        lastName: String,
                        birthday: (Int, Int, Int)
                      )

  object PersonDTO {
    implicit val schema: Schema[PersonDTO] = DeriveSchema.gen[PersonDTO]

    implicit val codec: JsonCodec[PersonDTO] = jsonCodec[PersonDTO](schema)
  }

  case class Person(name: String, birthdate: LocalDate)

  object Person {
    implicit val schema: Schema[Person] = DeriveSchema.gen[Person]

    val personDTOMapperSchema: Schema[Person] =
      PersonDTO.schema.transform(
        f = dto => {
          val (year, month, day) = dto.birthday
          Person(
            dto.firstName + " " + dto.lastName,
            birthdate = LocalDate.of(year, month, day)
          )
        },
        g = (person: Person) => {
          val fullNameArray = person.name.split(" ")
          PersonDTO(
            fullNameArray.head,
            fullNameArray.last,
            (
              person.birthdate.getYear,
              person.birthdate.getMonthValue,
              person.birthdate.getDayOfMonth
            )
          )
        }
      )
    implicit val codec: JsonCodec[Person] = jsonCodec[Person](schema)

    val personDTOJsonMapperCodec: JsonCodec[Person] =
      jsonCodec[Person](personDTOMapperSchema)
  }

  val json: String =
    """
      |{
      |   "firstName": "John",
      |   "lastName": "Doe",
      |   "birthday": [[1981, 07], 13]
      |}
      |""".stripMargin

  def run: zio.ZIO[Any, String, Any] =
    for {
      // Approach 1: Decode JSON String to PersonDTO and then Transform it into the Person object
      personDTO <- ZIO.fromEither(JsonCodec[PersonDTO].decodeJson(json))
      (year, month, day) = personDTO.birthday
      person1 = Person(
        name = personDTO.firstName + " " + personDTO.lastName,
        LocalDate.of(year, month, day)
      )
      _ <- ZIO.debug(
        s"person: $person1"
      )

      // Approach 2: Decode JSON string in one step into the Person object
      person2 <- ZIO.fromEither(
        JsonCodec[Person](Person.personDTOJsonMapperCodec).decodeJson(json)
      )
      _ <- ZIO.debug(
        s"person: $person2"
          )
    } yield assert(person1 == person2)
}
```

As we can see in the example above, the second approach is much simpler and more convenient than the first one.

The problem we solved in previous example is common in microservices architecture, where we transfer DTOs across the network. So we need to serialize and deserialize the data transfer objects.

In the next example, we will see how we can use schema migration, when we need to map data transfer object to domain object and vice versa. In this example, we do not require to serialize/deserialize any object, but the problem of mapping DTO to domain object persists.

In this example, similar to the previous one, we will define the schema for `Person` in terms of schema transformation from `PersonDTO` to `Person` and vice versa. The only difference is that we will utilize the `Schema#migrate` method to map `PersonDTO` to `Person`. This method returns `Either[String, PersonDTO => Either[String, Person]]`. If the migration is successful, we will receive `Right` with a function that converts `PersonDTO` to `Either[String, Person]`. Otherwise, if there is an error, we will receive `Left` along with an error message:

```scala

object MainApp extends ZIOAppDefault {

  case class PersonDTO(
                        firstName: String,
                        lastName: String,
                        birthday: (Int, Int, Int)
                      )

  object PersonDTO {
    implicit val schema: Schema[PersonDTO] = DeriveSchema.gen[PersonDTO]
  }

  case class Person(name: String, birthdate: LocalDate)

  object Person {
    implicit val schema: Schema[Person] = DeriveSchema.gen[Person]

    val personDTOMapperSchema: Schema[Person] =
      PersonDTO.schema.transform(
        f = dto => {
          val (year, month, day) = dto.birthday
          Person(
            dto.firstName + " " + dto.lastName,
            birthdate = LocalDate.of(year, month, day)
          )
        },
        g = (person: Person) => {
          val fullNameArray = person.name.split(" ")
          PersonDTO(
            fullNameArray.head,
            fullNameArray.last,
            (
              person.birthdate.getYear,
              person.birthdate.getMonthValue,
              person.birthdate.getDayOfMonth
            )
          )
        }
      )

    def fromPersonDTO(p: PersonDTO): IO[String, Person] =
      ZIO.fromEither(
        PersonDTO.schema
          .migrate(personDTOMapperSchema)
          .flatMap(_(p))
      )
  }

  def run: zio.ZIO[Any, String, Any] =
    for {
      personDTO <- ZIO.succeed(PersonDTO("John", "Doe", (1981, 7, 13)))
      person <- Person.fromPersonDTO(personDTO)
      _ <- ZIO.debug(s"person: $person")
    } yield ()

}
```

---

## Introduction to ZIO Schema

[ZIO Schema](https://github.com/zio/zio-schema) is a [ZIO](https://zio.dev)-based library for modeling the schema of data structures as first-class values.

[![Development](https://img.shields.io/badge/Project%20Stage-Development-green.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-schema/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-schema_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-schema_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-schema_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-schema_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-schema-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-schema-docs_2.13) [![ZIO Schema](https://img.shields.io/github/stars/zio/zio-schema?style=social)](https://github.com/zio/zio-schema)

## Introduction

ZIO Schema helps us to solve some of the most common problems in distributed computing, such as serialization, deserialization, and data migration.

It turns a compiled-time construct (the type of a data structure) into a runtime construct (a value that can be read, manipulated, and composed at runtime). A schema is a structure of a data type. ZIO Schema reifies the concept of structure for data types. It makes a high-level description of any data type and makes them first-class values.

Creating a schema for a data type helps us to write codecs for that data type. So this library can be a host of functionalities useful for writing codecs and protocols like JSON, Protobuf, CSV, and so forth.

## What Problems Does ZIO Schema Solve?

With schema descriptions that can be automatically derived for case classes and sealed traits, _ZIO Schema_ will be going to provide powerful features for free:

1. Metaprogramming without macros, reflection, or complicated implicit derivations.
    1. Creating serialization and deserialization codecs for any supported protocol (JSON, Protobuf, etc.)
    2. Deriving standard type classes (`Eq`, `Show`, `Ordering`, etc.) from the structure of the data
    3. Default values for data types
2. Automate ETL (Extract, Transform, Load) pipelines
    1. Diffing: diffing between two values of the same type
    2. Patching: applying a diff to a value to update it
    3. Migration: migrating values from one type to another
3. Computations as data: Not only we can turn types into values, but we can also turn computations into values. This opens up a whole new world of possibilities concerning distributed computing.

When our data structures need to be serialized, deserialized, persisted, or transported across the wire, then _ZIO Schema_ lets us focus on data modeling and automatically tackle all the low-level, messy details for us.

_ZIO Schema_ is used by a growing number of ZIO libraries, including [ZIO Flow](https://zio.dev/zio-flow), [ZIO Redis](https://zio-redis), [ZIO SQL](https://zio.dev/zio-sql) and [ZIO DynamoDB](https://zio.dev/zio-dynamodb).

## Installation

In order to use this library, we need to add the following lines in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-schema"          % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-avro"     % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-bson"     % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-json"     % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-msg-pack" % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-protobuf" % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-thrift"   % "1.7.5"
libraryDependencies += "dev.zio" %% "zio-schema-zio-test" % "1.7.5"

// Required for the automatic generic derivation of schemas
libraryDependencies += "dev.zio" %% "zio-schema-derivation" % "1.7.5"
libraryDependencies += "org.scala-lang" % "scala-reflect"  % scalaVersion.value % "provided"
```

## Example

In this simple example first, we create a schema for `Person` and then run the _diff_ operation on two instances of the `Person` data type, and finally, we encode a Person instance using _Protobuf_ protocol:

```scala

final case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person]    = DeriveSchema.gen
  val protobufCodec: BinaryCodec[Person] = ProtobufCodec.protobufCodec
}

object Main extends ZIOAppDefault {
  def run: ZIO[Any, IOException, Unit] =
    ZStream
      .succeed(Person("John", 43))
      .via(Person.protobufCodec.streamEncoder)
      .runCollect
      .flatMap(x =>
        Console.printLine(s"Encoded data with protobuf codec: ${toHex(x)}")
      )

  def toHex(chunk: Chunk[Byte]): String =
    chunk.map("%02X".format(_)).mkString
}
```

Here is the output of running the above program:

```scala
Encoded data with protobuf codec: 0A044A6F686E102B
```

## Resources

- [Zymposium - ZIO Schema](https://www.youtube.com/watch?v=GfNiDaL5aIM) by John A. De Goes, Adam Fraser, and Kit Langton (May 2021)
- [ZIO SCHEMA: A Toolkit For Functional Distributed Computing](https://www.youtube.com/watch?v=lJziseYKvHo&t=481s) by Dan Harris (Functional Scala 2021)
- [Creating Declarative Query Plans With ZIO Schema](https://www.youtube.com/watch?v=ClePN4P9_pg) by Dan Harris (ZIO World 2022)
- [Describing Data...with free applicative functors (and more)](https://www.youtube.com/watch?v=oRLkb6mqvVM) by Kris Nuttycombe (Scala World) on the idea behind the [xenomorph](https://github.com/nuttycom/xenomorph) library

---

## Integration with ZIO Streams

In addition to the regular `encode` and `decode` functions, each codec also has a streaming version of these functions called `streamEncoder` and `streamDecoder`. By invoking these methods on codecs, we can obtain a `ZPipeline` where the encoder and decoder are integrated into the `ZPipeline` stream transformer.

We can use the `ZPipline` to transform (encode/decode) a stream of values of type `A` into a stream of values of type `B`. 

For example, assume we have a stream of `Person` values, and we want to encode them into a stream of bytes and then convert back to `Person` values. We can do this as follows:

```scala

object Main extends ZIOAppDefault {
  case class Person(name: String, age: Int)

  object Person {
    implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
  }

  def run: ZIO[Any, Exception, Unit] =
    ZStream
      .fromIterable(Seq(Person("John", 42)))
      .debug("the input object is")
      .via(JsonCodec.schemaBasedBinaryCodec[Person].streamEncoder)
      .via(ZPipeline.utfDecode)
      .debug("json string of person")
      .via(ZPipeline.utf8Encode)
      .via(JsonCodec.schemaBasedBinaryCodec[Person].streamDecoder)
      .debug("person after roundtrip")
      .runDrain
}
```

---

## Manual Schema Construction

Assume we have a domain containing following models:

```scala
object Domain {
  final case class Person(name: String, age: Int)

  sealed trait PaymentMethod

  object PaymentMethod {
    final case class CreditCard(number: String, expirationMonth: Int, expirationYear: Int) extends PaymentMethod
    final case class WireTransfer(accountNumber: String, bankCode: String) extends PaymentMethod
  }

  final case class Customer(person: Person, paymentMethod: PaymentMethod)
  
}
```

Let's begin by creating a schema for the `Person` data type:

```scala

final case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] =
    Schema.CaseClass2[String, Int, Person](
      id0 = TypeId.fromTypeName("Person"),
      field01 = Schema.Field(name0 = "name", schema0 = Schema[String], get0 = _.name, set0 = (p, x) => p.copy(name = x)),
      field02 = Schema.Field(name0 = "age", schema0 = Schema[Int], get0 = _.age, set0 = (person, age) => person.copy(age = age)),
      construct0 = (name, age) => Person(name, age),
    )
}
```

The next step is writing schema for `PaymentMethod`:

```scala

sealed trait PaymentMethod

object PaymentMethod {
  implicit val schema: Schema[PaymentMethod] =
    Schema.Enum2[CreditCard, WireTransfer, PaymentMethod](
      id = TypeId.fromTypeName("PaymentMethod"),
      case1 = Schema.Case[PaymentMethod, CreditCard](
        id = "CreditCard",
        schema = CreditCard.schema,
        unsafeDeconstruct = pm => pm.asInstanceOf[PaymentMethod.CreditCard],
        construct = cc => cc.asInstanceOf[PaymentMethod],
        isCase = _.isInstanceOf[PaymentMethod.CreditCard],
        annotations = Chunk.empty
      ),
      case2 = Schema.Case[PaymentMethod, WireTransfer](
        id = "WireTransfer",
        schema = WireTransfer.schema,
        unsafeDeconstruct = pm => pm.asInstanceOf[PaymentMethod.WireTransfer],
        construct = wt => wt.asInstanceOf[PaymentMethod],
        isCase = _.isInstanceOf[PaymentMethod.WireTransfer],
        annotations = Chunk.empty
      )
    )

  final case class CreditCard(
      number: String,
      expirationMonth: Int,
      expirationYear: Int
  ) extends PaymentMethod

  object CreditCard {
    implicit val schema: Schema[CreditCard] =
      Schema.CaseClass3[String, Int, Int, CreditCard](
        id0 = TypeId.fromTypeName("CreditCard"),
        field01 = Schema.Field[CreditCard, String](
          name0 = "number",
          schema0 = Schema.primitive[String],
          get0 = _.number,
          set0 = (cc, n) => cc.copy(number = n)
        ),
        field02 = Schema.Field[CreditCard, Int](
          name0 = "expirationMonth",
          schema0 = Schema.primitive[Int],
          get0 = _.expirationMonth,
          set0 = (cc, em) => cc.copy(expirationMonth = em)
        ),
        field03 = Schema.Field[CreditCard, Int](
          name0 = "expirationYear",
          schema0 = Schema.primitive[Int],
          get0 = _.expirationYear,
          set0 = (cc, ey) => cc.copy(expirationYear = ey)
        ),
        construct0 = (n, em, ey) => CreditCard(n, em, ey)
      )
  }

  final case class WireTransfer(accountNumber: String, bankCode: String)
      extends PaymentMethod

  object WireTransfer {
    implicit val schema: Schema[WireTransfer] =
      Schema.CaseClass2[String, String, WireTransfer](
        id0 = TypeId.fromTypeName("WireTransfer"),
        field01 = Schema.Field[WireTransfer, String](
          name0 = "accountNumber",
          schema0 = Schema.primitive[String],
          get0 = _.accountNumber,
          set0 = (wt, an) => wt.copy(accountNumber = an)
        ),
        field02 = Schema.Field[WireTransfer, String](
          name0 = "bankCode",
          schema0 = Schema.primitive[String],
          get0 = _.bankCode,
          set0 = (wt, bc) => wt.copy(bankCode = bc)
        ),
        construct0 = (ac, bc) => WireTransfer(ac, bc)
      )
  }
}
```

And finally, we need to define the schema for the `Customer` data type:

```scala

final case class Customer(person: Person, paymentMethod: PaymentMethod)

object Customer {
  implicit val schema: Schema[Customer] =
    Schema.CaseClass2[Person, PaymentMethod, Customer](
      id0 = TypeId.fromTypeName("Customer"),
      field01 = Schema.Field[Customer, Person](
        name0 = "person",
        schema0 = Person.schema,
        get0 = _.person,
        set0 = (c, p) => c.copy(person = p)
      ),
      field02 = Schema.Field[Customer, PaymentMethod](
        name0 = "paymentMethod",
        schema0 = Schema[PaymentMethod],
        get0 = _.paymentMethod,
        set0 = (c, pm) => c.copy(paymentMethod = pm)
      ),
      construct0 = (p, pm) => Customer(p, pm)
    )
}
```

Now that we have written all the required schemas, we can proceed to create encoders and decoders (codecs) for each of our domain models. 

Let's start with writing protobuf codecs. We need to add the following line to our `build.sbt`:

```scala
libraryDependencies += "dev.zio" %% "zio-schema-protobuf" % 1.7.5
```

Here's an example that demonstrates a roundtrip test for protobuf codecs:

```scala

// Create a customer instance
val customer =
  Customer(
    person = Person("John Doe", 42),
    paymentMethod = PaymentMethod.CreditCard("1000100010001000", 6, 2024)
  )

// Create binary codec from customer 
val customerCodec: BinaryCodec[Customer] =
  ProtobufCodec.protobufCodec[Customer]

// Encode the customer object
val encodedCustomer: Chunk[Byte] = customerCodec.encode(customer)

// Decode the byte array back to the person instance
val decodedCustomer: Either[DecodeError, Customer] =
  customerCodec.decode(encodedCustomer)

assert(Right(customer) == decodedCustomer)
```

---

## The Motivation Behind ZIO Schema

ZIO Schema is a library used in many ZIO projects such as _ZIO Flow_, _ZIO Redis_, _ZIO Web_, _ZIO SQL_ and _ZIO DynamoDB_. It is all about reification of our types. Reification means transforming something abstract (e.g. side effects, accessing fields, structure)  into something "real" (values).

## Reification: Functional Effects

In functional effects, we reify by turning side-effects into values. For example, we might have a simple statement like;

```scala
println("Hello")
println("World")
```

In ZIO we reify this statement to a value like

```scala
val effect1 = Task(println("Hello"))
val effect2 = Task(println("World"))
```

And then we are able to do awesome things like:

```scala
(Task(println("Hello")) zipPar Task(println("World"))).retryN(100)
```

## Reification: Optics

In Scala, we have product types like this case class of a `Person`:

```scala
final case class Person(name: String, age: Int)
```

This case class has two fields:

- A field `name` of type `String`
- A field `age` of type `Int`

The Scala language provides special support to access the fields inside case classes using the dot syntax:

```scala
val person = Person("Michelle", 32)
val name = person.name
val age  = person.age
```

However, this is a "special language feature", it's not "real" like the side effects we've seen in the previous example (`println(..) vs. Task(println(...))`).

Because these basic operations are not "real," we are unable to create an operator that we can use, for example, we cannot combine two fields that are inside a nested structure.

The solution to this kind of problem is called an "Optic". Optics provide a way to access the fields of a case class and nested structures. There are three main types of optics:
- `Lens`: A lens is a way to access a field of a case class.
- `Prism`: A prism is a way to access a field of a nested structure or a collection.
- `Traversal`: A traversal is a way to access all fields of a case class, nested structures or collections.

Optics allow us to take things which are not a first-class **concept**, and turn that into a first-class **value**, namely the concept of
- drilling down into a field inside a case class or
- drilling down into a nested structure.

Once we have a value, we can compose these things together to solve hard problems in functional programming, e.g.
- handling nested case class copies,
- iterations down deep inside on elements of a nested structure or collections

For more information on optics, refer to the [ZIO Optics](https://zio.dev/zio-optics/) documentation.

## Reification: Schema

So far we've looked at how to
- reify side-effects into values (ZIO)
- how to reify accessing and modifying fields inside case classes or arbitrary structures by turning these operations into values as well (Optics)

**ZIO Schema** is now about how to **describe entire data structures using values**.

The "built-in" way in scala on how to describe data structures are `case classes` and `classes`.

For example, assume we have the `Person` data type, like this:

```scala
final case class Person(name: String, age: Int)
```

It has the following information:

- Name of the structure: `Person`
- Fields: `name` and `age`
- Type of the fields: `String` and `Int`
- Type of the structure: `Person`

ZIO Schema tries to reify the concept of structure for datatypes by turning the above information into values.

Not only for case classes, but also for other types like collections, tuples, enumerations etc.

---

## Diffing and Patching

ZIO Schema provides two methods called `diff` and `patch`:

```scala
sealed trait Schema[A] {
  def diff(thisValue: A, thatValue: A): Patch[A]

  def patch(oldValue: A, diff: Patch[A]): scala.util.Either[String, A]
} 
```

The `diff` method takes two values of the same type `A` and returns a `Patch[A]` value that describes the differences between the two values. conversely, the `patch` method takes a value of type `A` and a `Patch[A]` value and returns a new value of type `A` that is the result of applying the patch to the original value.

Here is a simple example that demonstrate the how to use `diff` and `patch`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
}

val oldValue = Person("John", 42)
val newValue = Person("John", 43)

val patch: Patch[Person] =
  Person.schema.diff(oldValue, newValue)

assert(
  Person.schema.patch(oldValue, patch) == Right(newValue)
)
```

---

## Dynamic Data Representation

DynamicValue is a way to describe the entire universe of possibilities for schema values. It does that in a way that we can interact with and introspect the data with its structure (type information). The structure of the data is baked into the data itself.

We can create a `DynamicValue` from a schema and a value using `DynamicValue.fromSchemaAndValue` (or `Schema#toDynamic`). We can turn it back into a typed value using `DynamicValue#toTypedValue`:

```scala
trait DynamicValue {
  def toTypedValue[A](implicit schema: Schema[A]): Either[String, A] =
}

object DynamicValue {
  def fromSchemaAndValue[A](schema: Schema[A], value: A): DynamicValue
}
```

Let's create a simple instance of `Person("John Doe", 42)` and convert it to `DynamicValue`:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
}

val person        = Person("John Doe", 42)
val dynamicPerson = DynamicValue.fromSchemaAndValue(Person.schema, person)
// or we can call `toDynamic` on the schema directly:
// val dynamicPerson = Person.schema.toDynamic(person)
println(dynamicPerson)
```

As we can see, the dynamic value of `person` is the mixure of the data and its structure:

```scala
// The output pretty printed manually
Record(
  Nominal(Chunk(dev,zio,quickstart),Chunk(),Person),
  ListMap(
    name -> Primitive(John Doe,string),
    age -> Primitive(42,int)
  )
)
```

This is in contrast to the relational database model, where the data structure is stored in the database schema and the data itself is stored in a separate location.

However, when we switch to document-based database models, such as JSON or XML, we can store both the data and its structure together. The JSON data model serves as a good example of self-describing data, as it allows us not only to include the data itself but also to add type information within the JSON. In this way, there is no need for a separate schema and data; everything is combined into a single entity.

## Schema: Converting to/from DynamicValue

With a `Schema[A]`, we can convert any value of type `A` to a `DynamicValue` and conversely we can convert it back to `A`:

```scala
sealed trait Schema[A] {
  def toDynamic(value: A): DynamicValue

  def fromDynamic(value: DynamicValue): scala.util.Either[String, A]
}
```

The `toDynamic` operation erases the type information of the value and places it into the value (the dynamic value) itself. The `fromDynamic` operation does the opposite: it takes the type information from the dynamic value and uses it to reconstruct the original value.

Please note that, if we have two types `A` and `B` that are isomorphic, we can convert a dynamic value of type `A` to a typed value of type `B` and vice versa:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = DeriveSchema.gen
}

case class User(name: String, age: Int)

object User {
  implicit val schema: Schema[User] = DeriveSchema.gen
}

val johnPerson = Person("John Doe", 42)
val johnUser   = User("John Doe", 42)

val dynamicJohnPerson = Person.schema.toDynamic(johnPerson)
val dynamicJohnUser   = User.schema.toDynamic(johnUser)

println(dynamicJohnPerson)
// Output: Record(Nominal(Chunk(dev,zio,quickstart),Chunk(Main),Person),ListMap(name -> Primitive(John Doe,string), age -> Primitive(42,int)))
println(dynamicJohnUser)
// Output: Record(Nominal(Chunk(dev,zio,quickstart),Chunk(Main),User),ListMap(name -> Primitive(John Doe,string), age -> Primitive(42,int)))

assert(dynamicJohnPerson.toTypedValue[User] == Right(johnUser))
assert(dynamicJohnUser.toTypedValue[Person] == Right(johnPerson))
```

## Manipulating Dynamic Values

When we turn a typed value `A` into a `DynamicValue`, we can manipulate its structure and data dynamically. For example, we can add a new field to a record or change the type of a field. This process is called dynamic value migration, which we will discuss in the [schema migration](schema-migration.md) section.

---

## ZIO Schema Operations

Once we have defined our schemas, we can use them to perform a variety of operations. In this section, we will explore some of the most common operations that we can perform on schemas.

Before diving into the details, let's see a quick overview of the operations that we can perform on schemas:

```scala
sealed trait Schema[A] {
  self =>

  type Accessors[Lens[_, _, _], Prism[_, _, _], Traversal[_, _]]

  def ? : Schema[Option[A]]

  def <*>[B](that: Schema[B]): Schema[(A, B)]

  def <+>[B](that: Schema[B]): Schema[scala.util.Either[A, B]]

  def defaultValue: scala.util.Either[String, A]

  def annotations: Chunk[Any]

  def ast: MetaSchema 

  def annotate(annotation: Any): Schema[A]

  def coerce[B](newSchema: Schema[B]): Either[String, Schema[B]]

  def diff(thisValue: A, thatValue: A): Patch[A]

  def patch(oldValue: A, diff: Patch[A]): scala.util.Either[String, A]

  def fromDynamic(value: DynamicValue): scala.util.Either[String, A] 

  def makeAccessors(b: AccessorBuilder): Accessors[b.Lens, b.Prism, b.Traversal]

  def migrate[B](newSchema: Schema[B]): Either[String, A => scala.util.Either[String, B]]

  def optional: Schema[Option[A]]

  def ordering: Ordering[A]

  def orElseEither[B](that: Schema[B]): Schema[scala.util.Either[A, B]]

  def repeated: Schema[Chunk[A]]

  def serializable: Schema[Schema[A]]

  def toDynamic(value: A): DynamicValue

  def transform[B](f: A => B, g: B => A): Schema[B]
  
  def transformOrFail[B](f: A => scala.util.Either[String, B], g: B => scala.util.Either[String, A]): Schema[B]

  def validate(value: A)(implicit schema: Schema[A]): Chunk[ValidationError]

  def zip[B](that: Schema[B]): Schema[(A, B)]
}
```

---

## Schema Migration

## Automatic Migration

With ZIO Schema, we can automatically migrate data from one version of a schema to another. As software evolves, we often need to add, change or remove old fields. ZIO Schema provides two methods called `migrate` and `coerce` which help migrate the old schema to the new one:

```scala
sealed trait Schema[A] {
  def migrate[B](newSchema: Schema[B]): Either[String, A => scala.util.Either[String, B]]

  def coerce[B](newSchema: Schema[B]): Either[String, Schema[B]]
}
```

The `migrate` method takes a new schema and returns a function that can migrate values of the old schema to values of the new schema as a `Right` value of `Either`. If the schemas have unambiguous transformations or are incompatible, the method returns a `Left` value containing an error message.

## Manual Migration

By having `DynamicValue` which its type information embedded in the data itself, we can perform migrations of the data easily by applying a sequence of migration steps to the data.

```scala
trait DynamicValue {
  def transform(transforms: Chunk[Migration]): Either[String, DynamicValue]
}
```

The `Migration` is a sealed trait with several subtypes:

```scala
sealed trait Migration
object Migration {
  final case class AddNode(override val path: NodePath, node: MetaSchema) extends Migration

  final case class DeleteNode(override val path: NodePath) extends Migration

  final case class AddCase(override val path: NodePath, node: MetaSchema) extends Migration

  // ...
}
```

Using the `Migration` ADT we can describe the migration steps and then we can apply them to the `DynamicValue`. Let's try a simple example:

```scala

case class Person1(name: String, age: Int)

object Person1 {
  implicit val schema: Schema[Person1] = DeriveSchema.gen
}

case class Person2(name: String)

object Person2 {
  implicit val schema: Schema[Person2] = DeriveSchema.gen
}

val person1: Person1 = Person1("John Doe", 42)

val migrations: Chunk[Migration] = Chunk(DeleteNode(NodePath.root / "age"))

val person2 = DeriveSchema
  .gen[Person1]
  .toDynamic(person1)
  .transform(migrations)
  .flatMap(_.toTypedValue[Person2])
  
assert(person2 == Right(Person2("John Doe")))
```

## Deriving Migrations

ZIO Schema provides a way to derive migrations automatically using the `Migration.derive` operation:

```scala
object Migration {
  def derive(from: MetaSchema, to: MetaSchema): Either[String, Chunk[Migration]]
}
```

It takes two `MetaSchema` values, the old and the new schema, and returns a `Chunk[Migration]` that describes the migrations steps. Let's try a simple example:

```scala

case class Person1(name: String, age: Int, language: String, height: Int)

object Person1 {
  implicit val schema: Schema[Person1] = DeriveSchema.gen
}

case class Person2(
    name: String,
    role: String,
    language: Set[String],
    height: Double
)

object Person2 {
  implicit val schema: Schema[Person2] = DeriveSchema.gen
}

val migrations = Migration.derive(
  MetaSchema.fromSchema(Person1.schema),
  MetaSchema.fromSchema(Person2.schema)
)

println(migrations)
```

The output of the above code is:

```scala
Right(
  Chunk(IncrementDimensions(Chunk(language,item),1),
  ChangeType(Chunk(height),double),
  AddNode(Chunk(role),string),
  DeleteNode(Chunk(age)))
)
```

This output describes a series of migration steps that should be applied to the old schema to be transformed into the new schema.

---

## Serialization of the Schema Itself

In distributed systems, we often need to move computations to data instead of moving data to computations. The data is big and the network is slow, so moving it is expensive and sometimes impossible due to the volume of data. So in distributed systems, we would like to move our functions to the data and apply the data to the functions and gather the results back.

So we need a way to serialize our computations and send them through the network. In ZIO Schema, each schema itself has a schema, so we can treat the structure as pure data! we can serialize our schemas by calling the `serializable` method:

```scala
sealed trait Schema[A] {
  def serializable: Schema[Schema[A]]
}
```

By calling this method, we can get the schema of a schema. So we can serialize it and send it across the wire, and it can be deserialized on the other side. After deserializing it, we have a schema that is isomorphic to the original schema. So all the operations that we can perform on the original type `A`, we can perform on any value that is isomorphic to `A` on the other side.

---

## Getting The Default Value

ZIO Schema provides a method called `defaultValue` that returns the default value of the underlying type described by the schema. This method returns a `scala.util.Either[String, A]` value, where `A` is the type described by the schema. If the schema does not have a default value, the method returns a `Left` value containing an error message. Otherwise, it returns a `Right` value containing the default value:

```scala
sealed trait Schema[A] {
  def defaultValue: scala.util.Either[String, A]
}
```

ZIO Schema have out of the box default values for all standard types, such as `String`, `Int`, `Boolean`, ..., `LocalDateTime` and `UUID`. For example, the default value of a schema for `String` is the empty string, and the default value of a schema for `Int` is `0`.

---

## Transforming Schemas

Using the `Schema#transform` method, we can transform a `Schema[A]` into a `Schema[B]` by supplying two functions that can transform between `A` and `B`.  In normal Scala code this would be the equivalent of `map`, but with isomorphism property.

```scala
object Schema {
  def transform[B](f: A => B, g: B => A): Schema[B] = ???
}
```

Therefore, if we have a schema for `A`, and isomorphism between `A` and `B`, we can derive a schema for `B` in terms of `Schema[A].

:::note
In type theory, isomorphism refers to a relationship between two types that have a bijective correspondence or mapping between their elements. More specifically, if two types, let's say Type `A` and Type `B`, are isomorphic, it means that there exists a pair of functions—one going from A to B (often called the forward function) and another going from B to A (often called the backward function)—that satisfy certain properties.
:::

In ZIO Schema this is modelled by the `Transform` type class:

```scala
object Schema {
  final case class Transform[A, B](
      codec: Schema[A],
      f: A => Either[String, B],
      g: B => Either[String, A]
    ) extends Schema[B]
}  
```

For example, assume we have a wrapper class `Age` that wraps an `Int` value, and it has some validation logic, e.g. the age must be between 0 and 120. We can define a `Schema[Age]` by using the `Schema.transform` method:

```scala

case class Age(i: Int)

object Age {
  implicit val schema: Schema[Age] =
    Schema[Int].transformOrFail(
      (i: Int) =>
        if (i >= 0 && i <= 120)
          Right(Age(i))
        else
          Left("Age must be between 1 and 120"),
      (age: Age) => Right(age.i)
    )
}
```

---

## Validation(Operations)

When we create a schema for a type, we can also specify validation rules for the type. Validations are a way to ensure that the data conforms to certain rules.

Using `Schema#validate` we can validate a value against the validation rules of its schema:

```scala
trait Schema[A] {
  def validate(value: A)(implicit schema: Schema[A]): Chunk[ValidationError]
}
```

Let's write a schema for the `Person` case class and add validation rules to it. For example, we can specify that the `age` field must be greater than 0 and less than 120 and the `name` field must be non-empty:

```scala

case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person] = CaseClass2(
    id0 = TypeId.fromTypeName("Person"),
    field01 = Schema.Field(
      name0 = "name",
      schema0 = Schema[String],
      validation0 = Validation.minLength(1),
      get0 = (p: Person) => p.name,
      set0 = { (p: Person, s: String) => p.copy(name = s) }
    ),
    field02 = Schema.Field(
      name0 = "age",
      schema0 = Schema[Int],
      validation0 = Validation.between(0, 120),
      get0 = (p: Person) => p.age,
      set0 = { (p: Person, age: Int) => p.copy(age = age) }
    ),
    construct0 = (name, age) => Person(name, age),
    annotations0 = Chunk.empty
  )
}
```

Both fields of the `Person` case class have validation rules. Let's see what happens when we try to validate a `Person` value that does not conform to the validation rules:

```scala

val result: Chunk[ValidationError] = Person.schema.validate(Person("John Doe", 130))
println(result)
// Output: 
// Chunk(EqualTo(130,120),LessThan(130,120))
```

Due to the failed validation rules, a list of the specific rules that were not met is generated. In this case, it indicates that the age is not equal, or less than 120.

---

## Standard Type Reference


ZIO Schema provides a number of built-in primitive types, that we can use to represent our data. These can be seen in the following table:

|Standard Type|JVM Support|ScalaJS Support|Scala Native Support|
|--------------|:--------------:|:--------------:|:--------------:|
|`Boolean`|✅|✅|✅|
|`Byte`|✅|✅|✅|
|`Char`|✅|✅|✅|
|`Chunk[Byte]`|✅|✅|✅|
|`Double`|✅|✅|✅|
|`Float`|✅|✅|✅|
|`Int`|✅|✅|✅|
|`Long`|✅|✅|✅|
|`Short`|✅|✅|✅|
|`String`|✅|✅|✅|
|`Unit`|✅|✅|✅|
|`java.math.BigDecimal`|✅|✅|✅|
|`java.math.BigInteger`|✅|✅|✅|
|`java.time.DayOfWeek`|✅|✅|✅|
|`java.time.Duration`|✅|✅|✅|
|`java.time.Instant`|✅|✅|✅|
|`java.time.LocalDate`|✅|✅|✅|
|`java.time.LocalDateTime`|✅|✅|✅|
|`java.time.LocalTime`|✅|✅|✅|
|`java.time.Month`|✅|✅|✅|
|`java.time.MonthDay`|✅|✅|✅|
|`java.time.OffsetDateTime`|✅|✅|✅|
|`java.time.OffsetTime`|✅|✅|✅|
|`java.time.Period`|✅|✅|✅|
|`java.time.Year`|✅|✅|✅|
|`java.time.YearMonth`|✅|✅|✅|
|`java.time.ZoneId`|✅|✅|✅|
|`java.time.ZoneOffset`|✅|✅|✅|
|`java.time.ZonedDateTime`|✅|✅|✅|
|`java.util.Currency`|✅|❌|❌|
|`java.util.UUID`|✅|❌|✅|

---

## ZIO Schema Use cases

ZIO Schema allows us to create representations of our data types as values.

Once we have a representation of our data types, we can use it to
  - Serialize and deserialize our types
  - Validate our types
  - Transform our types
  - Create instances of your types

We can then use one of the various codecs (or create our own) to serialize and deserialize your types.

Example of possible codecs are:

  - CSV Codec
  - JSON Codec (already available)
  - Apache Avro Codec (in progress)
  - Apache Thrift Codec (in progress)
  - XML Codec
  - YAML Codec
  - Protobuf Codec (already available)
  - QueryString Codec
  - etc.

Example use cases that are possible:

  - Serializing and deserializing JSON
  - Serializing and deserializing XML
  - Validating JSON
  - Validating XML
  - Transforming JSON
  - Transforming XML
  - Transforming JSON to XML
  - Transforming XML to JSON
  - Creating diffs from arbitrary data structures
  - Creating migrations / evolutions e.g. of Events used in Event-Sourcing
  - Transformation pipelines, e.g.
      1. Convert from protobuf to object, e.g. `PersonDTO`,
      2. Transform to another representation, e.g. `Person`,
      3. Validate
      4. Transform to JSON `JsonObject`
      5. Serialize to `String`

---

## Introduction to ZIO SQS

This library is a [ZIO](https://github.com/zio/zio)-powered client for AWS SQS. It is built on top of the [AWS SDK for Java 2.0](https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/basics.html) via the automatically generated wrappers from [zio-aws](https://github.com/vigoo/zio-aws).

## Add the dependency

To use `zio-sqs`, add the following line in your `build.sbt` file:

```
libraryDependencies += "dev.zio" %% "zio-sqs" % "0.5.0"
```

## How to use

In order to use the connector, you need to provide your program with a configured SQS client as an `Sqs` ZLayer. You can use `io.github.vigoo.zioaws.sqs.live` to use default AWS SDK settings or use `.customized` (refer to the [AWS SDK Documentation](https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/creating-clients.html) if you need help customizing it). See also the [ZIO documentation](https://zio.dev/docs/howto/howto_use_layers) on how to use layers.

### Publish messages

Use `Producer.make` to instantiate an instance of `Producer` trait that can be used to publish objects of type `T` to the queue.

```scala
def make[R, T](
    queueUrl: String,
    serializer: Serializer[T],
    settings: ProducerSettings = ProducerSettings()
  ): ZManaged[R with Sqs with Clock, Throwable, Producer[T]]
```

where:

- `queueUrl: String` - an SQS queue URL
- `serializer: Serializer[T]` - an instance of `zio.sqs.serialization.Serializer` that can be used to convert an object of type `T` to a string.

  ```scala
    trait Serializer[T] {
      def apply(t: T): String
    }
  ```
  If a published message is already a string, `Serializer.serializeString` can be used.
- `settings: ProducerSettings` - a set of settings (`ProducerSettings`) used to configure the producer.
    - `batchSize: Int` - The size of the batch to use, [1-10] (default: 10).
    - `duration: Duration` - Time to wait for the batch to be full (have the specified batchSize) (default: 500 milliseconds).
    - `parallelism: Int` - The number of concurrent requests to make to SQS (default: 16).
    - `retryDelay: Duration` - Time to wait before retrying event republishing if it failed with a recoverable error (default: 250 milliseconds).
      The errors returned from SQS could either recoverable or not. An example of recoverable error -- when the server returned the code: `ServiceUnavailable`
    - `retryMaxCount: Int` - The number of retries to make for a posted event (default: 10).

#### Producer

`Producer` contains two set of methods:
- methods that fail the resulting *Task* or *Stream* if SQS server returns an error for a published event.
    - `def produce(e: ProducerEvent[T]): Task[ProducerEvent[T]]` - Publishes a single event and fails the task.
      Fails the `Task` if the server returns an error.
    - `def produceBatch(es: Iterable[ProducerEvent[T]]): Task[List[ProducerEvent[T]]]` - Publishes a batch of events.
      Fails the `Task` if the server returns an error for any of the provided events.
    - `def sendStream: Stream[Throwable, ProducerEvent[T]] => ZStream[Any, Throwable, ProducerEvent[T]]` - Stream that takes the events and produces a stream with published events.
      Fails if the server returns an error for any of the published events.
    - `def sendSink: ZSink[Any, Throwable, Nothing, Iterable[ProducerEvent[T]], Unit]` - Sink that can be used to publish events.
      Fails if the server returns an error for any of the published events.

- methods that do not fail the operation but return `ErrorOrEvent[T]` (defied as `Either[ProducerError[T], ProducerEvent[T]]`).
    - `def sendStreamE: Stream[Throwable, ProducerEvent[T]] => ZStream[Any, Throwable, ErrorOrEvent[T]]` - Stream that takes the events and produces a stream with the results.
      Doesn't fail if the server returns an error for any of the published events.
    - `def produceBatchE(es: Iterable[ProducerEvent[T]]): Task[List[ErrorOrEvent[T]]]` - Publishes a batch of events. Completes when all input events were processed (published to the server or failed with an error).
      Doesn't fail the `Task` if the server returns an error for any of the provided events.

Producer tries to accumulate messages in batches and send them to the server.
If messages should be sent one by one and batching is not expected, set `ProducerSettings.batchSize` to `1`.

#### ProducerEvent

`ProducerEvent[T]` is an event that is published to SQS and contains the following parameters that could be configured:
- `data: T` - Object to publish to SQS. A serializer for this type should be provided when a `Producer` is instantiated.
- `attributes: Map[String, MessageAttributeValue]` - A map of [attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-attributes.html) to set.
- `groupId: Option[String]` - Assigns a specific [message group](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html) to the message.
- `deduplicationId: Option[String]` - Token used for [deduplication](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html) of sent messages.

If a plain string should be published without any additional attributes a `ProducerEvent` can be created directly:

```scala
val str: String = "message to publish"
val event: ProducerEvent = ProducerEvent(str)
```

#### ProducerError

`ProducerError[T]` represents an [error details]((https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_BatchResultErrorEntry.html)) that were returned from the server.
- `senderFault: Boolean` - Specifies whether the error happened due to the caller of the batch API action.
- `code: String` - An error code representing why the action failed on this entry.
- `message: Option[String]` - A message explaining why the action failed on this entry.
- `event: ProducerEvent[T]` - An event that triggered this error on the server.

#### Publish Example

```scala

object PublishExample extends zio.App {

  val client: ZLayer[Any, Throwable, Sqs] = zioaws.netty.default >>>
    zioaws.core.config.default >>>
    zioaws.sqs.live

  val events                                                = List("message1", "message2").map(ProducerEvent(_))
  val queueName                                             = "TestQueue"
  val program: RIO[Clock with Sqs, Either[Throwable, Unit]] = for {
    queueUrl    <- Utils.getQueueUrl(queueName)
    producer     = Producer.make(queueUrl, Serializer.serializeString)
    errOrResult <- producer.use(p => p.sendStream(ZStream(events: _*)).runDrain.either)
  } yield errOrResult

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    program.provideCustomLayer(client).exitCode
}
```

### Consume messages

Use `SqsStream.apply` to get a stream of messages from a queue. It returns a ZIO `Stream` that you can consume with all the operators available.

```scala
def apply(
  queueUrl: String,
  settings: SqsStreamSettings = SqsStreamSettings()
): ZStream[Sqs, Throwable, Message]
```

`SqsStreamSettings` allows your to configure a number of things:

- `autoDelete`: if `true`, messages will be automatically deleted from the queue when they're consumed by the stream, if `false` you have to delete them explicitly by calling `SqsStream.deleteMessage` (default `true`)
- `stopWhenQueueEmpty`: if `true` the stream will close when there the queue is empty, if `false` the stream will go on forever (default `false`)
- `attributeNames`: see the [related page on AWS docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html)
- `maxNumberOfMessages`: number of messages to query at once from SQS (default `1`)
- `messageAttributeNames`: see the [related page on AWS docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html)
- `visibilityTimeout`: see the [related page on AWS docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) (default `Some(30)`. If set to `None`, the queue's value will be used.)
- `waitTimeSeconds`: see the [related page on AWS docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) (default `Some(20)`. If set to `None`, the queue's value will be used.)

**Example:**

```scala

SqsStream(
  queueUrl,
  SqsStreamSettings(stopWhenQueueEmpty = true, waitTimeSeconds = Some(3))
).foreach(msg => UIO(println(msg.body)))
```

### Full example

```scala

object TestApp extends zio.App {
  val queueName = "TestQueue"

  val client: ZLayer[Any, Throwable, Sqs] = zioaws.netty.default ++
    ZLayer.succeed(
      CommonAwsConfig(
        region = Some(Region.of("ap-northeast-2")),
        credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create("key", "key")),
        endpointOverride = None,
        commonClientConfig = None
      )
    ) >>>
    zioaws.core.config.configured() >>>
    zioaws.sqs.live

  val program: RIO[Sqs with Clock, Unit] = for {
    _        <- Utils.createQueue(queueName)
    queueUrl <- Utils.getQueueUrl(queueName)
    producer  = Producer.make(queueUrl, Serializer.serializeString)
    _        <- producer.use { p =>
                  p.produce(ProducerEvent("hello"))
                }
    _        <- SqsStream(
                  queueUrl,
                  SqsStreamSettings(stopWhenQueueEmpty = true, waitTimeSeconds = Some(3))
                ).foreach(msg => UIO(println(msg.body)))
  } yield ()

  override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] =
    program.provideCustomLayer(client).exitCode
}
```

---

## Compression and archives with zio-streams

[ZIO Streams Compress](https://github.com/zio/zio-streams-compress) integrates several compression algorithms and
archive formats with [ZIO Streams](https://zio.dev).

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-streams-compress/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-streams-compress-docs_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-streams-compress-docs_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-streams-compress-docs_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-streams-compress-docs_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-streams-compress-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-streams-compress-docs_2.13) [![ZIO Streams Compress docs](https://img.shields.io/github/stars/zio/zio-streams-compress?style=social)](https://github.com/zio/zio-streams-compress) [![Scala Steward badge](https://img.shields.io/badge/Scala_Steward-helping-blue.svg?style=flat&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAAQCAMAAAARSr4IAAAAVFBMVEUAAACHjojlOy5NWlrKzcYRKjGFjIbp293YycuLa3pYY2LSqql4f3pCUFTgSjNodYRmcXUsPD/NTTbjRS+2jomhgnzNc223cGvZS0HaSD0XLjbaSjElhIr+AAAAAXRSTlMAQObYZgAAAHlJREFUCNdNyosOwyAIhWHAQS1Vt7a77/3fcxxdmv0xwmckutAR1nkm4ggbyEcg/wWmlGLDAA3oL50xi6fk5ffZ3E2E3QfZDCcCN2YtbEWZt+Drc6u6rlqv7Uk0LdKqqr5rk2UCRXOk0vmQKGfc94nOJyQjouF9H/wCc9gECEYfONoAAAAASUVORK5CYII=)](https://scala-steward.org)

## Installation

In order to use this library, we need to add one of the following line in our `build.sbt` file:

```sbt
libraryDependencies += "dev.zio" %% "zio-streams-compress-brotli" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-brotli4j" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-bzip2" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-gzip" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-lz4" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-snappy" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-tar" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-zip" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-zip4j" % "2.1.3"
libraryDependencies += "dev.zio" %% "zio-streams-compress-zstd" % "2.1.3"
```

_As of 2.2.0, zio-streams-compress requires zio 2.1.25 or later._

For Brotli you can choose between the 'brotli' and the 'brotli4j' version. The first is based on the official Java
library but only does decompression. The second is based on [Brotli4J](https://github.com/hyperxpro/Brotli4j) which does
compression and decompression.

For ZIP files you can choose between the 'zip' and the 'zip4j' version. The first allows you to tweak the compression
level, while the second allows you work with password-protected ZIP files.

Currently only jvm is supported. PRs for scala-js and scala-native are welcome.

### Example

```scala
// Example.sc
// Run with: scala-cli Example.sc
//> using dep dev.zio:zio-streams-compress-gzip:2.1.3
//> using dep dev.zio:zio-streams-compress-tar:2.1.3
//> using dep dev.zio:zio-streams-compress-zip4j:2.1.3

object ExampleApp extends ZIOAppDefault {
  override def run: ZIO[Any, Any, Any] =
    for {
      // Compress a file with GZIP
      _ <- ZStream
             .fromFileName("file")
             .via(GzipCompressor.compress)
             .run(ZSink.fromFileName("file.gz"))

      // List all items in a gzip tar archive:
      _ <- ZStream
             .fromFileName("file.tgz")
             .via(GzipDecompressor.decompress)
             .via(TarUnarchiver.unarchive)
             .mapZIO { case (archiveEntry, contentStream) =>
               for {
                 content <- contentStream.runCollect
                 _ <- Console.printLine(s"${archiveEntry.name} ${content.length}")
               } yield ()
             }
             .runDrain

      // Create an encrypted ZIP archive
      _ <- ZStream(archiveEntry("file1.txt", "Hello world!".getBytes(UTF_8)))
             .via(Zip4JArchiver(password = Some("it is a secret")).archive)
             .run(ZSink.fromFileName("file.zip"))
    } yield ()

  private def archiveEntry(
    name: String,
    content: Array[Byte],
  ): (ArchiveEntry[Some, Any], ZStream[Any, Throwable, Byte]) =
    (ArchiveEntry(name, Some(content.length.toLong)), ZStream.fromIterable(content))

}
```

## Unarchiving

As of zio-streams-compress 2.0, all unarchivers require the consumer to fully read the entry's contents before the next
archive entry is emitted.

⚠️ Not reading the entry's content, causes the unarchive pipeline to lock up and halt your program.

In zio-streams-compress 1.x, consuming the next archive entry, even by accident for example by using buffering or
rechunking, corrupts the content stream of all archive entries. With the new requirement the unarchiver can continue to
operate in a streaming fashion without corrupting content.

Most of the time the new requirement should not be an issue. For example, you can concatenate 'unarchive' and 'archive'
pipelines without problems.

Here are two cases where the requirement can be issue:

1. You are only interested in the metadata of the archive entries. In this case you can use the unarchiver's `list`
   method which drains the entry's contents behind the scenes.

2. Buffering, aggregation or rechunking is needed to improve throughput.
   In this case you can first slurp the content in memory, and then do the buffering/aggregation/rechunking.
   As the whole entry will be pulled into memory, checking the size is prudent. Be aware that the reported size might
   not be available, or may even be maliciously incorrect! Here is an example that loads at most `maxEntrySize` bytes
   per entry:

```scala
ZStream
  .fromFileName("file.zip")
  .via(ZipUnarchiver.unarchive)
  .mapZIO { case (archiveEntry, contentStream) =>
    if (entry.uncompressedSize.exists(_ > maxEntrySize)) ZIO.fail("archive entry too large")
    else contentStream.take(maxEntrySize + 1).runCollect.flatMap { content =>
      if (content.size > maxEntrySize) ZIO.fail("archive entry too large")
      else ZIO.succss((archiveEntry, content))
    }
  }
  .rechunk(10) // here it is safe to rechunk
```

---

## Introduction to ZIO Telemetry

[ZIO telemetry](https://github.com/zio/zio-telemetry) is purely-functional and type-safe. It provides clients for
[OpenTracing](https://opentracing.io/), [OpenCensus](https://opencensus.io/) and [OpenTelemetry](https://opentelemetry.io/).

[![Production Ready](https://img.shields.io/badge/Project%20Stage-Production%20Ready-brightgreen.svg)](https://github.com/zio/zio/wiki/Project-Stages) ![CI Badge](https://github.com/zio/zio-telemetry/workflows/CI/badge.svg) [![Sonatype Releases](https://img.shields.io/nexus/r/https/oss.sonatype.org/dev.zio/zio-opentracing_2.13.svg?label=Sonatype%20Release)](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-opentracing_2.13/) [![Sonatype Snapshots](https://img.shields.io/nexus/s/https/oss.sonatype.org/dev.zio/zio-opentracing_2.13.svg?label=Sonatype%20Snapshot)](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-opentracing_2.13/) [![javadoc](https://javadoc.io/badge2/dev.zio/zio-telemetry-docs_2.13/javadoc.svg)](https://javadoc.io/doc/dev.zio/zio-telemetry-docs_2.13) [![ZIO Telemetry](https://img.shields.io/github/stars/zio/zio-telemetry?style=social)](https://github.com/zio/zio-telemetry)

ZIO Telemetry consists of the following projects:

- [OpenTracing](opentracing.md)
- [OpenCensus](opencensus.md)
- [OpenTelemetry](opentelemetry.md)

## Introduction

In monolithic architecture, everything is in one place, and we know when a request starts and then how it goes through 
the components and when it finishes. We can obviously see what is happening with our request and where is it going. 
But, in distributed systems like microservice architecture, we cannot find out the story of a request through various 
services easily. This is where distributed tracing comes into play.

ZIO Telemetry is a purely functional client which helps up propagate context between services in a distributed environment.

## Installation

In order to use this library, we need to add the following line in our `build.sbt` file if we want to use [OpenTelemetry](https://opentelemetry.io/) client:

```scala
libraryDependencies += "dev.zio" %% "zio-opentelemetry" % "<version>"
```

If you're using [ZIO Logging](https://github.com/zio/zio-logging) you can combine OpenTelemetry with ZIO Logging using:

```scala
libraryDependencies += "dev.zio" %% "zio-opentelemetry-zio-logging" % "<version>"
```

For using [OpenTracing](https://opentracing.io/) client we should add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-opentracing" % "<version>"
```

And for using [OpenCensus](https://opencensus.io/) client we should add the following line in our `build.sbt` file:

```scala
libraryDependencies += "dev.zio" %% "zio-opencensus" % "<version>"
```

## Examples

You can find examples with full source code and instructions of how to run by following the links:
- [OpenTelemetry Example](opentelemetry-example.md)
- [OpenTelemetry Instrumentation Example](opentelemetry-instrumentation-example.md)
- [OpenTelemetry ZIO Logging Example](opentelemetry-zio-logging.md)
- [OpenTracing Example](opentracing-example.md)

## Articles

- [Trace your microservices with ZIO](https://kadek-marek.medium.com/trace-your-microservices-with-zio-telemetry-5f88d69cb26b) by Marek Kadek (September 2021)

---

## OpenCensus

OpenCensus is a stats collection and distributed tracing framework. Well known implementations are [Jaeger](https://www.jaegertracing.io)
and [Zipkin](https://www.zipkin.io).

## Installation

First, add the following dependency to your build.sbt:
```
"dev.zio" %% "zio-opencensus" % "<version>"
```

## Usage

To use ZIO Telemetry, you will need a `Tracing` service in your environment. You also need to provide a `tracer` implementation:

```scala

val tracerLayer = ZLayer.succeed(io.opencensus.trace.Tracing.getTracer)

val errorMapper = ErrorMapper[Throwable] { case _ => Status.UNKNOWN }

val app =
  ZIO.serviceWithZIO[Tracing] { tracing =>

    (for {
      _       <- tracing.putAttributes(Map("foo" -> "bar"))
      message <- Console.readline
    } yield message) @@ root("/app")
  }.provide(Tracing.live, tracerLayer)
```

After importing `import tracing.aspects._`, additional `ZIOAspect` combinators
on `ZIO`s are available to support starting child spans and adding attributes.

```scala
ZIO.serviceWithZIO[Tracing] { tracing =>

  // start a new root span and set some attributes
  val zio1 = ZIO.unit @@ root("root span", attributes = ("foo", "bar))
  // start a child of the current span
  val zio2 = ZIO.unit @@ span("child span", attributes = Map.empty)
}
```

To propagate contexts across process boundaries, extraction and injection can be
used. The current span context is injected into a carrier, which is passed
through some side channel to the next process. There it is injected back and a
child span of it is started.

Due to the use of the (mutable) OpenCensus carrier APIs, injection and extraction
are not referentially transparent.

```scala
ZIO.serviceWithZIO[Tracing] { tracing =>

  val textFormat                           = Tracing.getPropagationComponent().getB3Format()
  val carrier: mutable.Map[String, String] = mutable.Map().empty

  val getter: TextFormat.Getter[mutable.Map[String, String]] = new TextFormat.Getter[mutable.Map[String, String]] {
    override def keys(carrier: mutable.Map[String, String]): lang.Iterable[String] =
      carrier.keys.asJava

    override def get(carrier: mutable.Map[String, String], key: String): String =
      carrier.get(key).orNull
  }

  val setter: TextFormat.Setter[mutable.Map[String, String]] = new TextFormat.Setter[mutable.Map[String, String]] {
    override def put(carrier: mutable.Map[String, String], key: String, value: String): Unit =
      carrier.update(key, value)
  }
  
  val zio1 = tracing.inject(textFormat, carrier, setter) @@ 
    root("root span", attributes = Map.empty)
  
  val zio2 = ZIO.unit @@ 
    span("child span", attributes = Map(("foo", "bar"))) @@ 
    fromRootSpan(textFormat, carrier, getter, "foo", attributes = Map.empty)
}
```

---

## OpenTelemetry Example

You can find the source code [here](https://github.com/zio/zio-telemetry/tree/series/2.x/opentelemetry-example).

For an explanation in more detail, check the [OpenTracing Example](opentracing-example.md).

We're going to show an example of how to pass contextual information using [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) and collect traces, metrics, and logs.

# Run

## Print OTEL signals to console

By default the example code uses [OTLP Logging Exporters](https://github.com/open-telemetry/opentelemetry-java/tree/main/exporters/logging-otlp) to print all signals to stdout in OTLP JSON encoding. This means that you can run the application immediately and observe the results.

For this, you need to run proxy and backend parts of application in different terminals via sbt.

Run proxy:
```bash
sbt "opentelemetryExample/runMain zio.telemetry.opentelemetry.example.ProxyApp"
```
Run backend:
```bash
sbt "opentelemetryExample/runMain zio.telemetry.opentelemetry.example.BackendApp"
```
Now perform the following request to see the results immediately:
```bash
curl -X GET http://localhost:8080/statuses
```

## Publish OTEL signals to other observability platforms

In case you want to try different observability platforms such as [Jaeger](https://www.jaegertracing.io/), [Fluentbit](https://fluentbit.io/), [Seq](https://datalust.co/seq), [DataDog](https://www.datadoghq.com/), [Honeycomb](https://www.honeycomb.io/) or others, please change the [OtelSdk.scala](https://github.com/zio/zio-telemetry/blob/series/2.x/opentelemetry-example/src/main/scala/zio/telemetry/opentelemetry/example/otel/OtelSdk.scala) file by choosing from the available tracer, meter, and logger providers or by implementing your own. 

## Publish OTEL signals to Jaeger and Seq

We chose [Jaeger](https://www.jaegertracing.io/) for distributed traces and [Seq](https://datalust.co/seq) to store logs to demonstrate how the library works with available open-source observability platforms.

Start Jaeger by running the following command:
```bash
docker run --rm -it \
  -d \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 4317:4317 \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.47
```

To run Seq, you also need to specify an admin password (user is `admin`):
```bash
PH=$(echo 'admin123' | docker run --rm -i datalust/seq config hash)

docker run \
  -d \
  --restart unless-stopped \
  -e ACCEPT_EULA=Y \
  -e SEQ_FIRSTRUN_ADMINPASSWORDHASH="$PH" \
  -p 80:80 \
  -p 5341:5341 \
  datalust/seq
```

You must also switch the Tracing and Logging providers to Jaeger and Seq. For this, you need to swap the `stdout` providers we use by default in [OtelSdk.scala](https://github.com/zio/zio-telemetry/blob/series/2.x/opentelemetry-example/src/main/scala/zio/telemetry/opentelemetry/example/otel/OtelSdk.scala#L10) to [TracerProvider.jaeger](https://github.com/zio/zio-telemetry/blob/series/2.x/opentelemetry-example/src/main/scala/zio/telemetry/opentelemetry/example/otel/TracerProvider.scala#L37) and [LoggerProvider.seq](https://github.com/zio/zio-telemetry/blob/series/2.x/opentelemetry-example/src/main/scala/zio/telemetry/opentelemetry/example/otel/LoggerProvider.scala#L36)

Run the application and fire a curl request as shown above. Head over to [Jaeger UI](http://localhost:16686/) and [Seq UI](http://localhost:80/) to see the result.

---

## OpenTelemetry Automatic Instrumentation Example

You can find the source code [here](https://github.com/zio/zio-telemetry/tree/series/2.x/opentelemetry-instrumentation-example).

Firstly, we need to start the observability backends ([Jaeger](https://www.jaegertracing.io/) and [Seq](https://datalust.co/seq))

Start Jaeger by running the following command:
```bash
docker run --rm -it \
  -d \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 14250:14250 \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:1.47
 ```

To run Seq, you also need to specify an admin password (user is `admin`):
```bash
PH=$(echo 'admin123' | docker run --rm -i datalust/seq config hash)

docker run \
  -d \
  --restart unless-stopped \
  -e ACCEPT_EULA=Y \
  -e SEQ_FIRSTRUN_ADMINPASSWORDHASH="$PH" \
  -p 80:80 \
  -p 5341:5341 \
  datalust/seq
```

After this, we can kick off our application to generate some metrics.

For this, we have to download OpenTelemetry JVM agent JAR:
```bash
OTEL_AGENT_PATH=$(cs fetch --classpath "io.opentelemetry.javaagent:opentelemetry-javaagent:latest.release")
 ```

Then start the server application
```bash
sbt -J-javaagent:$OTEL_AGENT_PATH \
    -J-Dotel.service.name=example-server \
    -J-Dotel.traces.sampler=always_on \
    -J-Dotel.traces.exporter=otlp \
    -J-Dotel.logs.exporter=otlp \
    -J-Dotel.exporter.otlp.logs.protocol="http/protobuf" \
    -J-Dotel.exporter.otlp.logs.endpoint="http://localhost:5341/ingest/otlp/v1/logs" \
    -J-Dotel.metrics.exporter=none \
    "opentelemetryInstrumentationExample/runMain zio.telemetry.opentelemetry.instrumentation.example.ServerApp"
 ```

and the client application which will send one request to the server application
```bash
sbt -J-javaagent:$OTEL_AGENT_PATH \
    -J-Dotel.service.name=example-client \
    -J-Dotel.traces.sampler=always_on \
    -J-Dotel.traces.exporter=otlp \
    -J-Dotel.metrics.exporter=none \
    "opentelemetryInstrumentationExample/runMain zio.telemetry.opentelemetry.instrumentation.example.ClientApp"
 ```

Head over to [Jaeger UI](http://localhost:16686/) and [Seq UI](http://localhost:80/) to see the result.

---

## OpenTelemetry ZIO Logging

`zio-opentelemetry` logging facilities are implemented around OpenTelemetry Logging.

In order to use `zio-opentelemetry` feature with `zio-logging` you should use `zio-opentelemetry-zio-logging` module.

`OpenTelemetry ZIO Logging` contains utilities for combining ZIO Opentelemetry with ZIO Logging

## Installation

```scala
"dev.zio" %% "zio-opentelemetry-zio-logging" % "<version>"
```

## Features

### Log formats

This library implements [Log Format](https://zio.dev/zio-logging/formatting-log-records) for span information (`spanId` and `traceId`). 
To use them you need a `LogFormats` service in the environment. For this, use the `ZioLogging.logFormats` layer which in turn required a suitable `ContextStorage` implementation.

```scala
//> using scala "3.4.2"
//> using dep dev.zio::zio:2.1.7
//> using dep dev.zio::zio-opentelemetry:3.0.0-RC24
//> using dep dev.zio::zio-opentelemetry-zio-logging:3.0.0-RC24
//> using dep io.opentelemetry:opentelemetry-sdk:1.40.0
//> using dep io.opentelemetry:opentelemetry-sdk-trace:1.40.0
//> using dep io.opentelemetry:opentelemetry-exporter-logging-otlp:1.40.0
//> using dep io.opentelemetry.semconv:opentelemetry-semconv:1.22.0-alpha

object ZioLoggingApp extends ZIOAppDefault {

  val instrumentationScopeName = "dev.zio.LoggingApp"
  val resourceName             = "logging-app"

  // Prints to stdout in OTLP Json format
  val stdoutLoggerProvider: RIO[Scope, SdkLoggerProvider] =
    for {
      logRecordExporter  <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingLogRecordExporter.create()))
      logRecordProcessor <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleLogRecordProcessor.create(logRecordExporter)))
      loggerProvider     <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkLoggerProvider
              .builder()
              .setResource(Resource.create(Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addLogRecordProcessor(logRecordProcessor)
              .build()
          )
        )
    } yield loggerProvider

  // Prints to stdout in OTLP Json format
  val stdoutTracerProvider: RIO[Scope, SdkTracerProvider] =
    for {
      spanExporter   <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingSpanExporter.create()))
      spanProcessor  <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleSpanProcessor.create(spanExporter)))
      tracerProvider <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkTracerProvider
              .builder()
              .setResource(Resource.create(Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addSpanProcessor(spanProcessor)
              .build()
          )
        )
    } yield tracerProvider

  val otelSdkLayer: TaskLayer[api.OpenTelemetry] =
    OpenTelemetry.custom(
      for {
        tracerProvider <- stdoutTracerProvider
        loggerProvider <- stdoutLoggerProvider
        sdk            <- ZIO.fromAutoCloseable(
                            ZIO.succeed(
                              OpenTelemetrySdk
                                .builder()
                                .setTracerProvider(tracerProvider)
                                .setLoggerProvider(loggerProvider)
                                .build()
                            )
                          )
      } yield sdk
    )

  // Setup zio-logging with spanId and traceId labels
  val loggingLayer: URLayer[LogFormats, Unit] = ZLayer {
    for {
      logFormats     <- ZIO.service[LogFormats]
      format          =
        timestamp.fixed(32) |-|
          level |-|
          label("message", quoted(line)) |-|
          logFormats.spanIdLabel |-|
          logFormats.traceIdLabel
      myConsoleLogger = console(format.highlight)
    } yield Runtime.removeDefaultLoggers >>> myConsoleLogger
  }.flatten

  override def run =
    ZIO
      .serviceWithZIO[Tracing] { tracing =>
        val logic = for {
          // Read user input
          message <- Console.readLine
          // Print span and trace ids along with message
          _       <- ZIO.logInfo(s"User message: $message")
        } yield ()

        // All log messages produced by `logic` will be correlated with a "root_span" automatically
        logic @@ tracing.aspects.root("root_span")
      }
      .provide(
        otelSdkLayer,
        OpenTelemetry.logging(instrumentationScopeName),
        OpenTelemetry.tracing(instrumentationScopeName),
        OpenTelemetry.contextZIO,
        ZioLogging.logFormats,
        loggingLayer
      )

}
```

---

## OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data for analysis in order to understand your software's performance and behavior. Well known implementations are [Jaeger](https://www.jaegertracing.io) and [Zipkin](https://www.zipkin.io).

The library provides an idiomatic ZIO 2.0 interface to OpenTelemetry, ensuring seamless interoperability with the native ZIO capabilities and beyond.

Some of the key features:

- **ZIO native** - Pleasant API that leverages native ZIO features, such as [Resource Management](https://zio.dev/reference/resource/), [Depenency Injection](https://zio.dev/reference/di/), [Streaming](https://zio.dev/reference/stream/), [Logging](https://zio.dev/reference/observability/logging), [Metrics](https://zio.dev/reference/observability/metrics/), and [ZIO Aspect](https://zio.dev/reference/core/zio/#zio-aspect)
- **OpenTelemetry Java SDK and ZIO Runtime interoperability** - Protecting users from directly engaging in OTEL context manipulations, offering a straightforward and clear interface for instrumenting spans, metrics, logs, and baggage. In this scenario, the ZIO effect serves as the span's scope.
- **Seamless signals correlation** -  Automatically correlates spans, metrics, and logs with a surrounding span.
- **Integration with ZIO capabilities** - Propagation of log annotations, metrics, and other data from the ZIO runtime as OTEL attributes and metric signals.

## Installation

Add the following dependency to your `build.sbt` to use OpenTelemetry inside your ZIO application:

```scala
"dev.zio" %% "zio-opentelemetry" % "<version>"
```

You will also need SDK dependencies to be able to provide configured instances of [Tracer](https://javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/trace/Tracer.html), [Meter](https://javadoc.io/static/io.opentelemetry/opentelemetry-api/1.33.0/io/opentelemetry/api/metrics/Meter.html), and [Logger](https://javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/logs/Logger.html), such as:

```scala
"io.opentelemetry"         % "opentelemetry-sdk"           % <opentelemetry-java-version>
"io.opentelemetry"         % "opentelemetry-exporter-otlp" % <opentelemetry-java-version>
"io.opentelemetry.semconv" % "opentelemetry-semconv"       % <opentelemetry-java-version>
```

For the complete list of available Java artifacts, please consult the information available at the [link](https://github.com/open-telemetry/opentelemetry-java#releases)

## Usage

All examples below can be run using amazing [Scala CLI](https://scala-cli.virtuslab.org/). You can find their full copies in the `scala-cli/opentelemetry/` directory. To run, type `scala-cli <AppName>.scala` while in the directory where the file is located.

### Setup

The `zio.telemetry.opentelemetry.OpenTelemetry` (aka entry point) offers a comprehensive set of layers for instrumenting your ZIO application. 

First of all, you need to provide an instance of `io.opentelemetry.api.Opentelemetry`.
In case you don't need an automatic instrumentation, you can use `OpenTelemetry.custom` layer. It receives a scoped ZIO effect indicating that the provided instance will be closed when the application is shut down. Here is an example:

```scala

def custom(resourceName: String): TaskLayer[api.OpenTelemetry] =
  OpenTelemetry.custom(
    for {
      tracerProvider <- TracerProvider.stdout(resourceName)
      meterProvider  <- MeterProvider.stdout(resourceName)
      loggerProvider <- LoggerProvider.stdout(resourceName)
      openTelemetry  <- ZIO.fromAutoCloseable(
                          ZIO.succeed(
                            OpenTelemetrySdk
                              .builder()
                              .setTracerProvider(tracerProvider)
                              .setMeterProvider(meterProvider)
                              .setLoggerProvider(loggerProvider)
                              .build
                          )
                        )
    } yield openTelemetry
  )
```

The library depends only on `opentelemetry-api` which means you have to manage an initialization of providers and depenendencies for `opentelemetry-sdk`, and `opentelemetry-exporter-*` inside your application.

For more details, please have a look at the source code of the [example application](https://github.com/zio/zio-telemetry/tree/series/2.x/opentelemetry-example/src/main/scala/zio/telemetry/opentelemetry/example).

#### Usage with OpenTelemetry automatic instrumentation

OpenTelemetry provides a [JVM agent for automatic instrumentation](https://opentelemetry.io/docs/instrumentation/java/automatic/) which supports many [popular Java libraries](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md).
Since [version 1.25.0](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v1.25.0) OpenTelemetry JVM agent supports ZIO.

To enable interoperability between automatic instrumentation and `zio-opentelemetry`, `Tracing` has to be created
using `ContextStorage` backed by OpenTelemetry's native `Context` and `Tracer` provided by globally registered `TracerProvider`. It means that instead of `OpenTelemetry.contextZIO` and `OpenTelemetry.custom` you have to provide `OpenTelemetry.contextJVM` and `OpenTelemetry.global` layers.

### Tracing

To send [Trace signals](https://opentelemetry.io/docs/concepts/signals/traces/), you will need a `Tracing` service in your environment. For this, use the `OpenTelemetry.tracing` layer which in turn requires an instance of `OpenTelemetry` provided by Java SDK and a suitable `ContextStorage` implementation. The `Tracing` API includes methods for creating spans, as well as for adding attributes and events to them. 
Here are some of the main ones:

- `root` - sets the current span to be the new root span
- `span` - sets the current span to be the child of the current span
- `spanScoped` - sets the current span to be the child of the current span, but ends it only when the scope closes
- `extractSpan` - extracts the span from carrier and set its child span to be the current span
- `injectSpan` - injects the current span into carrier
- `setAttribute` - sets an attribute of the current span
- `addEvent` - adds an event to the current span

Some of the methods above are available via [ZIO Aspect](https://zio.dev/reference/core/zio/#zio-aspect) syntax. 

```scala
//> using scala "3.4.2"
//> using dep dev.zio::zio:2.1.7
//> using dep dev.zio::zio-opentelemetry:3.0.0-RC24
//> using dep io.opentelemetry:opentelemetry-sdk:1.40.0
//> using dep io.opentelemetry:opentelemetry-sdk-trace:1.40.0
//> using dep io.opentelemetry:opentelemetry-exporter-logging-otlp:1.40.0
//> using dep io.opentelemetry.semconv:opentelemetry-semconv:1.22.0-alpha

object TracingApp extends ZIOAppDefault {

  val instrumentationScopeName = "dev.zio.TracingApp"
  val resourceName             = "tracing-app"

  // Prints to stdout in OTLP Json format
  val stdoutTracerProvider: RIO[Scope, SdkTracerProvider] =
    for {
      spanExporter   <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingSpanExporter.create()))
      spanProcessor  <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleSpanProcessor.create(spanExporter)))
      tracerProvider <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkTracerProvider
              .builder()
              .setResource(Resource.create(Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addSpanProcessor(spanProcessor)
              .build()
          )
        )
    } yield tracerProvider

  val otelSdkLayer: TaskLayer[api.OpenTelemetry] =
    OpenTelemetry.custom(
      for {
        tracerProvider <- stdoutTracerProvider
        sdk            <- ZIO.fromAutoCloseable(
                            ZIO.succeed(
                              OpenTelemetrySdk
                                .builder()
                                .setTracerProvider(tracerProvider)
                                .build()
                            )
                          )
      } yield sdk
    )

  override def run =
    ZIO
      .serviceWithZIO[Tracing] { tracing =>
        val logic = for {
          // Set an attribute to the current span
          _       <- tracing.setAttribute("attr1", "value1")
          // Add an event to the current span
          _       <- tracing.addEvent("Waiting for the user input")
          // Read user input
          message <- Console.readLine
          // Add another event to the current span
          _       <- tracing.addEvent(s"User typed: $message")
        } yield message

        // Create a root span with a lifetime equal to the runtime of the given ZIO effect.
        // We use ZIO Aspect's @@ syntax here just for the sake of example.
        logic @@ tracing.aspects.root("root_span", SpanKind.INTERNAL)
      }
      .provide(
        otelSdkLayer,
        OpenTelemetry.tracing(instrumentationScopeName),
        OpenTelemetry.contextZIO
      )

}
```

### Metrics

To send [Metric signals](https://opentelemetry.io/docs/concepts/signals/metrics/), you will need a `Meter` service in your environment. For this, use the `OpenTelemetry.meter` layer which in turn requires an instance of `OpenTelemetry` provided by Java SDK and a suitable `ContextStorage` implementation. The `Meter` API lets you create [Counter](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#counter), [UpDownCounter](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#updowncounter), [Gauge](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#gauge), [Histogram](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#histogram) and their [asynchronous](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#asynchronous-instrument-api) (aka observable) counterparts. 
As a rule of thumb, observable instruments must be initialized on an application startup. They are scoped, so you should not be worried about shutting them down manually.
By default the metric instruments does not take ZIO log annotations into account. To turn it on pass `logAnnotated = true` parameter to the `OpenTelemetry.metrics` layer initializer.

```scala
//> using scala "3.4.2"
//> using dep dev.zio::zio:2.1.7
//> using dep dev.zio::zio-opentelemetry:3.0.0-RC24
//> using dep io.opentelemetry:opentelemetry-sdk:1.40.0
//> using dep io.opentelemetry:opentelemetry-sdk-trace:1.40.0
//> using dep io.opentelemetry:opentelemetry-exporter-logging-otlp:1.40.0
//> using dep io.opentelemetry.semconv:opentelemetry-semconv:1.22.0-alpha

object MetricsApp extends ZIOAppDefault {

  val instrumentationScopeName = "dev.zio.MetricsApp"
  val resourceName             = "metrics-app"

  // Prints to stdout in OTLP Json format
  val stdoutMeterProvider: RIO[Scope, SdkMeterProvider] =
    for {
      metricExporter <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingMetricExporter.create()))
      metricReader   <-
        ZIO.fromAutoCloseable(ZIO.succeed(PeriodicMetricReader.builder(metricExporter).setInterval(5.second).build()))
      meterProvider  <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkMeterProvider
              .builder()
              .registerMetricReader(metricReader)
              .setResource(Resource.create(common.Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .build()
          )
        )
    } yield meterProvider

  // Prints to stdout in OTLP Json format
  val stdoutTracerProvider: RIO[Scope, SdkTracerProvider] =
    for {
      spanExporter   <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingSpanExporter.create()))
      spanProcessor  <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleSpanProcessor.create(spanExporter)))
      tracerProvider <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkTracerProvider
              .builder()
              .setResource(Resource.create(common.Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addSpanProcessor(spanProcessor)
              .build()
          )
        )
    } yield tracerProvider

  val otelSdkLayer: TaskLayer[api.OpenTelemetry] =
    OpenTelemetry.custom(
      for {
        tracerProvider <- stdoutTracerProvider
        meterProvider  <- stdoutMeterProvider
        sdk            <- ZIO.fromAutoCloseable(
                            ZIO.succeed(
                              OpenTelemetrySdk
                                .builder()
                                .setTracerProvider(tracerProvider)
                                .setMeterProvider(meterProvider)
                                .build()
                            )
                          )
      } yield sdk
    )

  // Stores the number of seconds elapsed since the application startup
  val tickRefLayer: ULayer[Ref[Long]] =
    ZLayer(
      for {
        ref <- Ref.make(0L)
        _   <- ref
                 .update(_ + 1)
                 .repeat[Any, Long](Schedule.spaced(1.second))
                 .forkDaemon
      } yield ref
    )

  // Records the number of seconds elapsed since the application startup
  val tickCounterLayer: RLayer[Meter & Ref[Long], Unit] =
    ZLayer.scoped(
      for {
        meter <- ZIO.service[Meter]
        ref   <- ZIO.service[Ref[Long]]
        // Initialize observable counter instrument
        _     <- meter.observableCounter("tick_counter") { om =>
                   for {
                     tick <- ref.get
                     _    <- om.record(tick)
                   } yield ()
                 }
      } yield ()
    )

  override def run =
    ZIO
      .serviceWithZIO[Tracing] { tracing =>
        val logic = for {
          meter                <- ZIO.service[Meter]
          // Create a counter
          messageLengthCounter <- meter.counter("message_length_counter")
          // Read user input
          message              <- Console.readLine
          // Sleep for the number of seconds equal to the message length  to demonstrate the work of observable counter
          _                    <- ZIO.sleep(message.length.seconds)
          // Record the message length
          _                    <- messageLengthCounter.add(message.length, Attributes(Attribute.string("message", message)))
        } yield message

        // By wrapping our logic into a span, we make the `messageLengthCounter` data points correlated with a "root_span" automatically.
        // Additionally we implicitly add one more attribute to the `messageLenghtCounter` as it is wrapped into a `ZIO.logAnnotate` call.
        ZIO.logAnnotate("zio", "annotation")(logic) @@ tracing.aspects.root("root_span")
      }
      .provide(
        otelSdkLayer,
        OpenTelemetry.metrics(instrumentationScopeName, logAnnotated = true),
        OpenTelemetry.tracing(instrumentationScopeName),
        OpenTelemetry.contextZIO,
        tickCounterLayer,
        tickRefLayer
      )

}
```

#### Integration with ZIO metrics

To enable seamless integration with [ZIO metrics](https://zio.dev/reference/observability/metrics/), use the `OpenTelemetry.zioMetrics` layer. If you also need to publish JVM metrics, be sure to include `DefaultJvmMetrics.live.unit`.

### Logging

To send [Log signals](https://opentelemetry.io/docs/concepts/signals/logs/), you will need a `Logging` service in your environment. For this, use the `OpenTelemetry.logging` layer which in turn requires an instance of `OpenTelemetry` provided by Java SDK and a suitable `ContextStorage` implementation. You can achieve the same by incorporating [Logger MDC auto-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/logger-mdc-instrumentation.md), so the rule of thumb is to use the `Logging` service when you need to propagate ZIO log annotations as log record attributes or, for some reason you don't want to use auto-instrumentation.

```scala
//> using scala "3.4.2"
//> using dep dev.zio::zio:2.1.7
//> using dep dev.zio::zio-opentelemetry:3.0.0-RC24
//> using dep io.opentelemetry:opentelemetry-sdk:1.40.0
//> using dep io.opentelemetry:opentelemetry-sdk-trace:1.40.0
//> using dep io.opentelemetry:opentelemetry-exporter-logging-otlp:1.40.0
//> using dep io.opentelemetry.semconv:opentelemetry-semconv:1.22.0-alpha

object LoggingApp extends ZIOAppDefault {

  val instrumentationScopeName = "dev.zio.LoggingApp"
  val resourceName             = "logging-app"

  // Prints to stdout in OTLP Json format
  val stdoutLoggerProvider: RIO[Scope, SdkLoggerProvider] =
    for {
      logRecordExporter  <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingLogRecordExporter.create()))
      logRecordProcessor <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleLogRecordProcessor.create(logRecordExporter)))
      loggerProvider     <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkLoggerProvider
              .builder()
              .setResource(Resource.create(Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addLogRecordProcessor(logRecordProcessor)
              .build()
          )
        )
    } yield loggerProvider

  // Prints to stdout in OTLP Json format
  val stdoutTracerProvider: RIO[Scope, SdkTracerProvider] =
    for {
      spanExporter   <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingSpanExporter.create()))
      spanProcessor  <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleSpanProcessor.create(spanExporter)))
      tracerProvider <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkTracerProvider
              .builder()
              .setResource(Resource.create(Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addSpanProcessor(spanProcessor)
              .build()
          )
        )
    } yield tracerProvider

  val otelSdkLayer: TaskLayer[api.OpenTelemetry] =
    OpenTelemetry.custom(
      for {
        tracerProvider <- stdoutTracerProvider
        loggerProvider <- stdoutLoggerProvider
        sdk            <- ZIO.fromAutoCloseable(
                            ZIO.succeed(
                              OpenTelemetrySdk
                                .builder()
                                .setTracerProvider(tracerProvider)
                                .setLoggerProvider(loggerProvider)
                                .build()
                            )
                          )
      } yield sdk
    )

  override def run =
    ZIO
      .serviceWithZIO[Tracing] { tracing =>
        val logic = for {
          // Read user input
          message <- Console.readLine
          // Propagate a ZIO.logInfo message as an OTEL log signal and log annotations as log record attributes
          _       <- ZIO.logAnnotate("correlated", "true")(
                       ZIO.logInfo(s"User message: $message")
                     )
        } yield ()

        // All log messages produced by `logic` will be correlated with a "root_span" automatically
        logic @@ tracing.aspects.root("root_span")
      }
      .provide(
        otelSdkLayer,
        OpenTelemetry.logging(instrumentationScopeName),
        OpenTelemetry.tracing(instrumentationScopeName),
        OpenTelemetry.contextZIO
      )

}
```

### Baggage

To pass contextual information in [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/), you will need a `Baggage` service in your environment. For this, use the `OpenTelemetry.baggage` layer which in turn requires an instance of a suitable `ContextStorage` implementation. The `Baggage` API includes methods for getting/setting key/value pairs and injecting/extracting baggage data using the current context. By default the `Baggage` service does not take ZIO log annotations into account. To turn it on use `OpenTelemetry.baggage(logAnnotated = true)`.

```scala
//> using scala "3.4.2"
//> using dep dev.zio::zio:2.1.7
//> using dep dev.zio::zio-opentelemetry:3.0.0-RC24

object BaggageApp extends ZIOAppDefault {

  override def run =
    ZIO
      .serviceWithZIO[Baggage] { baggage =>
        for {
          // Read user input
          message <- Console.readLine
          // Set baggage key/value
          _       <- baggage.set("message", message)
          // Read all baggage data including ZIO log annotations
          data    <- ZIO.logAnnotate("message2", "annotation")(
                       baggage.getAll
                     )
          // Print the resulting data
          _       <- Console.printLine(s"Baggage data: $data")
        } yield message
      }
      .provide(
        OpenTelemetry.baggage(logAnnotated = true),
        OpenTelemetry.contextZIO
      )

}
```

### Context Propagation

Explicitly utilizing the context propagation API becomes relevant only when auto-instrumentation is not used.
Please note that injection and extraction are not referentially transparent due to the use of the mutable OpenTelemetry carrier Java API.

```scala
//> using scala "3.4.2"
//> using dep dev.zio::zio:2.1.7
//> using dep dev.zio::zio-opentelemetry:3.0.0-RC24
//> using dep io.opentelemetry:opentelemetry-sdk:1.40.0
//> using dep io.opentelemetry:opentelemetry-sdk-trace:1.40.0
//> using dep io.opentelemetry:opentelemetry-exporter-logging-otlp:1.40.0
//> using dep io.opentelemetry.semconv:opentelemetry-semconv:1.22.0-alpha

object PropagatingApp extends ZIOAppDefault {

  val instrumentationScopeName = "dev.zio.PropagatingApp"
  val resourceName             = "propagating-app"

  // Prints to stdout in OTLP Json format
  val stdoutTracerProvider: RIO[Scope, SdkTracerProvider] =
    for {
      spanExporter   <- ZIO.fromAutoCloseable(ZIO.succeed(OtlpJsonLoggingSpanExporter.create()))
      spanProcessor  <- ZIO.fromAutoCloseable(ZIO.succeed(SimpleSpanProcessor.create(spanExporter)))
      tracerProvider <-
        ZIO.fromAutoCloseable(
          ZIO.succeed(
            SdkTracerProvider
              .builder()
              .setResource(Resource.create(Attributes.of(ServiceAttributes.SERVICE_NAME, resourceName)))
              .addSpanProcessor(spanProcessor)
              .build()
          )
        )
    } yield tracerProvider

  val otelSdkLayer: TaskLayer[api.OpenTelemetry] =
    OpenTelemetry.custom(
      for {
        tracerProvider <- stdoutTracerProvider
        sdk            <- ZIO.fromAutoCloseable(
                            ZIO.succeed(
                              OpenTelemetrySdk
                                .builder()
                                .setTracerProvider(tracerProvider)
                                .build()
                            )
                          )
      } yield sdk
    )

  override def run =
    ZIO
      .serviceWithZIO[Tracing] { tracing =>
        val tracePropagator   = TraceContextPropagator.default
        val baggagePropagator = BaggagePropagator.default
        // Using the same kernel and carriers for baggage and tracing context propagation is safe
        // since their encodings occupy different keys in the OTEL context.
        val kernel            = mutable.Map.empty[String, String]
        val outgoingCarrier   = OutgoingContextCarrier.default(kernel)
        val incomingCarrier   = IncomingContextCarrier.default(kernel)

        ZIO.serviceWithZIO[Baggage] { baggage =>
          // Representing upstream service
          val upstreamService = for {
            // Read user input
            message <- Console.readLine
            // Set and propagate the baggage data using outgoing carrier
            _       <- baggage.set("message", message)
            _       <- baggage.inject(baggagePropagator, outgoingCarrier)
            // Emulate the computation to be wrapped in a root span
            logic    = for {
                         _ <- ZIO.logInfo(s"Message length is ${message.length}")
                         // Inject the current span using outgoing carrier
                         _ <- tracing.injectSpan(tracePropagator, outgoingCarrier)
                       } yield ()
            // Run the logic, wrapping it into a root span
            _       <- logic @@ tracing.aspects.root("upstream_root_span")
          } yield ()

          // Representing downstream service
          val downstreamService = for {
            // Extract the baggage data using incoming carrier
            _      <- baggage.extract(baggagePropagator, incomingCarrier)
            data   <- baggage.getAll
            message = data("message")
            // Emulate the logic that computes message length and sets an attribute of the current span
            logic   = for {
                        _ <- ZIO.logInfo(s"Message length is ${message.length}")
                        _ <- tracing.setAttribute("message", message)
                      } yield ()
            // Run the logic, wrapping it into a child span of the upstream root span
            _      <- logic @@ tracing.aspects.extractSpan(tracePropagator, incomingCarrier, "downstream_root_span")
          } yield ()

          // Simulate the interaction between services
          upstreamService *> downstreamService
        }

      }
      .provide(
        otelSdkLayer,
        OpenTelemetry.tracing(instrumentationScopeName),
        OpenTelemetry.baggage(),
        OpenTelemetry.contextZIO
      )

}
```

---

## OpenTracing Example

You can find the source code [here](https://github.com/zio/zio-telemetry/tree/series/2.x/opentracing-example).

Firstly, start [Jaeger](https://www.jaegertracing.io) by running the following command:
```bash
docker run --rm -it \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.36
``` 

To check if it's running properly visit [Jaeger UI](http://localhost:16686/).
More info can be found [here](https://www.jaegertracing.io/docs/1.6/getting-started/#all-in-one-docker-image).

Our application contains two services:
 1. [Proxy](https://github.com/zio/zio-telemetry/blob/series/2.x/opentracing-example/src/main/scala/zio/telemetry/opentracing/example/ProxyApp.scala) service
 2. [Backend](https://github.com/zio/zio-telemetry/blob/series/2.x/opentracing-example/src/main/scala/zio/telemetry/opentracing/example/BackendApp.scala) service

### Proxy Service

Represents the entry point of the distributed system example. It exposes the `/statuses` endpoint which returns a list of system's services statuses.

The service consists of `ProxyHttpServer` and `ProxyHttpApp`.

#### ProxyApp

In order to start the service run:
```bash
sbt "opentracingExample/runMain zio.telemetry.opentracing.example.ProxyApp"
```

The console should output
```bash
running zio.telemetry.opentracing.example.ProxyApp
```
if the server has been started properly.

#### ProxyHttpApp

Provides the implementation of the service, which returns the status of the backend service and the proxy service itself. 
`Client` is used to retrieve the status of the backend service.

This is also where the tracing of the application is done, by collecting the timings and logging things such as the span
type and the HTTP method. The context is injected into a carrier, and passed along to the backend through `Client`, 
where a child span is created, and logging of the backend service is done.

### Backend Service

Represents the "internal" service of the system. It exposes the `/status` endpoint which returns the status of the backend service.

The service consists of `BackendServer` and `BackendApp`.

#### BackendApp

In order to start the service run:
```bash
sbt "opentracingExample/runMain zio.telemetry.opentracing.example.BackendApp"
```

The console should output
```bash
running zio.telemetry.opentracing.example.BackendApp
```
if the server has been started properly.

#### BackendHttpApp

Provides the implementation of the service, which is to simply return the status of the backend service.

### Status

```scala
final case class Status(name: String, status: String)
```

Represents the status of a service.

### Statuses

```scala
final case class Statuses(data: List[Status]) extends AnyVal
```

Represents the statuses of a number of services.

### Configuration

Configuration is given in [application.conf](https://github.com/zio/zio-telemetry/blob/82787facf973feeb9c128f21a964fad15d7c591d/opentracing-example/src/main/resources/application.conf).

### Running

After both services are properly started, running the following command
```bash
curl -X GET http://localhost:8080/statuses
```
should return the following response:
```json
{"data":[{"name":"backend","status":"up"},{"name":"proxy","status":"up"}]}
```

Simultaneously, it will create trace that will be stored in Jaeger backend.

---

## OpenTracing

OpenTracing is a standard and API for distributed tracing, i.e. collecting timings,
and logs across process boundaries. Well known implementations are [Jaeger](https://www.jaegertracing.io) and [Zipkin](https://www.zipkin.io).

## Installation

First, add the following dependency to your build.sbt:

```
"dev.zio" %% "zio-opentracing" % "<version>"
```

## Usage

To use ZIO Telemetry, you will need an `OpenTracing` service in your
environment. You also need to provide a `tracer` (for this example we use `JaegerTracer.live` from `opentracing-example` module) implementation:

```scala

val app =
  ZIO.serviceWithZIO[OpenTracing] { tracing =>

    (for {
      _       <- tracing.tag(Tags.SPAN_KIND.getKey, Tags.SPAN_KIND_CLIENT)
      _       <- tracing.tag(Tags.HTTP_METHOD.getKey, "GET")
      _       <- tracing.setBaggageItem("proxy-baggage-item-key", "proxy-baggage-item-value")
      message <- Console.readline
      _       <- tracing.log("Message has been read")
    } yield message) @@ root("/app")
  }.provide(OpenTracing.live, JaegerTracer.live("my-app"))
```

After importing `import tracing.aspects._`, additional `ZIOAspect` combinators
on `ZIO`s are available to support starting child spans, tagging, logging and
managing baggage.

```scala
ZIO.serviceWithZIO[OpenTracing] { tracing =>

  // start a new root span and set some baggage item
  val zio1 = tracing.setBaggage("foo", "bar") @@ root("root span")

  // start a child of the current span, set a tag and log a message
  val zio2 = 
    (for {
      _ <- tracing.tag("http.status_code", 200)
      _ <- tracing.log("doing some serious work here!")
    } yield ()) @@ span("child span")
}
```

To propagate contexts across process boundaries, extraction and injection can be
used. The current span context is injected into a carrier, which is passed
through some side channel to the next process. There it is injected back and a
child span of it is started. For the example we use the standardized `TextMap`
carrier. For details about extraction and injection, please refer to 
[OpenTracing Documentation](https://opentracing.io/docs/overview/inject-extract/). 

Due to the use of the (mutable) OpenTracing carrier APIs, injection and extraction
are not referentially transparent.

```scala
ZIO.serviceWithZIO[OpenTracing] { tracing =>

  val buffer = new TextMapAdapter(mutable.Map.empty.asJava)
  for {
    _ <- tracing.inject(Format.Builtin.TEXT_MAP, buffer)
    _ <- ZIO.unit @@ spanFrom(Format.Builtin.TEXT_MAP, buffer, "child of remote span")
  } yield buffer
}
```

---

## ZIO 2.x Interoperation with Cats 2.x

## Installation

```sbt
libraryDependencies += "dev.zio" %% "zio-interop-cats" % "22.0.0.0"
```

## `ZIO` Cats Effect instances

**ZIO** integrates with Typelevel libraries by providing an instance of `ConcurrentEffect` for `IO` as required, for instance, by `fs2`, `doobie` and `http4s`. Actually, I lied a little bit, it is not possible to implement `ConcurrentEffect` for any error type since `ConcurrentEffect` extends `MonadError` of `Throwable`.

For convenience we have defined an alias as follow:

```scala
  type Task[A] = IO[Throwable, A]
```

Therefore, we provide an instance of `ConcurrentEffect[Task]`.

## ConcurrentEffect

In order to get a `ConcurrentEffect[Task]` or `ConcurrentEffect[RIO[R, *]]` we need an implicit `Runtime[R]` in scope. The easiest way to get it is using `ZIO.runtime`:

```scala

def getCE = {
  ZIO.runtime.map { implicit r: Runtime[Any] =>
    val F: ConcurrentEffect[Task] = implicitly
  }
}
```

`Task.concurrentEffectWith` method can automate this pattern:

```scala

def fork = {
  Task.concurrentEffectWith { implicit CE =>
    CE.start(Task(println("Started task")))
  }
}
```

### Timer

In order to get a `cats.effect.Timer[Task]` instance we need an extra import:

```scala

```

The reason it is not provided by the default "interop" import is that it makes testing programs that require timing capabilities hard therefore an extra import wherever needed makes reasoning about it much easier.
It is not a good idea to use this import as it causes limitations in testing since it uses the `live` implementation of `zio.Clock.Service`. If you want to retain testability, do the following instead:

```scala

ZIO.runtime[Clock].flatMap { implicit rts =>
  val clock: Timer[Task] = rts.environment.get.toTimer
  val ce: ConcurrentEffect[Task] = implicitly
  
  ce.race(clock.sleep(1.second), clock.sleep(1.second))
}
```

### cats-core

If you only need instances for `cats-core` typeclasses, not `cats-effect` import `zio.interop.catz.core._`:

````scala

````

Note that this library only has an `Optional` dependency on cats-effect – if you or your libraries don't depend on it, this library will not add it to the classpath.

### Example

The following example shows how to use ZIO with Doobie (a library for JDBC access) and FS2 (a streaming library), which both rely on Cats Effect instances:

```scala

val xa: Transactor[Task] = Transactor.fromDriverManager[Task](...)

def loadUsers: Stream[Task, User] =
  sql"""SELECT * FROM users""".query[User].stream.transact(xa)

val allUsers: List[User] = unsafeRun(loadUsers.compile.toList)
```