Exploring type level programming in Scala

Exploring
Type-Level Programming
in Scala

JORGE VÁSQUEZ
SCALA DEVELOPER

Agenda
● Motivations around Type-Level Programming in Scala
● Background
○ Dependent types
○ Path-dependent types
○ Abstract type members
● Examples of libraries that use Type-Level
Programming

Motivations
around
Type-Level
Programming

Why Type-Level Programming?
● We turn to Scala for type-safety
● But… Sometimes it’s not enough!

Concrete example: Spark DataFrame API
We have lots of potential for runtime bugs!

df.select(
$"username",
$"tweet"
)

Error: org.apache.spark.sql.
AnalysisException: cannot
resolve '`username`'

df.select(
$"timestamp" * 10
)

Error:
org.apache.spark.sql.
AnalysisException: cannot
resolve '(`timestamp` *
10)' due to data type
mismatch

df.filter(
$"text" === true
)
// Does not fail

df.select(
$"text" / 1000
)
// Does not fail

Can we do better?
Scala has a VERY powerful type system,
why not use it?

Concrete example: Spark with Frameless
Frameless helps us to eliminate a lot of bugs…
At compile time!

tds.select(
tds('username),
tds('tweet)
)

// Error:
No column Symbol with
shapeless.tag.Tagged[Str
ing("username")]

tds.select(
tds('timestamp) * 10
)

// Error:
overloaded method value
* with alternatives
[...] cannot be applied
to (Int)

tds.filter(
tds('text) === true
)

// Error:
overloaded method value
=== with alternatives
[...] cannot be applied
to (Boolean)

tds.select(
tds('text) / 1000
)

// Error:
overloaded method value /
with alternatives [...]
cannot be applied to
(Int)

In conclusion...
Type-level Programming lets you eliminate
bugs at compile-time

Our focus today: Dependent types
Dependent types are the heart of
Type-Level Programming in Scala

What are Dependent Types?
● Dependent Types are types that depend on
values.
● With this, we remove the usual separation
between the type and value worlds.

What are Dependent Types?
● Scala is not a fully dependently typed
language.
● However, it supports some form of
Dependent Types, which is called Path
Dependent Types.

How we deﬁne Path Dependent Types?
● In Scala, we can deﬁne nested components
● For example, a class inside a trait, a
trait inside a class, etc.

sealed trait Foo {
sealed trait Bar
}
val foo1 = new Foo {}
val foo2 = new Foo {}
val a: Foo#Bar = new foo1.Bar {} // OK
val b: Foo#Bar = new foo2.Bar {} // OK
val c: foo1.Bar = new foo1.Bar {} // OK
val d: foo2.Bar = new foo1.Bar {}
// Required: foo2.Bar, Found: foo1.Bar

● Another useful tool is Abstract Type
Members, which are types we don’t know
yet and that we can deﬁne later
trait Bar {
type T
}

Example 1: Merging Files
Define a merge function, which should take:
● A list of files
● A merge strategy: Single/Multiple/None
● A callback function: Which should expect:
○ A single file if merge strategy is Single
○ A list of files if merge strategy is Multiple
○ A unit value if merge strategy is None

import java.io.File
sealed trait MergeStrategy {
type Output
}
object MergeStrategy {
case object Single extends MergeStrategy { type Output = File }
case object Multiple extends MergeStrategy { type Output = List[File] }
case object None extends MergeStrategy { type Output = Unit }
}
def mergeFiles(files: List[File]): File = ???

def merge[T](files: List[File], mergeStrategy: MergeStrategy)
(f: mergeStrategy.Output => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeFiles(files))
case MergeStrategy.Multiple => f(files)
case MergeStrategy.None => f(())
}

def merge[O, T](
files: List[File],
mergeStrategy: MergeStrategy { type Output = O }
)(f: O => T): T =
case MergeStrategy.Single => f(mergeFiles(files))
case MergeStrategy.Multiple => f(files)
}

val files: List[File] = ???
merge(files, MergeStrategy.Single) { file: File =>
// Do some processing
}
merge(files, MergeStrategy.Multiple) { files: List[File] =>
}
merge(files, MergeStrategy.None) { _: Unit =>
}

Example 2: Merging Elements
Deﬁne a merge function, which should take:
● A list of elements of any type
● A merge strategy: Single/Multiple/None
● A callback function: Which should expect:
○ A single element if merge strategy is Single
○ A list of elements if merge strategy is Multiple
○ A unit value if merge strategy is None

sealed trait MergeStrategy {
type Output[_]
}
object MergeStrategy {
case object Single extends MergeStrategy { type Output[A] = A }
case object Multiple extends MergeStrategy { type Output[A] = List[A] }
case object None extends MergeStrategy { type Output[_] = Unit }
}
def mergeElements[E](elements: List[E]): E = ???

def merge[E, O[_], T](
elements: List[E],
mergeStrategy: MergeStrategy { type Output[A] = O[A] }
)(f: O[E] => T): T =
case MergeStrategy.Single => f(mergeElements(elements))
case MergeStrategy.Multiple => f(elements)
}

val messages: List[String] = ???
merge(messages, MergeStrategy.Single) { message: String =>
}
merge(messages, MergeStrategy.Multiple) { messages: List[String] =>
}
merge(messages, MergeStrategy.None) { _: Unit =>
}

In conclusion...
● Path-dependent types are the heart and
soul of Scala's type system
● They help you to improve compile-time type
safety

In conclusion...
DOTTY = DOT Calculus = Path Dependent Types

Examples of libraries that use
Type Level Programming
● Shapeless: Generic programming
○ Generic Product Type: HList
○ Generic Sum Type: Coproduct

● Frameless: Expressive types for Spark

● Reﬁned: Reﬁnement types

● ZIO SQL: Type-safe SQL queries

References
● Dependent types in Scala, blog post by Yao Li
● Type Level Programming in Scala step by
step, blog series by Luigi Antonini
● The Type Astronaut’s Guide to Shapeless
Book, by Dave Gurnell
● Introduction to Apache Spark with Frameless,
by Brian Clapper

Special thanks
● To micro sphere.it organizers for hosting this
presentation
● To John De Goes for guidance and support

@jorvasquez2301
jorge.vasquez@scalac.io
jorge-vasquez-2301
Contact me

Exploring type level programming in Scala

More Related Content

What's hot

Similar to Exploring type level programming in Scala

More from Jorge Vásquez

Recently uploaded

Exploring type level programming in Scala