Kotlin Mullets

Kotlin
Mullets
Business on the Front End, Party in Back
Chet Haase, Developer Advocate, Android
James Ward, Developer Advocate, Google Cloud

Track
● Track
Motion
Events
Record
● Add
Locations
to Path(s)
View
● Invalidate
ML
● Ready to
Analyze
Draw
● Draw Path
Data
User Drawings

UI Layout
<androidx.constraintlayout.widget.ConstraintLayout ...>
<Spinner ... />
<Button ... />
<Button ... />
<ToggleButton... />
<Button ... />
<TextView... />
<TextView ... />
<TextView ... />
<TextView ... />
<com.jamesward.airdraw.DrawingCanvas ... />
</androidx.constraintlayout.widget.ConstraintLayout>

UI Layout, with custom view
<androidx.constraintlayout.widget.ConstraintLayout ...>
< ... />
<com.jamesward.airdraw.DrawingCanvas
android:id="@+id/drawingCanvas"
android:layout_width="0dp"
android:layout_height="0dp"
android:layout_margin="8dp"
android:background="@android:color/black"
app:layout_constraintDimensionRatio="1:1"
app:layout_constraintEnd_toEndOf="@id/rightGuide"
app:layout_constraintHorizontal_bias="0.0"
app:layout_constraintStart_toStartOf="@id/leftGuide"
app:layout_constraintTop_toBottomOf="@+id/fourthGuess" />
</androidx.constraintlayout.widget.ConstraintLayout>

DrawingCanvas
class DrawingCanvas(context: Context?, attr: AttributeSet) :
View(context, attr) {
val paint: Paint = Paint(Paint.ANTI_ALIAS_FLAG)
private val drawingPath = Path()
fun getBitmap():Bitmap { ... }
fun setBitmap(bitmap: Bitmap) { ... }
override fun onTouchEvent(event: MotionEvent?): Boolean { ... }
fun clear() { ... }
override fun onDraw(canvas: Canvas?) { ... }
}

Track Motion Events
override fun onTouchEvent(event: MotionEvent?): Boolean {
var handled = false
if (event != null) {
when {
event.action == MotionEvent.ACTION_DOWN -> {
drawingPath.moveTo(event.x, event.y)
drawingPath.lineTo(event.x, event.y)
invalidate()
handled = true
}
event.action == MotionEvent.ACTION_MOVE -> {
drawingPath.lineTo(event.x, event.y)
invalidate()
handled = true
}
event.action == MotionEvent.ACTION_UP -> {
handled = true
}
}
}
return handled
}

Draw Path
override fun onDraw(canvas: Canvas?) {
canvas?.drawPath(drawingPath, paint)
}

Interlude: Learning About Machine Learning
Tensor FlowTraining Data Model
TFLite Model
On-Device
Model
User Data
Results

Interlude: ML
Results
Labels
Barcodes
Text
Objects
Faces
Translations
+ Confidence Values

Or…. Use Built-in Models
Tensor FlowTraining Data Model
TFLite Model
On-Device
Model
User Data
Results

Convert Drawing
● Extract
Bitmap
Setup Labeler
● Create
Firebase
Vision
Image
● Create On-
Device
Labeler
Send Data
● Process
Data
Receive
Results
● onSuccess
callback
receives
results
Get Image Labels
Display
Results
● Show
Labels and
Confidence

Get Bitmap Data
fun getBitmap():Bitmap {
val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
val canvas = Canvas(bitmap)
draw(canvas)
return bitmap
}

Get Bitmap Data
fun getBitmap() = createBitmap(width, height).applyCanvas(::draw)
fun getBitmap():Bitmap {
val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
val canvas = Canvas(bitmap)
draw(canvas)
return bitmap
}

Call Image Labeler
val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap)
val labeler = FirebaseVision.getInstance().onDeviceImageLabeler
labeler.processImage(firebaseImage).addOnSuccessListener {
// . . .
}
{
{“label”, confidence},
{“label”, confidence},
…
}

Digital Processing: MNIST
Image: ©Josef Steppan, https://commons.wikimedia.org/wiki/File:MnistExamples.png
Modified National Institute of Standards and Technology
Hand-written digits for image-processing

Create Interpreter
● Load model
● Create
Interpreter
with model
Get Bitmap
● Get user
data as
bitmap
● Resize to
match
model
Run Interpreter
● Process
data
Process
Results
● Sort results
by
confidence
Get Image Labels
Display
Results
● Show
results and
confidence

Get Model as ByteBuffer
val model = loadModelFile()
private fun loadModelFile(): ByteBuffer {
val fileDescriptor = assets.openFd("mnist.tflite")
val inputStream = FileInputStream(fileDescriptor.fileDescriptor)
val fileChannel = inputStream.channel
val startOffset = fileDescriptor.startOffset
val declaredLength = fileDescriptor.declaredLength
return fileChannel.map(FileChannel.MapMode.READ_ONLY,
startOffset, declaredLength)
}

Initialize Interpreter, Sizes
val options = Interpreter.Options()
options.setUseNNAPI(true)
val interpreter = Interpreter(model, options)
val inputShape = interpreter.getInputTensor(0).shape()
inputImageWidth = inputShape[1]
inputImageHeight = inputShape[2]
modelInputSize = 4 * inputImageWidth * inputImageHeight

Get Results
private fun detectDigit(bitmap: Bitmap,
resultsListener: (List<LabelAnnotation>) -> Unit) {
lateinit var digitSequence: List<Pair<String,Float>>
val resizedImage = Bitmap.createScaledBitmap(bitmap,
inputImageWidth, inputImageHeight, true)
val byteBuffer = convertBitmapToByteBuffer(resizedImage)
val result = Array(1) { FloatArray(10) }
interpreter.run(byteBuffer, result)
// format and display results
}

Type-safe Builds
dependencies {
implementation(kotlin("stdlib-jdk8"))
implementation("androidx.appcompat:appcompat:1.1.0")
...
}
build.gradle.kts
fun Project.dependencies(configuration: DependencyHandlerScope.() -> Unit) =
DependencyHandlerScope.of(dependencies).configuration()
fun DependencyHandler.`implementation`(dependencyNotation: Any): Dependency? =
add("implementation", dependencyNotation)

Declarative(ish) & Imperative
if (startParameter.taskRequests.find { it.args.contains(":server:shadowJar") } == null) {
include("common", "android", "web", "server")
} else {
include("common", "web", "server")
}
settings.gradle.kts

ML in the Cloud
val labeler = FirebaseVision.getInstance().onDeviceImageLabeler
labeler.processImage(firebaseImage).addOnSuccessListener { … }
On-Device Labeling

ML in the Cloud
val labeler = FirebaseVision.getInstance().cloudImageLabeler
labeler.processImage(firebaseImage).addOnSuccessListener { … }
Cloud Labeling

Get Results
private suspend fun detectDigit(bitmap: Bitmap,
resultsListener: (List<LabelAnnotation>) -> Unit) {
lateinit var digitSequence: List<Pair<String,Float>>
val resizedImage = Bitmap.createScaledBitmap(bitmap,
inputImageWidth, inputImageHeight, true)
val byteBuffer = convertBitmapToByteBuffer(resizedImage)
val result = Array(1) { FloatArray(10) }
withContext(Dispatchers.IO) {
interpreter.run(byteBuffer, result)
}
// format and display results
}

Draw
● Capture
Bitmap
Phone ML
● Predict with
Local Model
● Display
Results
REST Service
● Send
Bitmap &
Results to
REST API
Browser
● Poll d' Bus
● Display
Bitmap &
Results
Pub/Sub
● On d' Bus
Browser
Bus
Server

Shared Common
Root Gradle Project
Common
ServerAndroid
data class LabelAnnotation(
val description: String,
val score: Float)
data class ImageResult(
val image: ByteArray,
val labelAnnotations: List<LabelAnnotation>)
common/src/commonMain/kotlin/Data.kt

Multiplatform Common
plugins {
id("com.android.library")
kotlin("multiplatform")
}
java {
sourceCompatibility = JavaVersion.VERSION_1_8
}
android {
defaultConfig {
compileSdkVersion(29)
}
}
common/build.gradle.ktx

Multiplatform Common
kotlin {
sourceSets {
commonMain {
dependencies {
implementation(kotlin("stdlib"))
}
}
}
jvm {
val main by compilations.getting {
kotlinOptions {
jvmTarget = JavaVersion.VERSION_1_8
}
}
}
android()
}
common/build.gradle.ktx

Multiplatform Common Dependency
dependencies {
implementation(project(":common"))
}
android/build.gradle.ktx
server/build.gradle.kts

Cloud Service - Deps
plugins {
kotlin("kapt")
}
dependencies {
implementation("io.micronaut:micronaut-runtime:1.2.6")
implementation("io.micronaut:micronaut-http-server-netty:1.2.6")
implementation("io.micronaut:micronaut-views:1.2.0")
api("ch.qos.logback:logback-classic:1.2.3")
api("com.fasterxml.jackson.module:jackson-module-kotlin:2.10.1")
api("org.thymeleaf:thymeleaf:3.0.11.RELEASE")
kapt("io.micronaut:micronaut-inject-java:1.2.6")
}
server/build.gradle.ktx

Cloud Service
@Post("/show")
fun show(@Body imageResult: ImageResult): HttpResponse<String> {
bus.put(imageResult)
return HttpResponse.ok("")
}
@Get("/events")
fun events(): HttpResponse<ImageResult> {
val maybe = bus.take()
return if (maybe != null)
HttpResponse.ok(maybe)
else
HttpResponse.noContent()
}
server/src/main/kotlin/WebApp.kt

Client
@Client("${drawurl}")
interface DrawService {
@Post("/show")
fun show(@Body imageResult: ImageResult): Single<Unit>
}
@Inject
var drawService: DrawService? = null
drawService?.show(imageResult)?.subscribe()?.dispose()
android/src/main/kotlin/MainActivity.kt

Setup DI
class BaseApplication: Application() {
private var ctx: ApplicationContext? = null
override fun onCreate() {
super.onCreate()
val pm = applicationContext.packageManager
val ai = pm.getApplicationInfo(packageName, PackageManager.GET_META_DATA)
val propertySource = AndroidMetadataPropertySource(ai.metaData)
val appCtxBuilder = applicationContext.build(MainActivity::class.java, Environment.ANDROID)
ctx = appCtxBuilder.propertySources(propertySource).start()
registerActivityLifecycleCallbacks(object: ActivityLifecycleCallbacks {
override fun onActivityCreated(activity: Activity?, bundle: Bundle?) {
if (activity != null) ctx?.inject(activity)
}
})
}
android/src/main/kotlin/BaseApplication.kt

Config via AndroidManfest.xml
<meta-data android:name="drawurl" android:value="${drawurl}"/>
val drawUrl: String? by project
if (drawUrl != null) {
manifestPlaceholders = mapOf("drawurl" to drawUrl)
}
else {
// 10.0.2.2 is the IP for your machine from the Android emulator
manifestPlaceholders = mapOf("drawurl" to "http://10.0.2.2:8080")
}
android/build.gradle.ktx
android/src/main/AndroidManifest.xml
drawUrl=http://192.168.1.23:8080
gradle.properties

Cloud Run, Run
FROM adoptopenjdk/openjdk8 as builder
WORKDIR /app
COPY . /app
RUN ./gradlew --no-daemon --console=plain :server:shadowJar
FROM adoptopenjdk/openjdk8:jre
COPY --from=builder /app/server/build/libs/server.jar /server.jar
RUN apt-get update && apt-get install -y --no-install-recommends fontconfig
CMD ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/server.jar"]
Dockerfile

Build & Deploy
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA', '/workspace']
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['beta', 'run', 'deploy', '--image=gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA',
'--platform=managed', '--project=$PROJECT_ID', '--region=us-central1', '--allow-
unauthenticated', '--memory=512Mi', '$REPO_NAME']
cloudbuild.yaml

Jetpack Compose
Reactive, declarative UI Toolkit
Kotlin-first
Developer Preview
(Pre-Alpha)

Track
● Track
Motion
Events
Record
● Add
Locations
to Path(s)
View
● Invalidate
ML
● Ready to
Analyze
Draw
● Draw Path
Data
User Drawings
In Compose!

UI Layout
@Composable
fun BuildUI(guesses: Guesses = Guesses()) {
MaterialTheme() {
val invalidator = +invalidate
Column(Spacing(8.dp), crossAxisAlignment = CrossAxisAlignment.Stretch) {
FlexRow(mainAxisAlignment = MainAxisAlignment.Center) {
val radioOptions = listOf("Shape", "Digit")
val (selectedOption, onOptionSelected) = +state { radioOptions[0] }
inflexible {
RadioGroup(
options = radioOptions,
selectedOption = selectedOption,
onSelectedChange = onOptionSelected
)
}
flexible(1f) {
Column(crossAxisAlignment = CrossAxisAlignment.Stretch) {
Button(text = "Local", ...)
Button(text = "Cloud", ...)
}
}
}
Button(text = "Sensorify", ...)
HeightSpacer(8.dp)
GuessDisplay(guesses)
HeightSpacer(8.dp)
DrawingCanvas(path)
HeightSpacer(8.dp)
Button(text = "Clear", ...)
HeightSpacer(8.dp)
}
}
}

DrawingCanvas
@Composable
fun DrawingCanvas(path: Path) {
val invalidate = +invalidate
RawDragGestureDetector(dragObserver = MyDragObserver(path, invalidate)) {
Surface(color = Color.Black) {
Container(modifier = ExpandedHeight, width = 200.dp, height = 350.dp) {
Draw { canvas, parentSize ->
canvas.drawPath(path, fingerPaint)
}
}
}
}
}

Track Motion Events
class MyDragObserver(val dragPath: Path, val recompose: () -> Unit): DragObserver {
override fun onStart(downPosition: PxPosition) {
dragPath.moveTo(downPosition.x.value, downPosition.y.value)
}
override fun onDrag(dragDistance: PxPosition): PxPosition {
dragPath.relativeLineTo(dragDistance.x.value, dragDistance.y.value)
recompose()
return dragDistance
}
}

Machine Learning Code
Button(text = "Local", onClick = {
machineLearningStuff.detectObject(true, bitmap!!,
selectedOption == "Shape") {
// display results in Text objects...
}
})

Shared Common
Root Gradle Project
Common
ServerAndroid Web
Jar

Kotlin JS - Build
plugins {
kotlin("js")
}
dependencies {
implementation(kotlin("stdlib-js"))
implementation(project(":common"))
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core-js:1.3.2")
implementation("org.jetbrains.kotlinx:kotlinx-html-js:0.6.12")
}
web/build.gradle.kts

Kotlin JS - Assemble JsJar
tasks.withType<org.jetbrains.kotlin.gradle.tasks.Kotlin2JsCompile> {
kotlinOptions {
metaInfo = false
sourceMap = true
outputFile = "$buildDir/classes/kotlin/main/${project.name}.js"
}
}
task<Copy>("assembleJsLib") {
from(...)
into("$buildDir/classes/kotlin/main")
dependsOn("mainClasses")
}
tasks {
JsJar {
into("META-INF/resources")
dependsOn("assembleJsLib")
}
}

Kotlin JS - Polling
fun main() {
GlobalScope.launch { poll() }
}
suspend fun poll() {
val res = window.fetch("/events").await()
when (res.status.toInt()) {
200 -> {
val json = res.json().await().asDynamic()
...
val imageResult = ImageResult(byteArray, labelAnnotations)
...
}
}
window.setTimeout({ GlobalScope.launch { poll() } }, 1000)
}
web/src/main/kotlin/Main.kt

Kotlin JS - Rendering
val urlImage = "url('data:image/png;base64,${imageResult.image.decodeToString()}')"
document.body?.style?.backgroundImage = urlImage
document.body?.clear()
val div = document.create.div()
imageResult.labelAnnotations.forEach { labelAnnotation ->
div.append {
p {
+"${labelAnnotation.description} = ${round(labelAnnotation.score *
100)}%"
}
}
}
document.body?.append(div)
web/src/main/kotlin/Main.kt

Kotlin JS - Serving Static Resources
micronaut.router.static-resources.resources.enabled=true
micronaut.router.static-resources.resources.paths=classpath:META-INF/resources
micronaut.router.static-resources.resources.mapping=/resources/**
server/src/main/resources/application.properties
dependencies {
api(files("../web/build/libs/web.jar"))
}
tasks.withType<org.jetbrains.kotlin.gradle.internal.KaptWithKotlincTask> {
dependsOn(":web:JsJar")
}

Kotlin JS - HTML Page
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Air Draw</title>
<link rel="stylesheet" href="assets/index.css">
<script src="resources/kotlin.js"></script>
<script src="resources/kotlinx-coroutines-core.js"></script>
<script src="resources/kotlinx-html-js.js"></script>
<script src="resources/kotlinx-serialization-kotlinx-serialization-
runtime.js"></script>
<script src="resources/common.js"></script>
<script src="resources/web.js"></script>
</head>
<body>
waiting for drawings...
</body>
</html>
server/src/main/kotlin/resources/views/index.html

One Project To Kotlin Them All
Root Gradle Project
Common
ServerAndroid Web
build.gradle.kts
common/build.gradle.kts
android/build.gradle.kts

One Project To Rule Them All
buildscript {
repositories {
mavenLocal()
mavenCentral()
jcenter()
google()
}
dependencies {
classpath(kotlin("gradle-plugin", "1.3.61"))
classpath("com.android.tools.build:gradle:4.0.0-alpha04")
}
}
build.gradle.kts (Root Gradle Project)

Move Data Processing to the Server

Air Draw
● Capture
Sensor Data
REST Service
● Smoooooth
Data
● Render to
Image
Cloud ML
● Send
Bitmap to
Cloud ML
● Send
Bitmap &
results to
Phone
Browser
● Poll d' Bus
● Display
Bitmap &
Results
Pub/Sub
● On d' Bus

Mapping Sensor Data to an Image

Sensors & Vision
data class Orientation(val azimuth: Float, val pitch: Float, val timestamp: Long)
SensorManager.getRotationMatrixFromVector(rotationMatrix, e.values)
val m = FloatArray(3)
SensorManager.getOrientation(rotationMatrix, orientationAngles)
val orientation = Orientation(m[0], m[1], e.timestamp)
readings.add(orientation)
common/src/commonMain/kotlin/Data.kt

SEND d' DATA!
@Client("${drawurl}")
interface DrawService {
@Post("/draw")
fun draw(@Body readings: List<Orientation>): Single<ImageResult>
}
machineLearningStuff.sensorAction(on) { orientations ->
val imageResultMaybe = drawService?.draw(orientations)?.blockingGet()
imageResultMaybe?.let { imageResult ->
displayResults(imageResult)
}
}
android/src/main/kotlin/MainActivity.kt

RECEIVE d’ DATA!
@Post("/draw")
fun draw(@Body readingsSingle: List<Orientation>):
Single<ImageResult> {
return readingsSingle.map { readings ->
airDraw.run(readings)?.let { imageResult ->
bus.put(imageResult)
HttpResponse.ok(imageResult)
}
}
}

Things get crazy
val xl = KrigingInterpolation1D(t, x)
...
import java.awt.image.BufferedImage
val bi = BufferedImage(canvas.width, canvas.height, BufferedImage.TYPE_INT_ARGB)
...
val imgBytes = ByteString.copyFrom(bytes)
val img = Image.newBuilder().setContent(imgBytes).build()
val feature = Feature.newBuilder().setType(Type.LABEL_DETECTION).build()
val request = AnnotateImageRequest.newBuilder().addFeatures(feature).setImage(img).build()
val r = myImageAnnotatorClient.imageAnnotatorClient.batchAnnotateImages(arrayListOf(request))

Pitfalls
● Gradle Build Plugins
● JSON
● Externalizing the URL
● Math is hard
● ML is hard
● Compose is Young

github.com/GoogleCloudPlatform/air-draw-demo

Kotlin Mullets

More Related Content

What's hot

Similar to Kotlin Mullets

More from James Ward

Recently uploaded

Kotlin Mullets

Editor's Notes