KEMBAR78
CMP-7543 Retain AnnotatedString in ClipEntries on Desktop by rock3r · Pull Request #2092 · JetBrains/compose-multiplatform-core · GitHub
Skip to content

Conversation

@rock3r
Copy link

@rock3r rock3r commented May 9, 2025

Prior to this, the implementation of ClipboardUtils.desktop.kt would "flatten" AnnotatedStrings into their plaintext representation. This means that if you implemented a custom Clipboard, it would only receive the plaintext representation of the copied text in setEntry. With this change, the AnnotatedString is actually preserved when copying, so Clipboard can access the rich text and serialize it as it prefers.

It should not change the observed behaviour when using the default Clipboard, but allows users to create custom ones which can do things like JEWEL-815.

PS: this removes the withContext calls from ClipboardUtils.desktop.kt since they optimise for rare cases (extremely large content in the clipboard, probably O(100MB) or more) which are going to clog on the AWT side or during composition anyway. It can cause a small perf degradation in the overwhelmingly more common scenarios of tiny-to-large clipboard contents due to the dispatching of the coroutine to the IO dispatcher. This was discussed with someone from the Compose Text team.

Testing

Ran tests, smoke tested by hand.

Release Notes

Features - Desktop

  • Full AnnotatedString is available as a data flavor in ClipEntry, instead of only its text.

DataFlavor(AnnotatedString::class.java, "AnnotatedString")

@Suppress("UnusedReceiverParameter")
val DataFlavor.annotatedStringFlavor: DataFlavor
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A public API change require running ./gradlew :mpp:jbApiDump and commiting the change.

Copy link
Member

@eymar eymar May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @igordmn, Igor adding you as a reviewer as here we have an API change for desktop. (Also there was a discussion in a slack thread - https://jetbrains.slack.com/archives/C076VK0LGH1/p1746625562243429)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, API dump updated!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, this is necessary for users to be able to use the Transferable#isDataFlavorSupported(DataFlavor) and Transferable#getTransferData(DataFlavor) APIs. Otherwise, they'd have to guess what the flavour to use is...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave it another thought.

IMO, It would be better to move a new public property to ui module, in PlatformClipboard.desktop.kt, so the new property won't expose an extra public class ClipboardUtils_desktopKt (we can see it now in foundation.api dump).

And adding @ExperimentalComposeUiApi to it would be safer for now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is easy to break Jewel and any other users when there is no actual API that you're changing/breaking

It will be a definite breakage after CMP-7558, as the main purpose of it is to not expose implementation details to the external users.

So, is the main use case is to retrieve the annotated string from ClipEntry? If so, then my suggestion of removing won't help your case if you still need creating DataFlavor("AnnotatedString").

Considering the AWT coupling issue, the best way to solve this case is to have public functions that provide AnnotatedString. Similar to the internal ones:
image

It is a big task, which should be done in AOSP first as common API. There are probably alternatives, but before suggesting one, I would like to understand the main issue more.

After this PR we don't lose information in the standard copy/paste actions anymore. Why we still need a custom clipboard and retrieve the annotated string from the default one? Suppose, we copied a text from Text/BasicTextField, and Compose created an AnnotatedString and stored it in the default clipboard. Where we need to retrieve it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be a definite breakage after CMP-7558, as the main purpose of it is to not expose implementation details to the external users.

We can change the Jewel implementation (we'll keep it internal) when CMP-7558 ships and we get it in Jewel, that is not a huge problem.

So, is the main use case is to retrieve the annotated string from ClipEntry? If so, then my suggestion of removing won't help your case if you still need creating DataFlavor("AnnotatedString").

Yes it is, but the main issue this PR fixes is that right now the AnnotatedString that gets passed to Clipboard is stripped of all annotations. I can live with having to have my own copy of the DataFlavor until CMP-7558 is done, and we can use whatever better API there will be.

Considering the AWT coupling issue, the best way to solve this case is to have public functions that provide AnnotatedString. Similar to the internal ones:
[...]
It is a big task, which should be done in AOSP first as common API. There are probably alternatives, but before suggesting one, I would like to understand the main issue more.

Agreed, that would be the ideal scenario, but I was told we could not add any new public API, especially in commonMain. In the meantime though this is a way (undocumented, unofficial) that still allows Jewel to fix a pretty bad UX when copying rich text.

After this PR we don't lose information in the standard copy/paste actions anymore. Why we still need a custom clipboard and retrieve the annotated string from the default one? Suppose, we copied a text from Text/BasicTextField, and Compose created an AnnotatedString and stored it in the default clipboard. Where we need to retrieve it?

Jewel will implement its own Clipboard (see this PR for example) where it will take the AnnotatedString and convert it to raw Markdown, which is a format that can be then put in the AWT clipboard. This is necessary, because we can't put non-serializable content in the system clipboard. It is also what IJ does in general when you copy some highlighted code to the clipboard — it has a mechanism to "convert" said rich text to HTML and RTF, so it can preserve its formatting when pasted elsewhere.

Copy link
Collaborator

@igordmn igordmn Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is okay that it will definitely break, we can merge it as is, though, let's add a comment for private val annotatedStringFlavor: DataFlavor:

This implementation detail is used by Jewel. When removing it, please provide an alternative of retrieving an annotated string, and notify a Jewel developer that they need to change the implementation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After https://youtrack.jetbrains.com/issue/CMP-7558/New-Clipboard-Adoption.-Review-API or https://youtrack.jetbrains.com/issue/CMP-7624/Design-ClipEntry-common-API there should be a way to retrieve the annotated string: defaultClipEntryFactory.asAnnotatedString(clipEntry), clipEntry.asAnnotatedString().

Something like awtAnnotatedStringFlavor: DataFlavor probably won't be exposed as API for desktop, as it is only one of the many ways to provide it. Instead of putting the annotated string to transferable, we could put our own class with metadata, or even don't create Transferable until it is needed for passing to the clipboard.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is okay that it will definitely break, we can merge it as is, though, let's add a comment for private val annotatedStringFlavor: DataFlavor

Done

After https://youtrack.jetbrains.com/issue/CMP-7558/New-Clipboard-Adoption.-Review-API or https://youtrack.jetbrains.com/issue/CMP-7624/Design-ClipEntry-common-API there should be a way to retrieve the annotated string: defaultClipEntryFactory.asAnnotatedString(clipEntry), clipEntry.asAnnotatedString().

Something like awtAnnotatedStringFlavor: DataFlavor probably won't be exposed as API for desktop, as it is only one of the many ways to provide it. Instead of putting the annotated string to transferable, we could put our own class with metadata, or even don't create Transferable until it is needed for passing to the clipboard.

That's fine. As long as we still have access to the full AnnotatedString in Clipboard via whatever API ends up being there, we are ok with that.

@eymar eymar requested a review from igordmn May 9, 2025 09:47
@rock3r rock3r force-pushed the sebp/annotatedstrings-in-clipentries-on-desktop branch from 150f8b6 to 7f24fbc Compare May 9, 2025 10:37
DataFlavor(AnnotatedString::class.java, "AnnotatedString")

@Suppress("UnusedReceiverParameter")
val DataFlavor.annotatedStringFlavor: DataFlavor
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, API dump updated!

DataFlavor(AnnotatedString::class.java, "AnnotatedString")

@Suppress("UnusedReceiverParameter")
val DataFlavor.annotatedStringFlavor: DataFlavor
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, this is necessary for users to be able to use the Transferable#isDataFlavorSupported(DataFlavor) and Transferable#getTransferData(DataFlavor) APIs. Otherwise, they'd have to guess what the flavour to use is...

// TODO: https://youtrack.jetbrains.com/issue/CMP-7543/Support-styled-text-in-PlatformClipboard-implementations
internal actual suspend fun ClipEntry.readAnnotatedString(): AnnotatedString? {
return readText()?.let { AnnotatedString(it) }
if (!hasAnnotatedString()) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is done to retain the existing behaviour in androidx.compose.foundation.text.selection.TextFieldSelectionManager#paste, which only tries to get an AnnotatedString:

val text = clipboard?.getClipEntry()?.readAnnotatedString() ?: return@launch

I will propose to Jetpack Compose to alter this slightly as:

val text = clipboard?.getClipEntry()?.readAnnotatedString()
  ?: clipboard?.getClipEntry()?.readText()?.let { AnnotatedString(it) }
  ?: return@launch

To make it more explicit at the usage site, but for now this covers that case.

}

override fun lostOwnership(clipboard: Clipboard?, contents: Transferable?) {
// Empty
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why StringSelection implements ClipboardOwner only to have its only function do nothing; I however am going to retain the same behaviour, just in case it covers some unintuitive edge cases. It shouldn't cause any issues, judging by the JavaDoc of ClipboardOwner.

}

companion object {
private val supportedFlavors = arrayOf(_annotatedStringFlavor, DataFlavor.stringFlavor)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: when AWT puts this transferable in the clipboard, it drops the AnnotatedString flavour, since it's not serializable. The net result is having just the plain text put in the system clipboard, which is consistent with the current behaviour.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it drops the AnnotatedString flavour, since it's not serializable.

What if I have this code in the same app (ignoring the fact I use internal functions here):

val annotatedString = AnnotatedString(...)
val clipEntry = annotatedString.toClipEntry()!!
clipboard.setClipEntry(clipEntry)
...

val result = clipboard.getClipEntry().readAnnotatedString()

is result expected to be equal to annotatedString ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's what one of the tests is checking, too.

In prod code however the clip entry is handed immediately to the Clipboard, which in turns sets it immediately in the AWT clipboard, which drops the annotated string as described above. So, the only time the annotated string really "exists" in the transferable is while in transit via the Compose clipboard. As soon as it's handed over to AWT it is discarded.

}
val typeface = when (font) {
is ResourceFont -> typefaceResource(font.name)
// TODO: replace with FontMgr.makeFromFile(font.file.toString())
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this TODO I left here a couple years ago as it seems it was addressed in the line below. Someone just forgot to remove the TODO :)

transferable ?: EmptyTransferable,
null
/* contents = */ transferable ?: EmptyTransferable,
/* owner = */ transferable as? ClipboardOwner,
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was technically wrong before, as we are supposed to provide the owner to the AWT clipboard if there is one. Given the Transferable itself is the only way we could have an owner, if it implements ClipboardOwner we pass it as owner.

ClipboardOwner is a simple interface that only provides a callback to know "you're not the owner of the clipboard any longer", meaning something else was put in the clipboard. I think this can be needed to do some cleanup in some Transferables which may be holding on to resources internally.

@rock3r rock3r force-pushed the sebp/annotatedstrings-in-clipentries-on-desktop branch 2 times, most recently from ce7e4c9 to 4a22adf Compare May 13, 2025 10:36
@rock3r rock3r force-pushed the sebp/annotatedstrings-in-clipentries-on-desktop branch from 4a22adf to 45a28c9 Compare May 15, 2025 09:10
@m-sasha
Copy link
Member

m-sasha commented May 15, 2025

I'd like for Igor to weigh in on this before merging. It feels somewhat of a hack that is only useful because Clipboard is exposed as an overridable composition local, allowing JEWEL to override it. Typical users should not be expected to do this.

Locally, I don't like annotatedStringFlavor and the need to keep it in sync it across modules. Can we use DataFlavor.javaJVMLocalObjectMimeType?

@rock3r
Copy link
Author

rock3r commented May 15, 2025

@m-sasha

It feels somewhat of a hack that is only useful because Clipboard is exposed as an overridable composition local [...] Typical users should not be expected to do this.

This is in any case a necessary change, even if you wanted to also figure out a way to serialize and deserialize the AnnotatedString in a format that AWT’s clipboard likes. You can also do that, but it doesn’t invalidate this step being needed anyway. I would not block this change that has no adverse side effects, and is not a hack, on that other effort, which is non-trivial to say the least (trust me, I tried).

Yes, it unblocks Jewel being able to access the AnnotatedString, and beyond us maybe only Kirill’s Aurora theme could benefit from this directly, but... why is that an issue?

Can we use DataFlavor.javaJVMLocalObjectMimeType?

We could, that doesn’t really change the substance, though, and still requires manually adjusting the mime type to also carry the information about the original data class. For example, where IJ uses it, it does something like:

// PsiCopyPasteManager
ourDataFlavor = new DataFlavor(DataFlavor.javaJVMLocalObjectMimeType + ";class=" + flavorClass.getName());

Which to me feels like even more of a hack. It also has the same drawback of requiring users to know how to conjure up the right mime type, unless we provide a DataFlavor for them, which in turn makes it essentially identical to the what is already in this PR.

@m-sasha
Copy link
Member

m-sasha commented May 15, 2025

I guess I don't understand why we need to expose the DataFlavor, an AWT thing, at all.

@m-sasha
Copy link
Member

m-sasha commented May 15, 2025

We don't expose the fact that plain text goes into the clipboard with DataFlavor.stringFlavor and DataFlavor.plainTextFlavor.

@rock3r
Copy link
Author

rock3r commented May 15, 2025

Because y'all have aliased the native APIs to the AWT ones, and it's not like there are many alternatives to that... I can't think of any, at least. If you have alternative ideas I am happy to hear them. Would making ClipEntry.hasAnnotatedString and ClipEntry.readAnnotatedString public (experimental) API be better? That saves you from exposing AWT related APIs, but also requires copy-pasting those methods in ui-desktop from foundation-common...

@rock3r
Copy link
Author

rock3r commented May 15, 2025

We don't expose the fact that plain text goes into the clipboard with DataFlavor.stringFlavor and DataFlavor.plainTextFlavor.

Due to a lack of any real API to deal with ClipEntries, desktop users who want to access that need to know that ClipEntry is an alias to Transferable, and then that they can request the plain text by using DataFlavor.string, which leaves them at exactly the same level as using the PR's implementation leaves them at.

The difference is that they have those flavour constants defined in AWT already, while there isn't one defined for AnnotatedString. If we remove the flavour, users would be in the same situation as if DataFlavor.string didn't exist (plainTextFlavor is deprecated and I'll be ignoring it).

We can remove the new data flavor and I can just have it in Jewel, but at that point we're really just obfuscating something for the sake of obfuscation.

internal fun ClipEntry?.hasAnnotatedString(): Boolean {
if (this == null) return false
val transferable = nativeClipEntry as? Transferable ?: return false
return transferable.isDataFlavorSupported(annotatedStringFlavor)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the transferable is plain text, we should probably return true here as well.

Copy link
Author

@rock3r rock3r May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not done on purpose, because if you do, you have no way to distinguish the case where there is plain text, and there is no AnnotatedString. It doesn't happen when we put things in the clipboard, but nothing prevents users from putting whatever they want in the clipboard, including a situation like this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure you need to distinguish that. If you're expecting to receive an annotated string, you call has/readAnnotatedString and that's it.

As-is, what we have is inconsistent: hasAnnotatedString can return false and then readAnnotatedString returns a non-null value.

Copy link
Author

@rock3r rock3r May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also to note, we're assuming that the plain text and annotated string contents are interchangeable as at most the plain text is annotatedString.text. Compose does, in general. In normal cases, they are. But here again, absolutely nothing says they must be. You might have a different semantic representation of the same content between plain text and AnnotatedString.

An example; an app puts this text as AnnotatedString: "Hello world". But, for some reason that makes sense to them, the plain text content is Hello *world* — the app is expressing the bold part in the plain text as asterisks, to maintain the intent, even though the representations now have diverged. This is a perfectly valid use case.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure you need to distinguish that. If you're expecting to receive an annotated string, you call has/readAnnotatedString and that's it.

These are internal APIs. hasAnnotatedString isn't even an expect/actual, it's literally only used in the desktop target. Users can't access these.

As-is, what we have is inconsistent: hasAnnotatedString can return false and then readAnnotatedString returns a non-null value.

This is only needed because @eymar said that we shouldn't change the commonMain code that uses these APIs (it only uses read* APIs), but this works around an issue in the implementation that only ever calls readAnnotatedString and never falls back to readText. I think you can find that conversation in some other comments to this PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Said problematic code in commonMain:

image

@m-sasha
Copy link
Member

m-sasha commented May 15, 2025

The reason there's no normal API to put/get clipboard data is that we haven't implemented it yet.
We should do that, and without exposing its internals.

@rock3r
Copy link
Author

rock3r commented May 15, 2025

If the only thing that's stopping this PR from going in is the public flavour, I can remove it. Until there is a proper API we can use, I'll keep a copy of the flavour in Jewel and use it there. Not the end of the world for me.

@m-sasha
Copy link
Member

m-sasha commented May 15, 2025

I think my opposition to this is due to the fact that I haven't yet thought about what we want to do with the clipboard API, so I don't know whether this change is aligned with that.

But if this improves the situation for jewel, without creating problems for us down the line, I'm not against it.

I would still like for Igor to look at it, though.

DataFlavor(AnnotatedString::class.java, "AnnotatedString")

@Suppress("UnusedReceiverParameter")
val DataFlavor.annotatedStringFlavor: DataFlavor
Copy link
Collaborator

@igordmn igordmn Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, this is necessary for users to be able to use the Transferable#isDataFlavorSupported(DataFlavor) and Transferable#getTransferData(DataFlavor) APIs.

Could you elaborate more, and provide code how it should be used in outside code?

If it is not for the outside code, I would suggest to remove it as public API. The AWT implementation should be only one of many possible implementations.

Detailed explanation

There is an architectural issue in the current API in jb-main that the Clipboard API is tightly coupled with AWT.

It is tightly coupled because it creates AWT-based classes in one place (in toClipEntry), and uses casting to AWT classes in another place (in readAnnotatedString/readText). Instead of that we had to keep ClipEntry platform-independent, and store raw data in it, not the AWT based objects. Convert to AWT based class only:

  • in places where it is needed - in the copy/paste actions itself, with supporting different implementations via CompositionLocal or a global factory.
  • as interop functions for external users, when they know that the current implementation is the AWT one.

In other words, we shouldn't use nativeClipEntry and derivatives (awtTransferable) in Compose itself, we should provide them for external interop.

The overall review and necessary changes to the current API will be done in https://youtrack.jetbrains.com/issue/CMP-7558/New-Clipboard-Adoption.-Review-API. For this PR, we should try to not make things more coupled.

return readText()?.let { AnnotatedString(it) }
}

val transferable = nativeClipEntry as? Transferable
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having readAnnotatedString isn't fine because of the issue I described, but because we already have it coupled in readText, it is fine for this PR, and it will be fixed together with the other functions later

@rock3r rock3r force-pushed the sebp/annotatedstrings-in-clipentries-on-desktop branch from 45a28c9 to 7d6d0ec Compare June 10, 2025 14:45
Prior to this, the implementation of ClipboardUtils.desktop.kt would "flatten" AnnotatedStrings into their plaintext representation. This means that if you implemented a custom Clipboard, it would only receive the plaintext representation of the copied text in setEntry. With this change, the AnnotatedString is actually preserved when copying, so Clipboard can access the rich text and serialize it as it prefers.
@rock3r rock3r force-pushed the sebp/annotatedstrings-in-clipentries-on-desktop branch from 7d6d0ec to ad7f146 Compare June 11, 2025 08:13
@igordmn igordmn merged commit 9d7a908 into JetBrains:jb-main Jun 11, 2025
3 checks passed
@rock3r rock3r deleted the sebp/annotatedstrings-in-clipentries-on-desktop branch June 11, 2025 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants