The Image() type is set to have a storage of string for "path" and not large_string. Therefore while your change does work to do the conversion, it can create issues in other places. For example I'm pretty sure you wouldn't be able to concatenate the resulting dataset with a dataset with Image() using string.

Maybe we can convert large_string data to string somehow to make this work ?

@lhoestq thanks for the review. Just to be thorough I checked the concat example and this seems to work:

import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))

import pandas as pd
import polars as pl
from datasets import Dataset, Image, concatenate_datasets
import pyarrow as pa

image_path = "tests/features/data/test_image_rgb.jpg"


df_pl = pl.DataFrame({"image": [image_path]})
dset_pl = Dataset.from_polars(df_pl).cast_column("image", Image())


df_pd = pd.DataFrame({"image": [image_path]})
dset_pd = Dataset.from_pandas(df_pd).cast_column("image", Image())


concatenated = concatenate_datasets([dset_pl, dset_pd])
print(concatenated._data)

outputs:

ConcatenationTable
image: struct<bytes: binary, path: string>
  child 0, bytes: binary
  child 1, path: string
----
image: [
  -- is_valid: all not null
  -- child 0 type: binary
[null]
  -- child 1 type: string
["tests/features/data/test_image_rgb.jpg"],
  -- is_valid: all not null
  -- child 0 type: binary
[null]
  -- child 1 type: string
["tests/features/data/test_image_rgb.jpg"]]

(not quite sure though if this is a really what you meant). I agree that there could be pro a lot of problems if we rely on implicit conversion therefore I updated the PR. I also checked the exception handling locally and it works, am unsure though if we want to create such large objects in the CI, if desired I can add a test for that.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Apologies @lhoestq @CloseChoice , I unintentionally reverted this PR earlier. Leaving it as is.