The training process of ImageBind-LLM consists of the pre-training and fine-tuning phases.

Pre-training

Data

We use multiple datasets with image-text pairs for pre-training. The texts are English-only.
For each dataset, the meta file should be organized in the .csv format as following:
```
url		caption
/path/to/image1		caption1
/path/to/image2		caption2
...
```
Alternatively, you may modify the PretrainDataset implementation to adapt to your own meta file format.
Write a .yaml config file to specify the datasets for pre-training:
```
META:
  - '/path/to/cc3m.csv'
  - '/path/to/coco.csv'
  ...
```

Start pre-training

We are now ready to start pre-training (please make sure that the original LLaMA / Open-Chinese-LLaMA weights are available in /path/to/llama_model_weights).

. exps/pretrain.sh /path/to/llama_model_weights /path/to/pretrain-data-config.yaml /output/path

Pre-trained checkpoint

We release our image-text pre-trained checkpoint for reproduction. You may directly use it for fine-tuning.

Name	Link
7B-pretrained.pth	File Link

Fine-tuning

Data

We fine-tune ImageBind-LLM on text-only as well as image-text instruction following datasets.
The following lists the datasets we use for training our release weights:

Name Link

alpaca_gpt4_data.json File Link

alpaca_gpt4_data_zh.json File Link

llava_instruct_150k.json File Link

alpaca_data_zh_51k.json File Link

Similar to pre-training, write a .yaml config file to specify the datasets for fine-tuning:

META:
  - '/path/to/alpaca_gpt4_data.json'
  - '/path/to/alpaca_gpt4_data_zh.json'
  ...

Start fine-tuning

. exps/finetune.sh \
 /path/to/llama_model_weights /path/to/pre-trained/checkopint.pth \
 /path/to/finetune-data-config.yaml /output/path

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!