KEMBAR78
fix megatron bert convert state dict naming by Codle · Pull Request #15820 · huggingface/transformers · GitHub
Skip to content

fix megatron bert convert state dict naming #15820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 18, 2022
Merged

fix megatron bert convert state dict naming #15820

merged 1 commit into from
Apr 18, 2022

Conversation

@Codle
Copy link
Contributor

@Codle Codle commented Feb 24, 2022 •
edited
Loading

What does this PR do?

Fixes some key of state_dict changing error in Megatron-Bert convert script. In checkpoint v3, MegatronLM seems to change attention -> self_attention, and HF has updated this key at GPT convert script#12007 while the BERT is still not.

layers.0.input_layernorm.weight
layers.0.input_layernorm.bias
layers.0.self_attention.query_key_value.weight
layers.0.self_attention.query_key_value.bias
layers.0.self_attention.dense.weight
layers.0.self_attention.dense.bias
layers.0.post_attention_layernorm.weight
layers.0.post_attention_layernorm.bias
layers.0.mlp.dense_h_to_4h.weight
layers.0.mlp.dense_h_to_4h.bias
layers.0.mlp.dense_4h_to_h.weight
layers.0.mlp.dense_4h_to_h.bias

Who can review?

@LysandreJik

@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

LysandreJik
LysandreJik approved these changes Mar 1, 2022
View reviewed changes
Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this looks good to me! Pinging @jdemouth as well!

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@LysandreJik
Copy link
Member

Merging

@LysandreJik LysandreJik merged commit 33cd4be into huggingface:main Apr 18, 2022
@LysandreJik LysandreJik mentioned this pull request Apr 18, 2022
Closed
5 tasks
elusenji pushed a commit to elusenji/transformers that referenced this pull request Jun 12, 2022
@Codle @elusenji
fix megatron bert convert state dict naming (huggingface#15820)
f5ab6da
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LysandreJik LysandreJik LysandreJik approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Codle @HuggingFaceDocBuilder @LysandreJik