KEMBAR78
Update to reflect new default benchmark settings by Quentin-Anthony · Pull Request #142 · NVIDIA-Merlin/HierarchicalKV · GitHub
Skip to content

Update to reflect new default benchmark settings #142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 13, 2023
Merged

Update to reflect new default benchmark settings #142

merged 1 commit into from
Jun 13, 2023

Conversation

@Quentin-Anthony
Copy link

@Quentin-Anthony Quentin-Anthony commented Jun 12, 2023 •
edited
Loading

It seems that this commit 921e9b8 updates the benchmark numbers but not the settings for the new largest KV.

i.e. dim = 64, capacity = 1024 Million-KV, HBM = 56 GB, HMEM = 200 GB in 921e9b8#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L159
should now be dim = 64, capacity = 512 Million-KV, HBM = 32 GB, HMEM = 96 GB

My numbers with the new benchmark defaults:

## Benchmark

* GPU: 1 x NVIDIA A100 80GB PCIe: 8.0
* Key Type = uint64_t
* Value Type = float32 * {dim}
* Key-Values per OP = 1048576
* Evict strategy: LRU
* `λ`: load factor
* `find*` means the `find` API that directly returns the addresses of values.
* `find_or_insert*` means the `find_or_insert` API that directly returns the addresses of values.
* ***Throughput Unit: Billion-KV/second***

### On pure HBM mode: 

* dim = 4, capacity = 64 Million-KV, HBM = 32 GB, HMEM = 0 GB

|    λ | insert_or_assign |   find | find_or_insert | assign |  find* | find_or_insert* | insert_and_evict |
|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|-----------------:|
| 0.50 |            1.169 |  3.019 |          1.837 |  1.974 |  4.469 |           1.843 |            1.033 |
| 0.75 |            1.019 |  2.981 |          0.681 |  0.888 |  1.980 |           1.288 |            0.899 |
| 1.00 |            0.370 |  3.031 |          0.372 |  0.507 |  0.942 |           0.393 |            0.324 |

* dim = 64, capacity = 64 Million-KV, HBM = 16 GB, HMEM = 0 GB

|    λ | insert_or_assign |   find | find_or_insert | assign |  find* | find_or_insert* | insert_and_evict |
|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|-----------------:|
| 0.50 |            0.863 |  2.132 |          0.926 |  1.116 |  4.483 |           1.841 |            0.771 |
| 0.75 |            0.672 |  2.119 |          0.571 |  0.792 |  1.989 |           1.292 |            0.579 |
| 1.00 |            0.324 |  2.179 |          0.324 |  0.470 |  0.931 |           0.393 |            0.232 |

### On HBM+HMEM hybrid mode: 

* dim = 64, capacity = 128 Million-KV, HBM = 16 GB, HMEM = 16 GB

|    λ | insert_or_assign |   find | find_or_insert | assign |  find* | find_or_insert* |
|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|
| 0.50 |            0.094 |  0.129 |          0.100 |  0.131 |  4.094 |           1.803 |
| 0.75 |            0.092 |  0.128 |          0.098 |  0.128 |  1.925 |           1.141 |
| 1.00 |            0.077 |  0.115 |          0.080 |  0.105 |  0.930 |           0.392 |

* dim = 64, capacity = 512 Million-KV, HBM = 32 GB, HMEM = 96 GB

|    λ | insert_or_assign |   find | find_or_insert | assign |  find* | find_or_insert* |
|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|
| 0.50 |            0.048 |  0.067 |          0.048 |  0.065 |  3.551 |           1.423 |
| 0.75 |            0.047 |  0.068 |          0.047 |  0.064 |  1.872 |           1.257 |
| 1.00 |            0.043 |  0.067 |          0.043 |  0.060 |  0.916 |           0.391 |

@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/HierarchicalKV/review/pr-142

@rhdong
Copy link
Member

rhdong commented Jun 12, 2023

/blossom-ci

@rhdong
Copy link
Member

rhdong commented Jun 13, 2023

Hi @Quentin-Anthony, thank you for taking the time to contribute and improve our project!

rhdong
rhdong approved these changes Jun 13, 2023
View reviewed changes
Copy link
Member

@rhdong rhdong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rhdong rhdong merged commit 2479f56 into NVIDIA-Merlin:master Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rhdong rhdong rhdong approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Quentin-Anthony @rhdong