[C++ Frontend] Kaiming Initialization #14718

JoshVarty · 2018-12-03T21:00:22Z

Working on #14582

The corresponding python implementations are at: pytorch/torch/nn/init.py

Here is my initial implementation of Kaiming Initialization. I have not been able to figure out how to successfully run tests locally so I haven't added any yet.

A couple questions:

Are the enums defined in the right place? I copied their names from Python, but do you prefer different naming conventions for C++?
To run tests locally do I use python setup.py test? Can I run just a subset of the tests somehow?
Should I add my tests at test/cpp/api/misc.cpp?

torch/csrc/api/src/nn/init.cpp

goldsborough

When you build PyTorch from source (see CONTRIBUTING.md), there will be a test_api binary in the build folder (use find test_api if you can't find it). You can filter tests by using `test_api --gtest_filter="TestSuite.TestCase" -- see https://github.com/abseil/googletest/blob/master/googletest/docs/primer.md. You'd want to add your test here

torch/csrc/api/include/torch/nn/init.h

torch/csrc/api/src/nn/init.cpp

ezyang · 2018-12-06T05:18:55Z

Hi @goldsborough, any update on this PR?

JoshVarty · 2018-12-06T05:27:18Z

Sorry for the delay, I've made the changes but haven't quite figured out how to run the tests. I'll try to get them running ASAP. I'd be interested in adding tests to this PR.

goldsborough · 2018-12-06T16:29:47Z

@JoshVarty where are you getting stuck? Have you located the test_api binary? You can write your test here

JoshVarty · 2018-12-06T19:14:04Z

@goldsborough I can't seem to find the binary, but I do see a folder called test_api/.

I'm running:

python setup.py build
cd build
find test_api

Which returns:

test_api/
test_api/CMakeFiles
test_api/CMakeFiles/progress.marks
test_api/CMakeFiles/CMakeDirectoryInformation.cmake
test_api/CMakeFiles/test_api.dir
test_api/CMakeFiles/test_api.dir/tensor.cpp.o
test_api/CMakeFiles/test_api.dir/build.make
test_api/CMakeFiles/test_api.dir/module.cpp.o
test_api/CMakeFiles/test_api.dir/DependInfo.cmake
test_api/CMakeFiles/test_api.dir/ordered_dict.cpp.o
test_api/CMakeFiles/test_api.dir/depend.internal
test_api/CMakeFiles/test_api.dir/tensor_options_cuda.cpp.o
test_api/CMakeFiles/test_api.dir/misc.cpp.o
test_api/CMakeFiles/test_api.dir/integration.cpp.o
test_api/CMakeFiles/test_api.dir/depend.make
test_api/CMakeFiles/test_api.dir/dataloader.cpp.o
test_api/CMakeFiles/test_api.dir/static.cpp.o
test_api/CMakeFiles/test_api.dir/link.txt
test_api/CMakeFiles/test_api.dir/__
test_api/CMakeFiles/test_api.dir/__/common
test_api/CMakeFiles/test_api.dir/__/common/main.cpp.o
test_api/CMakeFiles/test_api.dir/expanding-array.cpp.o
test_api/CMakeFiles/test_api.dir/progress.make
test_api/CMakeFiles/test_api.dir/cmake_clean.cmake
test_api/CMakeFiles/test_api.dir/serialize.cpp.o
test_api/CMakeFiles/test_api.dir/sequential.cpp.o
test_api/CMakeFiles/test_api.dir/tensor_cuda.cpp.o
test_api/CMakeFiles/test_api.dir/tensor_options.cpp.o
test_api/CMakeFiles/test_api.dir/jit.cpp.o
test_api/CMakeFiles/test_api.dir/CXX.includecache
test_api/CMakeFiles/test_api.dir/any.cpp.o
test_api/CMakeFiles/test_api.dir/parallel.cpp.o
test_api/CMakeFiles/test_api.dir/optim.cpp.o
test_api/CMakeFiles/test_api.dir/memory.cpp.o
test_api/CMakeFiles/test_api.dir/modules.cpp.o
test_api/CMakeFiles/test_api.dir/rnn.cpp.o
test_api/CMakeFiles/test_api.dir/flags.make
test_api/cmake_install.cmake
test_api/CTestTestfile.cmake
test_api/Makefile

P.S. I believe there's a PyTorch Slack right? If it's easier, we could discuss this build stuff there. I sent a couple emails to slack@pytorch.org last week but I haven't heard back.

goldsborough · 2018-12-06T19:40:29Z

You should run python setup.py build develop and then run find . -name test_api, find test_api just lists that folder. I think the binary is in build/bin/test_api. @soumith can you check about the slack invite?

JoshVarty · 2018-12-06T23:05:23Z

I wanted to reimplement these tests in C++ but I'm having trouble because those tests use scipy.stats.kstest to compare two distributions against one another.

Is there a similar implementation in C++? Or would I essentially be looking at re-implementing this SciPy functionality? If I'd have to reimplement it, it may be better to create a new issue and discuss that approach there. One positive is that it would also allow us to test the xavier_* weight initialization methods as well.

test/cpp/api/misc.cpp

goldsborough · 2018-12-07T17:00:40Z

@JoshVarty One thing you can do is verify that the output in C++ matches that in Python. We do this for the optimizers for example: https://github.com/pytorch/pytorch/blob/master/test/cpp/api/optim_baseline.py

For this, you'd write a small PyTorch script that generates a C++ header file containing some constants for every test case, and then include that header in the C++ test file, and compare the output of the same code in C++ to the Python outputs you generated. What do you think of this idea?

torch/csrc/api/include/torch/nn/init.h

torch/csrc/api/src/nn/init.cpp

ezyang · 2018-12-20T21:37:16Z

@goldsborough is this ready to go?

goldsborough · 2019-01-03T17:35:25Z

@ezyang It is not because there are no tests yet. @JoshVarty are you still interested in wrapping up this PR by adding some tests? It looks great so far, thanks for your work.

JoshVarty · 2019-01-07T18:16:58Z

Yeah I would like to add the tests. I haven't had much time to look at this over the holidays but I anticipate I'll have some free time this week to look at it again.

ShahriarRezghi · 2019-02-01T19:37:37Z

Hi.
I am able to build and test the code so if there is anything I can do to speed this along please let me know.

JoshVarty · 2019-02-02T07:47:09Z

I wiped my Ubuntu partition and started from scratch and that seems to have got everything working again. I've started the baselines and will create the C++ tests shortly.

torch/csrc/api/src/nn/init.cpp

yf225 · 2019-02-04T21:39:54Z

@JoshVarty Could you pull master into your branch to fix the macos test errors? Thanks :)

…ingInitCpp

JoshVarty · 2019-02-05T00:40:38Z

@yf225 I merged master and the tests re-ran but they seem to fail on linux.

…ingInitCpp

yf225 · 2019-02-05T23:29:16Z

@JoshVarty the current CI errors are unrelated and can be ignored

@goldsborough @fmassa do you think the tests look good and the PR is ready to go?

goldsborough

Looks great @JoshVarty, thanks! I left a few comments on things that could be improved, like the lambdas in init.cpp and moving the other nn::init-related tests into init.cpp. I'll approve it already, and merge it once you have time for the last few fixes

test/cpp/api/init.cpp

test/cpp/api/misc.cpp

goldsborough · 2019-02-07T17:14:16Z

JoshVarty · 2019-02-12T05:08:19Z

@goldsborough This is probably good for another review if you get a minute. I think the only outstanding issue is my comment/question about how to handle the lambdas.

goldsborough · 2019-02-12T05:31:08Z

Looks great, I'll land it tomorrow. Thanks again!

facebook-github-bot

@goldsborough is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@goldsborough is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@goldsborough is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

JoshVarty added 2 commits December 3, 2018 15:54

Add Kaiming and calculate_gain to API

5d8355e

Implement Kaiming initialization

426cc8e

JoshVarty requested review from ebetica and goldsborough as code owners December 3, 2018 21:00

JoshVarty commented Dec 3, 2018

View reviewed changes

torch/csrc/api/src/nn/init.cpp Show resolved Hide resolved

vishwakftw reviewed Dec 4, 2018

View reviewed changes

torch/csrc/api/src/nn/init.cpp Outdated Show resolved Hide resolved

vishwakftw reviewed Dec 4, 2018

View reviewed changes

torch/csrc/api/src/nn/init.cpp Outdated Show resolved Hide resolved

vishwakftw reviewed Dec 4, 2018

View reviewed changes

torch/csrc/api/src/nn/init.cpp Outdated Show resolved Hide resolved

goldsborough reviewed Dec 4, 2018

View reviewed changes

goldsborough changed the title ~~WIP: Kaiming Initialization for C++ API~~ [WIP][C++ Frontend] Kaiming Initialization Dec 4, 2018

goldsborough added the module: cpp Related to C++ API label Dec 4, 2018

JoshVarty added 3 commits December 4, 2018 15:44

Address PR feedback

8fca2e8

Run clang-format on changes

a1e3a98

Use std::sqrt instead of sqrt

8af0bd4

Add a few tests for calculate_gain

f305348

Format with clang-form

6ed28a6

goldsborough reviewed Dec 7, 2018

View reviewed changes

test/cpp/api/misc.cpp Outdated Show resolved Hide resolved

goldsborough reviewed Dec 7, 2018

View reviewed changes

torch/csrc/api/include/torch/nn/init.h Outdated Show resolved Hide resolved

torch/csrc/api/include/torch/nn/init.h Outdated Show resolved Hide resolved

torch/csrc/api/src/nn/init.cpp Outdated Show resolved Hide resolved

torch/csrc/api/src/nn/init.cpp Outdated Show resolved Hide resolved

JoshVarty added 2 commits December 7, 2018 17:33

Address PR Feedback

b233a79

Use ASSERT_DOUBLE_EQ for floating point comparisons

d195d84

Initial implementation of initialization baseline

f3dde2c

JoshVarty added 3 commits February 3, 2019 00:43

Add tests for Xavier and Kaiming Init

e4b4444

Add NoGrad to Kaiming init

7e2dbb9

Format init.cpp

f467040

JoshVarty commented Feb 3, 2019

View reviewed changes

torch/csrc/api/src/nn/init.cpp Show resolved Hide resolved

Fix python formatting

d78ce07

Merge branch 'master' of https://github.com/pytorch/pytorch into Kaim…

359f916

…ingInitCpp

Merge branch 'master' of https://github.com/pytorch/pytorch into Kaim…

5cc188a

…ingInitCpp

goldsborough approved these changes Feb 7, 2019

View reviewed changes

Address most of PR Feedback

1c47fa5

facebook-github-bot reviewed Feb 12, 2019

View reviewed changes

goldsborough changed the title ~~[WIP][C++ Frontend] Kaiming Initialization~~ [C++ Frontend] Kaiming Initialization Feb 15, 2019

Merge branch 'master' into KaimingInitCpp

aa72ba8

facebook-github-bot reviewed Feb 15, 2019

View reviewed changes

facebook-github-bot closed this in 1cdcdd7 Feb 15, 2019

JoshVarty deleted the KaimingInitCpp branch February 16, 2019 01:00

ezyang added open source merged labels Jun 24, 2019

yf225 mentioned this pull request Jul 24, 2019

Add Kaiming initialization functions to C++ API #14582

Closed

yf225 mentioned this pull request Sep 25, 2019

Python/C++ API Parity: torch.nn modules and functional #25883

Open

[C++ Frontend] Kaiming Initialization #14718

[C++ Frontend] Kaiming Initialization #14718

Uh oh!

Conversation

JoshVarty commented Dec 3, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

goldsborough left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ezyang commented Dec 6, 2018

Uh oh!

JoshVarty commented Dec 6, 2018

Uh oh!

goldsborough commented Dec 6, 2018

Uh oh!

JoshVarty commented Dec 6, 2018

Uh oh!

goldsborough commented Dec 6, 2018

Uh oh!

JoshVarty commented Dec 6, 2018

Uh oh!

Uh oh!

goldsborough commented Dec 7, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ezyang commented Dec 20, 2018

Uh oh!

goldsborough commented Jan 3, 2019

Uh oh!

JoshVarty commented Jan 7, 2019

Uh oh!

ShahriarRezghi commented Feb 1, 2019

Uh oh!

JoshVarty commented Feb 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yf225 commented Feb 4, 2019

Uh oh!

JoshVarty commented Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yf225 commented Feb 5, 2019

Uh oh!

goldsborough left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

goldsborough commented Feb 7, 2019

Uh oh!

JoshVarty commented Feb 12, 2019

Uh oh!

goldsborough commented Feb 12, 2019

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

JoshVarty commented Feb 2, 2019 •

edited

Loading

JoshVarty commented Feb 5, 2019 •

edited

Loading