[C++ API] Add named submodule support to nn::Sequential #17552

yf225 · 2019-02-27T20:06:08Z

Previously, we were not able to assign names to nn::Sequential's submodules. This PR adds this feature to match the Python API. Example use:

Sequential sequential(named_submodule({
      {"linear", Linear(10, 3)},
      {"conv2d", Conv2d(1, 2, 3)},
      {"dropout", Dropout(0.5)},
      {"batchnorm", BatchNorm(5)},
      {"embedding", Embedding(4, 10)},
      {"lstm", LSTM(4, 5)}
}));

It also enables loading parameters of Python nn.Sequential module with custom submodules names into C++ frontend, unblocking pytorch/vision#728 (comment).

facebook-github-bot

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

gchanan · 2019-02-28T18:31:00Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+
+  /// Adds a new named module to the `Sequential` container, with name of `std::string` type.
+  template <typename M>
+  void push_back(std::string name, M module) {


I don't understand why the public string parameter versions of push_back don't match the public non-string versions. I would expect you would just have string overloads for the 3 existing ones?

gchanan · 2019-02-28T18:32:33Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+
+  /// Adds a new named module to the `Sequential` container, with name of `const char*` type.
+  template <typename M>
+  void push_back(const char* name, M module) {


why do you have both string and const char* versions of push_back?

torch/csrc/api/include/torch/nn/modules/sequential.h

gchanan · 2019-02-28T18:46:29Z

test/cpp/api/sequential.cpp

+
+  Sequential sequential(
+    std::make_shared<M>(1),
+    "m2", std::make_shared<M>(2),


this behavior is kind of crazy and doesn't match python.

ebetica · 2019-02-28T18:28:37Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+  template <typename M>
+  void push_back(std::string name, M module) {
+    auto index = add_to_modules(module);
+    register_module(name, modules_[index].ptr());


nit: can std::move(name) here.

test/cpp/api/sequential.cpp

ebetica · 2019-02-28T18:47:22Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+
+  /// Adds a new named module to the `Sequential` container, with name of `const char*` type.
+  template <typename M>
+  void push_back(const char* name, M module) {


I believe this does an extra copy of the module unnecessarily, and doesn't really follow the convention set above. I'm not sure how expensive the copy constructor is for Module.

It may be better to follow the construct given above, having a separate function for each of the possible types, i.e.

std::shared_ptr

const ModuleHolder&

M&& with M being a module

Alternatively if you think the construct doesn't matter and the extra copy is whatever, then we can simplify the above code, but I think it's important to be consistent.

gchanan · 2019-02-28T18:55:43Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+  template <typename M, typename = torch::detail::enable_if_module_t<M>>
+  size_t add_to_modules(M&& module) {
+    // Need to get rid of any reference components for make_unique.
+    using Type = typename std::remove_reference<M>::type;


I also don't quite get the point of add_to_modules (are you always properly moving/forwarding parameters around)? Why don't the "final" non-string versions of push_back just call the string versions?

yf225 · 2019-02-28T19:42:56Z

Note to self: make sure we are not making unnecessary copies (by adding tests for it)

goldsborough

I think @gchanan captures my opinion here in that the changes to Sequential are more complicated than they need to be. Essentially you should just need one overload for every existing push_back that has a string as the first parameter. The existing methods can just call into the new methods, using modules_.size() (the next index) as the name. I would do it like this:

Starting with the previous code, turn every push_back into a named version. All the logic should be in the named functions.
Thread the name through to the final insertion into the map
For every named version, add an unnamed overload (the previous signatures), but all they do is call push_back(std::to_string(modules_.size()), <module>).

Then you effectively just have to add three methods and that's it

goldsborough · 2019-03-05T04:00:54Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+
+  /// Adds a new named module to the `Sequential` container, with name of `const char*` type.
+  template <typename M>
+  void push_back(const char* name, M module) {


I don't think we need const char* in the interface, since we eventually store std::string. It's ok to just have std::string as the argument type and move that properly into the eventual storage

goldsborough · 2019-03-05T04:02:44Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+
+  /// Matches `Sequential("m1", Module(1), ...)` case
+  template <typename Module, typename... Rest>
+  void push_back(const char* name, Module&& module, Rest&&... rest) {


You can remove this overload and the whole const char* business, it's not worth it

gchanan · 2019-03-06T19:14:29Z

@goldsborough a few opinion questions for you:

what do you think about mixing named and "unnamed" parameters. Should we allow it or not?
Having a constructor that takes an OrderedDict of modules would match the python interface. But it seems like constructing an OrderedDict of modules by hand is annoying because of the different module types. Should we have a helper that does it? I guess you'd basically have to reproduce the module bits of Sequential here to do it though, i.e. three overloads: shared_ptr<ModuleType>, ModuleHolder<M>, AnyModule.
Is there documentation that explains those three overloads anywhere? I think I found some documentation around why there isn't a single module type (which makes sense), but I couldn't find docs about why those 3 are the "magic 3".

yf225 · 2019-03-07T00:06:46Z

Another issue I found is that if I remove the copy constructor (by adding M(const M&) = delete;) for the concrete module type test: https://github.com/pytorch/pytorch/blob/master/test/cpp/api/sequential.cpp#L38-L44, the test will fail to compile, because

template <typename M, typename = torch::detail::enable_if_module_t<M>>
void push_back(M&& module) {
    // Need to get rid of any reference components for make_unique.
    using Type = typename std::remove_reference<M>::type;
    // Here we move (or copy) the module into a new shared_ptr.
    push_back(std::make_shared<Type>(std::forward<M>(module))); // NOTE: This line copies the module
}

(https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/nn/modules/sequential.h#L202) actually copies the module and expects the copy constructor to exist. Ideally this copy should be avoided, and I am investigating how we can achieve so.

This reverts commit 848c7fc.

yf225 · 2019-03-11T20:07:41Z

I figured out how to have a simple API for making OrderedDict work, and updated the PR to reflect the new approach.

With the new named_submodules() API, we will be creating an OrderedDict of named submodules much like how we do it with the Python API, which is much better for API parity.

yf225 · 2019-03-11T20:08:17Z

test/cpp/api/sequential.cpp

+    M(const M& other) : torch::nn::Module(other) {
+      // NOTE: The current implementation expects the module to be copied once
+      // when it's passed into `std::make_shared<T>()`.
+      // TODO: Find a way to avoid copying, and then delete the copy constructor.


I filed an issue to document this problem: #17879.

…quential_named_submodules_split

goldsborough

I left some comments

torch/csrc/api/include/torch/nn/modules/any.h

goldsborough · 2019-03-27T16:25:55Z

torch/csrc/api/include/torch/nn/modules/any.h

+// `modules_ordered_dict({{"m1", M(1)}, {"m2", M(2)}})`,
+// if we use the second signature, at the template argument deduction step
+// the compiler is not able to deduce the type of `ModuleType` to the type of
+// `M(1)` or `M(2)`, since the compiler doesn't actually look into the


Did you try whether std::pair<std::string, AnyModule> works?

I tried to change the modules_ordered_dict(...) function to torch::OrderedDict<std::string, AnyModule> modules_ordered_dict(std::initializer_list<std::pair<std::string, AnyModule>> named_modules), however it doesn't seem to work and throws:

../test/cpp/api/sequential.cpp: In member function ‘virtual void SequentialTest_ConstructsFromConcreteType_Test::TestBody()’: ../test/cpp/api/sequential.cpp:72:4: error: could not convert ‘{{"m1", SequentialTest_ConstructsFromConcreteType_Test::TestBody()::M(1)}, {std::__cxx11::basic_string<char>(((const char*)"m2"), std::allocator<char>()), SequentialTest_ConstructsFromConcreteType_Test::TestBody()::M(2)}, {"m3", SequentialTest_ConstructsFromConcreteType_Test::TestBody()::M(3)}}’ from ‘<brace-enclosed initializer list>’ to ‘std::initializer_list<std::pair<std::__cxx11::basic_string<char>, torch::nn::AnyModule> >’

The compiler is not able to match std::initializer_list<std::pair<std::string, AnyModule>> to the nested braced-init list {{"m1", M(1)}, {std::string("m2"), M(2)}, {"m3", M(3)}}. So I think the NamedAnyModule approach here is necessary.

goldsborough · 2019-03-27T16:27:53Z

torch/csrc/api/include/torch/nn/modules/any.h

+inline torch::OrderedDict<std::string, AnyModule> modules_ordered_dict(
+  std::initializer_list<NamedAnyModule> named_modules) {
+  torch::OrderedDict<std::string, AnyModule> dict;
+  for (auto named_module : named_modules) {


Note that you're making copies here?

I think we need to make named_module non-const one way or another, because std::initializer_list<T> only provides access to an array of objects of type const T, not T (according to https://en.cppreference.com/w/cpp/utility/initializer_list), but we need named_module to be of non-const type to be able to do std::move(named_module.module()).

In the latest commit, I changed std::move(named_module.module()) to std::move(const_cast<NamedAnyModule&>(named_module).module())) so that we can avoid doing copies in this line.

torch/csrc/api/include/torch/nn/modules/any.h

goldsborough · 2019-03-27T16:30:41Z

torch/csrc/api/include/torch/nn/modules/sequential.h

+  /// or `push_back("name", module)`, since they should be handled by their respective
+  /// `push_back` functions.
+  template <typename First, typename Second, typename... Rest,
+    typename = torch::disable_if_t<std::is_same<First, std::string>::value ||


Why do we need the const char* guard? I think we're only using std::string in here?

If we don't add the guard for const char* here, a call such as sequential->push_back("shared_m1", M(1)) will be template-matched to this push_back(First&& first, Second&& second, Rest&&... rest) method, which is not what we want (we want it to match to push_back(std::string name, M&& module) instead).

goldsborough · 2019-03-27T16:31:06Z

torch/csrc/api/include/torch/nn/modules/sequential.h

  }

  /// Adds a type-erased `AnyModule` to the `Sequential`.
  void push_back(AnyModule any_module) {


Is this method still used now?

It is still used by this line:

pytorch/torch/csrc/api/include/torch/nn/modules/sequential.h

Line 111 in c3e3c5c

clone->push_back(module.clone(device));

torch/csrc/api/include/torch/nn/modules/sequential.h

…quential_named_submodules_split

goldsborough

👍

facebook-github-bot

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@yf225 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2019-03-29T22:02:42Z

@yf225 merged this pull request in 6ebfbdf.

yf225 requested review from ebetica and goldsborough as code owners February 27, 2019 20:06

facebook-github-bot reviewed Feb 27, 2019

View reviewed changes

yf225 requested review from ezyang and gchanan February 27, 2019 23:56

gchanan requested changes Feb 28, 2019

View reviewed changes

ebetica approved these changes Feb 28, 2019

View reviewed changes

gchanan reviewed Feb 28, 2019

View reviewed changes

ezyang approved these changes Feb 28, 2019

View reviewed changes

goldsborough suggested changes Mar 5, 2019

View reviewed changes

Will Feng added 2 commits March 7, 2019 16:55

allow copy only once

61333d3

simple ordereddict test

c644b8e

yf225 force-pushed the cpp_sequential_named_submodules_split branch from b11b92a to 0038784 Compare March 8, 2019 22:29

Will Feng added 7 commits March 8, 2019 14:36

try to simplify ordereddict creation

ed59d7e

[WIP]

b0c73b1

try helper method approach

4f84c64

[WIP]

97a521c

better comments

0913b12

try something different

848c7fc

Revert "try something different"

be0496d

This reverts commit 848c7fc.

yf225 mentioned this pull request Mar 11, 2019

C++ Models pytorch/vision#728

Merged

better comment

64c4092

yf225 force-pushed the cpp_sequential_named_submodules_split branch from 0038784 to 64c4092 Compare March 11, 2019 19:59

yf225 commented Mar 11, 2019

View reviewed changes

better comments

68300d0

yf225 force-pushed the cpp_sequential_named_submodules_split branch from 1bb5884 to da55865 Compare March 25, 2019 17:05

Will Feng added 3 commits March 25, 2019 13:08

Merge branch 'master' of https://github.com/yf225/pytorch into cpp_se…

74ab871

…quential_named_submodules_split

better formatting for any.h

ee73795

better comment

2924d2d

yf225 force-pushed the cpp_sequential_named_submodules_split branch from 3c9843e to 2924d2d Compare March 25, 2019 20:30

Will Feng added 3 commits March 25, 2019 16:49

nit

02cef0b

move to constructor

175c9d5

fix formatting

77c8662

goldsborough reviewed Mar 27, 2019

View reviewed changes

Will Feng added 4 commits March 27, 2019 14:49

fixes

b8aeb12

Merge branch 'master' of https://github.com/yf225/pytorch into cpp_se…

8716c94

…quential_named_submodules_split

fix

16b0c40

const_cast

4449858

yf225 force-pushed the cpp_sequential_named_submodules_split branch 3 times, most recently from 4c6a5cc to 0839730 Compare March 27, 2019 20:02

named_any

7875d06

yf225 force-pushed the cpp_sequential_named_submodules_split branch from 0839730 to 7875d06 Compare March 27, 2019 20:05

Will Feng added 2 commits March 27, 2019 16:52

add docs for NamedAnyModule

94e2559

try to fix Windows build error

a91034c

goldsborough approved these changes Mar 27, 2019

View reviewed changes

const_cast is not allowed by clang-tidy

e9a35af

facebook-github-bot reviewed Mar 28, 2019

View reviewed changes

fix cpp doc check

0c700cd

facebook-github-bot reviewed Mar 29, 2019

View reviewed changes

facebook-github-bot closed this in 6ebfbdf Mar 29, 2019

facebook-github-bot added the merged label Mar 29, 2019

mullerhai mentioned this pull request Aug 6, 2023

[Pytorch] how to create & use AnyModule object in new version ? bytedeco/javacpp-presets#1399

Closed

[C++ API] Add named submodule support to nn::Sequential #17552

[C++ API] Add named submodule support to nn::Sequential #17552

Uh oh!

Conversation

yf225 commented Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yf225 commented Feb 28, 2019

Uh oh!

goldsborough left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchanan commented Mar 6, 2019

Uh oh!

yf225 commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yf225 commented Mar 11, 2019

Uh oh!

yf225 Mar 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goldsborough left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yf225 Mar 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

goldsborough left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

yf225 commented Feb 27, 2019 •

edited

Loading

goldsborough left a comment •

edited

Loading

yf225 commented Mar 7, 2019 •

edited

Loading

yf225 Mar 11, 2019 •

edited

Loading

yf225 Mar 27, 2019 •

edited

Loading