Tag Anys with their provenance #3786

ilinum · 2017-08-01T04:56:27Z

This PR tags each instance of AnyType with where it comes from.
Each Any can be one (and only one) of the following:

implicit (if this Any type was inferred without a type annotation)
explicit (if this Any comes from an explicit Any in a type annotation)
from_unimported_type (if this Any comes from an unfollowed import)
from_omitted_generics (if this Any comes from an ommitted generic suchs as Any in List)
from_error (if this Any comes from an error)
special_form (if this Any is created for internal mypy purposes)

Plus, this PR contains an addition to AnyExpressionsReport to report most common "tags" of Anys (implicit, explicit, from_unimported_type, etc).

Note that I was not able to identify some of the Anys, specifically, Anys related to generators. I added a todo to each one with the following exact words(# todo: is type of this Any correct?). All of these should probably be resolved until this PR is ready to be merged.

Also, perhaps, it would be better to split the PR into two? (one would tag Anys with their provenance and the other PR would add the new kind of Any Exprs Report)

ddfisher

At a high level, I think this is a good approach. See comments for some initial feedback.

ddfisher · 2017-08-02T20:36:57Z

mypy/report.py

+            line += ('{:>{}} ' * len(widths)).format(*itertools.chain(*zip(values, widths)))
+            f.write(line + '\n')
+
+    def _report_any_exprs(self) -> None:


Nit: this should probably go above the previous function.

ddfisher · 2017-08-02T20:40:43Z

mypy/report.py

+        # On the following line, mypy complains about implicit generic Any and wants us to use
+        # typing.Counter() instead of collections.Counter().
+        # We cannot use typing.Counter() because it doesn't work with python 3.5 and older.
+        total_counter = collections.Counter()  # type: ignore


Instead, give this # type: typing.Counter[str] (or whatever the item type is, exactly).

ddfisher · 2017-08-02T20:42:56Z

mypy/report.py

    def __init__(self, reports: Reports, output_dir: str) -> None:
        super().__init__(reports, output_dir)
        self.counts = {}  # type: Dict[str, Tuple[int, int]]
+        self.any_types_counter = {}  # type: Dict[str, Any]


This should be precisely typed.

ddfisher · 2017-08-02T20:45:56Z

mypy/report.py

+            (TypeOfAny.special_form, "Special Form"),
+            (TypeOfAny.from_another_any, "From Another Any"),
+        ])  # type: collections.OrderedDict[TypeOfAny, str]
+        file_column_name = "Name"


I wonder if the rest of this function could be abstracted out/combined with similar looking code from the function below.

Done! I think code is much cleaner now :)

ddfisher · 2017-08-02T21:57:41Z

mypy/checker.py

                return True
        else:
-            gt = self.named_generic_type('typing.Generator', [AnyType(), AnyType(), AnyType()])
+            # todo: is type of this Any correct?


These Anys don't escape the function -- special form seems right here.

ddfisher · 2017-08-02T21:58:12Z

mypy/checker.py

        try:
-            agt = self.named_generic_type('typing.AsyncGenerator', [AnyType(), AnyType()])
+            # todo: is type of this Any correct?
+            any_type = AnyType(TypeOfAny.special_form)


This Any doesn't escape the function -- special form seems right here.

ddfisher · 2017-08-02T22:00:33Z

mypy/checker.py

            # (However, see https://github.com/python/mypy/issues/1933)
-            return AnyType()
+            # todo: is type of this Any correct?
+            return AnyType(TypeOfAny.special_form)


It's not clear to me that this should even return Any, but I think special form is fine here.

ddfisher · 2017-08-02T22:04:48Z

mypy/subtypes.py

                else:
-                    iter_type = AnyType()
+                    # todo: is type of this Any correct?
+                    iter_type = AnyType(TypeOfAny.from_error)


This should be special form.

I think that ideally this should be assert False, since all these types have one type parameter and types with incorrect type argument numbers (e.g. zero) should be "patched" in the third pass of semantic analysis. There were some efforts recently to weed out all instances with incorrect type argument count (including test fixtures), but maybe some are still there.

@ilevkivskyi, hmm I see. I think it's better to have a separate PR for that one. This doesn't seem to be relevant to this PR and this PR is already huge.

# Conflicts: # mypy/checker.py

ilinum · 2017-08-03T01:13:30Z

All right, all code review feedback should be addressed now!

ddfisher · 2017-08-03T23:22:48Z

mypy/semanal.py

        node = sym.node
        assert isinstance(node, TypeInfo)
-        return Instance(node, [AnyType()] * len(node.defn.type_vars))
+        return Instance(node, [AnyType(TypeOfAny.implicit)] * len(node.defn.type_vars))


I think this should be special form.

ddfisher · 2017-08-03T23:29:31Z

mypy/types.py

        fdef.arg_kinds,
        [None if argument_elide_name(n) else n for n in fdef.arg_names],
-        ret_type or AnyType(),
+        ret_type or AnyType(TypeOfAny.implicit),


I think this (and the other Any in this function) should be special form.

Okay, after making this change, tests were failing, so I made them implicit again.

ddfisher · 2017-08-03T23:31:06Z

mypy/semanal.py

            # TODO: assert len(args) == len(node.defn.type_vars)
            return Instance(node, args)
-        return Instance(node, [AnyType()] * len(node.defn.type_vars))
+        return Instance(node, [AnyType(TypeOfAny.implicit)] * len(node.defn.type_vars))


I thiiiink this should be special form.

ddfisher

Let's do this.

(NewType is not supported in python 3.5.1)

ddfisher · 2017-08-04T22:23:02Z

👍

ddfisher · 2017-08-04T22:48:51Z

🎉

ilinum added 8 commits July 31, 2017 13:00

Tag Anys with their provenance

a6633f5

Fix broken test

694e30d

Report most common sources of Any

1650bc2

Visit inner types in StatisticsVisitor

4ddd163

Mark unchecked functions as special_form

6d062ea

Reword comment

a95d7ea

Do not use typing.Counter() because it doesn't exist in 3.5 and older

988ba83

Use OrderedDict for column names in any-exprs-report

6477941

ddfisher reviewed Aug 2, 2017

View reviewed changes

ilinum added 6 commits August 2, 2017 16:23

Record type of Anys coming from other Anys as type of source Any.

d05b7f0

Use precise types for typing.Counter in type comments.

87399d9

Refactor AnyExpressionsReport to avoid code duplication

8645315

Remove resolved todos about type of Any -- all of them are good now

a088b95

Merge branch 'master' into tag-anys

c01ede9

Merge branch 'master' into tag-anys

cd13986

# Conflicts: # mypy/checker.py

ilinum changed the title ~~[WIP] Tag Anys with their provenance~~ Tag Anys with their provenance Aug 3, 2017

ddfisher reviewed Aug 3, 2017

View reviewed changes

ilinum added 2 commits August 3, 2017 17:57

[StatsVisitor] Do not visit lvalues twice

e0d8411

Report unqualified unresolved symbols as from_error

d30dec4

ilinum mentioned this pull request Aug 4, 2017

Report sources of Any in html-report per-line as hovertext #3798

Merged

ilinum added 5 commits August 4, 2017 00:29

Merge branch 'tag-anys'

a923af1

Fix error after merge

2b3c1a1

Convert several implicit Anys to special form (as per code review)

498fa67

Partially revert previous commit that were breaking tests

81cb76f

Do not run any-expressions-report in untyped defs

be84bee

ilinum added 4 commits August 4, 2017 12:46

Revert accidental typeshed change

389d9eb

Remove redundant variable type annotation

392b773

Merge branch 'master' into tag-anys

f5ac528

fix typeshed commit

78dcf05

ddfisher approved these changes Aug 4, 2017

View reviewed changes

ilinum added 3 commits August 4, 2017 14:39

Makes each TypeOfAny an instance of NewType

6ef0af4

Use actual class instead of NewType

107053c

(NewType is not supported in python 3.5.1)

Use NewType for typechecking, alias for runtime

970613f

Fix lint

5260af8

ilinum merged commit 8984321 into python:master Aug 4, 2017

ilinum deleted the tag-anys branch August 8, 2017 01:55

Uh oh!

Tag Anys with their provenance #3786

Tag Anys with their provenance #3786

Uh oh!

Conversation

ilinum commented Aug 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddfisher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilinum commented Aug 3, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddfisher left a comment

Choose a reason for hiding this comment

Uh oh!

ddfisher commented Aug 4, 2017

Uh oh!

ddfisher commented Aug 4, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ilinum commented Aug 1, 2017 •

edited

Loading