KEMBAR78
Inconsistent `ast.parse` result for `python<3.9` and PEP614 · Issue #96427 · python/cpython · GitHub
Skip to content

Inconsistent ast.parse result for python<3.9 and PEP614 #96427

@sobolevn

Description

@sobolevn

PEP614 introduces relaxed decorator grammar in 3.9: https://peps.python.org/pep-0614/

Right now, ast.parse with feature_version=(3, 8) and bellow behave differently from specified python versions themself.

For example, main branch is able to parse this code:

Python 3.12.0a0 (heads/main-dirty:e860e521ec, Aug 28 2022, 21:36:18) [Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> code = """
... @decs[0].func
... def some(): pass
... """
>>> ast.parse(code)
<ast.Module object at 0x102db8150>
>>> ast.parse(code, feature_version=(3, 8))
<ast.Module object at 0x102d9d320>

While python==3.8.x is not able to:

Python 3.8.9 (default, May  3 2021, 12:15:25) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> code = """
... @decs[0].func
... def some(): pass
... """
>>> ast.parse(code)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sobolev/.pyenv/versions/3.8.9/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 2
    @decs[0].func
         ^
SyntaxError: invalid syntax

When looking at we-love-parsers/pegen - it does have a version check for this: https://github.com/we-like-parsers/pegen/blame/599255a5ba3930e86367cd3fc78fae91ee184dd5/data/python.gram#L660-L662

Pros and Cons

I think that there are several key points to this, pros:

  1. Correct parsing of older versions
  2. Consistency with other parser like we-love-parsers/pegen

However, there are some serious cons to it:

  1. Grammar is made quite significanly more complex
  2. There might be new bugs in new behaviour: locations / performance / etc

I think it is up to @pablogsal and other core devs to decide.

Proposed solution

I've modified an existing we-love-parsers/pegen's solution a bit, here's what it looks like:

decorators[asdl_expr_seq*]: a[asdl_expr_seq*]=decorator+
decorator[expr_ty]:
    | a=('@' f=dec_maybe_call NEWLINE { f }) { a }
    | a=('@' f=named_expression NEWLINE { f }) {
        CHECK_VERSION(expr_ty, 9, "Relaxed decorator syntax is", a) }
dec_maybe_call[expr_ty]:
    | a=dec_primary '(' b=[arguments] ')' {
        _PyAST_Call(a,
                    (b) ? ((expr_ty) b)->v.Call.args : NULL,
                    (b) ? ((expr_ty) b)->v.Call.keywords : NULL,
                    EXTRA) }
    | dec_primary
dec_primary[expr_ty]:
    | a=dec_primary '.' b=NAME { _PyAST_Attribute(a, b->v.Name.id, Load, EXTRA) }
    | a=NAME

I will send a PR with all the changes / tests / auto-generated files, so we can take a closer look and decide whether we want this or not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions