Issue #17514: New ANTLR Grammar for Javadoc Comments #17837

mahfouz72 · 2025-09-27T20:11:45Z

Closes #17514
Related to: New ANTLR Grammar for Javadoc Comments Project - GSoC '25

Key Changes

Implemented a new Javadoc lexer and parser in ANTLR with full coverage of the Javadoc Specification
Added a visitor-based AST construction layer to map ANTLR parse trees into Checkstyle’s internal AST.
Integrated the new parser into Checkstyle’s APIs.
Updated all AbstractJavadocCheck subclasses and Javadoc-related checks to work with the new AST.
Added extensive AST tests and regression tests to ensure correctness and stability.

Impact

Improves maintainability and long-term stability of Javadoc parsing.
Provides a cleaner and more consistent AST structure for checks.
Backward compatibility preserved, existing checks continue to work after migration.
Expected minor differences in parsing behavior compared to the old parser (verified through regressions).

Testing

Unit tests cover the full grammar and AST conversion layer.
Regression testing performed on large projects confirmed correctness and identified only expected differences.

Diff Regression config: https://gist.githubusercontent.com/mahfouz72/5c0b5dccbeb20670915d5f8fe96e9527/raw/4c0abeb793315fd57facd9f2636ba1621e08b1e4/google_checks.xml
Diff Regression projects: https://gist.githubusercontent.com/mahfouz72/e175be6712b9073ad7340fdaa752fc4e/raw/b562141180075cab53efa10a46c8a3d5a1b3050c/projects-to-test-on.properties

mahfouz72 · 2025-09-27T21:12:48Z

Github, generate report for configs in PR description

github-actions · 2025-09-27T21:19:16Z

Report generation failed. Please check the logs for more details.

Link: https://github.com/checkstyle/checkstyle/actions/runs/18065012961

romani · 2025-09-27T21:21:01Z

Github, generate report

github-actions · 2025-09-27T21:25:14Z

Report generation failed on phase "make_report",
step "Generate report".
Link: https://github.com/checkstyle/checkstyle/actions/runs/18065103825

mahfouz72 · 2025-09-27T21:28:14Z

Github, generate report

mahfouz72 · 2025-09-27T21:29:19Z

Github, generate report

github-actions · 2025-09-27T21:32:30Z

Report generation failed on phase "make_report",
step "Generate report".
Link: https://github.com/checkstyle/checkstyle/actions/runs/18065175662

github-actions · 2025-09-27T21:37:07Z

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/85c468c_2025213648/reports/diff/index.html

mahfouz72 · 2025-09-27T22:47:39Z

Github, generate report

mahfouz72 · 2025-09-28T09:05:14Z

Github, generate report

github-actions · 2025-09-28T10:32:15Z

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/85c468c_2025103147/reports/diff/index.html

romani · 2025-10-01T12:56:10Z

Task :postgresql:checkstyleMain
[ant:checkstyle] [ERROR] /home/circleci/project/.ci-temp/pgjdbc/pgjdbc/src/main/java/org/postgresql/jdbc/
ArrayEncoding.java:30:4: <p> tag should not precede HTML block-tag '<ul>', <p> tag should be removed.
 [JavadocParagraph]
[ant:checkstyle] [ERROR] /home/circleci/project/.ci-temp/pgjdbc/pgjdbc/src/main/java/org/postgresql/sspi/
NTDSAPI.java:23:6: <p> tag should not precede HTML block-tag '<pre>', <p> tag should be removed.
 [JavadocParagraph]

please fix false positives

romani

please apply the same logic to most others in whitelist

romani · 2025-10-02T13:05:24Z

config/jsoref-spellchecker/whitelist.words

 Fheader
 fht
 fieldannotations
+Fieldname


✔ ~/github/romani/checkstyle [checkstyle-GSoC25/new-javadoc-grammar L|✔] $ ag "Fieldname" config/jsoref-spellchecker/whitelist.words 432:Fieldname src/test/java/com/puppycrawl/tools/checkstyle/checks/javadoc/AbstractJavadocCheckTest.java 132: 29, "token recognition error at: '-'", "Fieldname"), 419: 29, "token recognition error at: '-'", "Fieldname"), src/test/resources/com/puppycrawl/tools/checkstyle/checks/javadoc/abstractjavadoc/InputAbstractJavadocInvalidLexing.java 10: * @serialField Fieldname-fieldtype-fielddescription 13: // Details: token recognition error at: '-' while parsing Fieldname src/test/resources/com/puppycrawl/tools/checkstyle/checks/javadoc/abstractjavadoc/InputAbstractJavadocNoWsBeforeDescriptionInJavadocTags.java 27: * @serialField Fieldname-fieldtype-fielddescription 30: // Details: token recognition error at: '-' while parsing Fieldname 35: * @serialField Fieldname fieldtype-fielddescription 40: * @serialField Fieldname -fieldtype -fielddescription

I think we can easily avoid this new word in while list.
Fieldname --> fieldName

romani · 2025-10-02T13:09:04Z

config/jsoref-spellchecker/whitelist.words

 htdocs
 html
 HTMLJS
+htmltags


✔ ~/github/romani/checkstyle [checkstyle-GSoC25/new-javadoc-grammar L|✔] $ ag -s "htmltags" config/jsoref-spellchecker/whitelist.words 560:htmltags src/test/java/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocCommentsAstRegressionTest.java 40: return getPath("htmltags" + File.separator + filename);

can we use htmlTags ? we already have such folder

This is a package name, we use all lowercase letters for package names across the codebase

romani · 2025-10-02T13:10:30Z

config/jsoref-spellchecker/whitelist.words

 Popup
 Postgresql
 powershell
+prc


✔ ~/github/romani/checkstyle [checkstyle-GSoC25/new-javadoc-grammar L|✔] $ ag -s "prc" config/jsoref-spellchecker/whitelist.words 1099:prc src/main/java/com/puppycrawl/tools/checkstyle/JavadocCommentsAstVisitor.java 153: if (tag instanceof ParserRuleContext prc) { 154: final Token tagName = (Token) prc.getChild(1).getPayload(); 283: if (tagContent instanceof ParserRuleContext prc) { 284: final Token tagName = (Token) prc.getChild(0).getPayload();

ruleContext or context

romani · 2025-10-03T00:56:41Z

GitHub, generate website

github-actions · 2025-10-03T01:05:52Z

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/index.html

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/annotation/missingdeprecated.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/atclauseorder.html#Examples

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/javadocblocktaglocation.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/javadocleadingasteriskalign.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/javadocmissingleadingasterisk.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/javadocmissingwhitespaceafterasterisk.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/javadocparagraph.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/javadoctagcontinuationindentation.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/nonemptyatclausedescription.html#Properties

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/requireemptylinebeforeblocktaggroup.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/singlelinejavadoc.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/javadoc/summaryjavadoc.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/9f50866_20251003010133/checks/misc/orderedproperties.html#Description

romani · 2025-10-04T15:34:28Z

GitHub, generate website

romani · 2025-10-04T15:34:54Z

Github, generate report

github-actions · 2025-10-04T15:41:51Z

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/index.html

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/annotation/missingdeprecated.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/atclauseorder.html#Examples

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/javadocblocktaglocation.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/javadocleadingasteriskalign.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/javadocmissingleadingasterisk.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/javadocmissingwhitespaceafterasterisk.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/javadocparagraph.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/javadoctagcontinuationindentation.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/nonemptyatclausedescription.html#Properties

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/requireemptylinebeforeblocktaggroup.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/singlelinejavadoc.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/javadoc/summaryjavadoc.html#Violation_Messages

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_20251004154112/checks/misc/orderedproperties.html#Description

github-actions · 2025-10-04T17:07:39Z

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8dad8c7_2025170701/reports/diff/index.html

romani

Thanks a lot for your hard work!!!!

I can not review all code changes, but it was reviewed by others, so let's move forward

This was referenced Sep 28, 2025

Issue #17223: implementation of Indentation TextBlockGoogleStyleFormattingCheck #17395

Draft

Issue #17223: New Check TextBlockGoogleStyleFormattingCheck #17329

Open

mahfouz72 force-pushed the new-javadoc-grammar branch 3 times, most recently from 85507a4 to b7e082b Compare October 1, 2025 09:52

mahfouz72 force-pushed the new-javadoc-grammar branch 6 times, most recently from 04a3c99 to c9f2947 Compare October 1, 2025 23:16

romani requested changes Oct 2, 2025

View reviewed changes

mahfouz72 force-pushed the new-javadoc-grammar branch 4 times, most recently from 4c0e4ff to e65052c Compare October 3, 2025 08:24

mahfouz72 force-pushed the new-javadoc-grammar branch 3 times, most recently from 6f3e470 to 77c774a Compare October 3, 2025 21:16

Issue checkstyle#17514: New ANTLR Grammar for Javadoc Comments

8dad8c7

mahfouz72 force-pushed the new-javadoc-grammar branch from 77c774a to 8dad8c7 Compare October 4, 2025 15:25

romani approved these changes Oct 4, 2025

View reviewed changes

romani merged commit c8c13f4 into checkstyle:master Oct 4, 2025
120 checks passed

mahfouz72 mentioned this pull request Oct 5, 2025

doc: update writing javadoc checks page in the documentation #17869

Merged

Uh oh!

Issue #17514: New ANTLR Grammar for Javadoc Comments #17837

Issue #17514: New ANTLR Grammar for Javadoc Comments #17837

Uh oh!

Conversation

mahfouz72 commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Changes

Impact

Testing

Uh oh!

mahfouz72 commented Sep 27, 2025

Uh oh!

github-actions bot commented Sep 27, 2025

Uh oh!

romani commented Sep 27, 2025

Uh oh!

github-actions bot commented Sep 27, 2025

Uh oh!

mahfouz72 commented Sep 27, 2025

Uh oh!

mahfouz72 commented Sep 27, 2025

Uh oh!

github-actions bot commented Sep 27, 2025

Uh oh!

github-actions bot commented Sep 27, 2025

Uh oh!

mahfouz72 commented Sep 27, 2025

Uh oh!

mahfouz72 commented Sep 28, 2025

Uh oh!

github-actions bot commented Sep 28, 2025

Uh oh!

romani commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romani left a comment

Choose a reason for hiding this comment

Uh oh!

romani Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

romani Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

mahfouz72 Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

romani Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

romani commented Oct 3, 2025

Uh oh!

github-actions bot commented Oct 3, 2025

Uh oh!

romani commented Oct 4, 2025

Uh oh!

romani commented Oct 4, 2025

Uh oh!

github-actions bot commented Oct 4, 2025

Uh oh!

github-actions bot commented Oct 4, 2025

Uh oh!

romani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mahfouz72 commented Sep 27, 2025 •

edited

Loading

romani commented Oct 1, 2025 •

edited

Loading