texlive[74771] Master/texmf-dist: expltools (28mar25)
commits+karl at tug.org
commits+karl at tug.org
Fri Mar 28 22:25:19 CET 2025
Revision: 74771
https://tug.org/svn/texlive?view=revision&revision=74771
Author: karl
Date: 2025-03-28 22:25:19 +0100 (Fri, 28 Mar 2025)
Log Message:
-----------
expltools (28mar25)
Modified Paths:
--------------
trunk/Master/texmf-dist/doc/support/expltools/CHANGES.md
trunk/Master/texmf-dist/doc/support/expltools/README.md
trunk/Master/texmf-dist/doc/support/expltools/project-proposal.pdf
trunk/Master/texmf-dist/doc/support/expltools/warnings-and-errors-03-syntactic-analysis.md
trunk/Master/texmf-dist/doc/support/expltools/warnings-and-errors.pdf
trunk/Master/texmf-dist/scripts/expltools/explcheck-cli.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-config.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-config.toml
trunk/Master/texmf-dist/scripts/expltools/explcheck-format.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-issues.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-lexical-analysis.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-obsolete.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-parsers.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-preprocessing.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-ranges.lua
Added Paths:
-----------
trunk/Master/texmf-dist/doc/support/expltools/e300-01.tex
trunk/Master/texmf-dist/doc/support/expltools/e300-02.tex
trunk/Master/texmf-dist/doc/support/expltools/e300-03.tex
trunk/Master/texmf-dist/doc/support/expltools/e301.tex
trunk/Master/texmf-dist/doc/support/expltools/w302.tex
trunk/Master/texmf-dist/doc/support/expltools/w303.tex
trunk/Master/texmf-dist/scripts/expltools/explcheck-evaluation.lua
trunk/Master/texmf-dist/scripts/expltools/explcheck-syntactic-analysis.lua
Modified: trunk/Master/texmf-dist/doc/support/expltools/CHANGES.md
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/CHANGES.md 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/doc/support/expltools/CHANGES.md 2025-03-28 21:25:19 UTC (rev 74771)
@@ -1,5 +1,108 @@
# Changes
+## expltools 2025-03-27
+
+### explcheck v0.8.0
+
+#### Development
+
+- Add syntactic analysis. (#66)
+
+- Add Lua option `verbose` and a command-line option `--verbose` that
+ prints extra information in human-readable output. (#66)
+
+ For example, here is the output of processing the files `markdown.tex` and
+ `markdownthemewitiko_markdown_defaults.sty` of the Markdown package for TeX
+ with TeX Live 2024:
+
+ ```
+ $ explcheck --verbose `kpsewhich markdown.tex markdownthemewitiko_markdown_defaults.sty`
+ ```
+ ```
+ Checking 2 files
+
+ Checking /usr/local/texlive/2024/texmf-dist/tex/generic/markdown/markdown.tex OK
+
+ File size: 103,972 bytes
+
+ Preprocessing results:
+ - Doesn't seem like a LaTeX style file
+ - Six expl3 parts spanning 97,657 bytes (94% of file):
+ 1. Between 48:14 and 620:11
+ 2. Between 637:14 and 788:4
+ 3. Between 791:14 and 2104:8
+ 4. Between 2108:14 and 3398:4
+ 5. Between 3413:14 and 4210:4
+ 6. Between 4287:14 and 4444:4
+
+ Lexical analysis results:
+ - 19,344 TeX tokens in expl3 parts
+
+ Syntactic analysis results:
+ - 645 top-level expl3 calls spanning all tokens
+
+ Checking /.../tex/latex/markdown/markdownthemewitiko_markdown_defaults.sty OK
+
+ File size: 34,894 bytes
+
+ Preprocessing results:
+ - Seems like a LaTeX style file
+ - Seven expl3 parts spanning 18,515 bytes (53% of file):
+ 1. Between 47:14 and 349:4
+ 2. Between 382:14 and 431:2
+ 3. Between 446:14 and 512:4
+ 4. Between 523:14 and 564:2
+ 5. Between 865:14 and 931:2
+ 6. Between 969:14 and 1003:2
+ 7. Between 1072:14 and 1328:2
+
+ Lexical analysis results:
+ - 3,848 TeX tokens in expl3 parts
+
+ Syntactic analysis results:
+ - 69 top-level expl3 calls spanning 2,082 tokens (54% of tokens, ~29% of file)
+
+ Total: 0 errors, 0 warnings in 2 files
+
+ Aggregate statistics:
+ - 138,866 total bytes
+ - 116,172 expl3 bytes (84% of files) containing 23,192 TeX tokens
+ - 714 top-level expl3 calls spanning 21,426 tokens (92% of tokens, ~77% of files)
+ ```
+
+- Add Lua option `terminal_width` that determines the layout of the
+ human-readable command-line output. (#66)
+
+- Stabilize the Lua API of processing steps. (#64)
+
+ All processing steps are now functions that accept the following arguments:
+ 1. The filename of a processed file
+ 2. The content of the processed file
+ 3. A registry of issues with the processed file (write-only)
+ 4. Intermediate analysis results (read-write)
+ 5. Options (read-only, optional)
+
+#### Fixes
+
+- During preprocessing, only consider standard delimiters of expl3 parts that
+ are either not indented or not in braces. (discussed in #17, fixed in #66)
+
+- During preprocessing, support `\endinput`, `\tex_endinput:D`, and
+ `\file_input_stop:` as standard delimiters of expl3 parts. (#66)
+
+- During preprocessing, do not produce warning W101 (Unexpected delimiters) for
+ a `\ProvidesExpl*` after `\ExplSyntaxOn`. (#66)
+
+- Prevent newlines from being recognized as catcode 15 (invalid) with Lua 5.2
+ due to unreliable order of table keys. (#66)
+
+#### Continuous integration
+
+- Add regression tests for TeX Live 2024. (#66)
+
+- Configure Dependabot version updates for GitHub Actions.
+ (contributed by @koppor in #70)
+
## expltools 2025-02-25
### explcheck v0.7.1
Modified: trunk/Master/texmf-dist/doc/support/expltools/README.md
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/README.md 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/doc/support/expltools/README.md 2025-03-28 21:25:19 UTC (rev 74771)
@@ -45,10 +45,6 @@
For example, here is Lua code that applies the preprocessing step to the code from a file named `code.tex`:
``` lua
-local new_issues = require("explcheck-issues")
-local preprocessing = require("explcheck-preprocessing")
-local lexical_analysis = require("explcheck-lexical-analysis")
-
-- LuaTeX users must initialize Kpathsea Lua module searchers first.
local using_luatex, kpse = pcall(require, "kpse")
if using_luatex then
@@ -55,16 +51,25 @@
kpse.set_program_name("texlua", "explcheck")
end
+-- Import explcheck.
+local new_issues = require("explcheck-issues")
+
+local preprocessing = require("explcheck-preprocessing")
+local lexical_analysis = require("explcheck-lexical-analysis")
+local syntactic_analysis = require("explcheck-syntactic-analysis")
+
-- Process file "code.tex" and print warnings and errors.
local filename = "code.tex"
local issues = new_issues()
+local results = {}
local file = assert(io.open(filename, "r"))
local content = assert(file:read("*a"))
assert(file:close())
-local _, expl_ranges = preprocessing(issues, filename, content)
-lexical_analysis(issues, filename, content, expl_ranges)
+preprocessing.process(filename, content, issues, results)
+lexical_analysis.process(filename, content, issues, results)
+syntactic_analysis.process(filename, content, issues, results)
print(
"There were " .. #issues.warnings .. " warnings, "
@@ -120,8 +125,9 @@
issues:ignore("w100")
issues:ignore("S204")
-local _, expl_ranges = preprocessing(issues, content, options)
-lexical_analysis(issues, content, expl_ranges, options)
+preprocessing.process(filename, content, issues, results, options)
+lexical_analysis.process(filename, content, issues, results, options)
+syntactic_analysis.process(filename, content, issues, results, options)
```
Command-line options, configuration files, and Lua code allow you to ignore certain warnings and errors everywhere.
Added: trunk/Master/texmf-dist/doc/support/expltools/e300-01.tex
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/e300-01.tex (rev 0)
+++ trunk/Master/texmf-dist/doc/support/expltools/e300-01.tex 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,12 @@
+\cs_new:Nn
+ \example_foo:n
+ { foo~#1 }
+\cs_new:Nn
+ \example_bar:
+ { \example_foo:n }
+\cs_new:Nn
+ \example_baz:
+ {
+ \example_bar:
+ { bar }
+ }
Property changes on: trunk/Master/texmf-dist/doc/support/expltools/e300-01.tex
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: trunk/Master/texmf-dist/doc/support/expltools/e300-02.tex
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/e300-02.tex (rev 0)
+++ trunk/Master/texmf-dist/doc/support/expltools/e300-02.tex 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,3 @@
+\cs_new:Nn
+ { unexpected } % error on this line
+ \l_tmpa_tl
Property changes on: trunk/Master/texmf-dist/doc/support/expltools/e300-02.tex
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: trunk/Master/texmf-dist/doc/support/expltools/e300-03.tex
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/e300-03.tex (rev 0)
+++ trunk/Master/texmf-dist/doc/support/expltools/e300-03.tex 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,5 @@
+{
+ \cs_new:Npn
+ \example_foo:w
+ #1 } % error on this line
+ { foo~#1 }
Property changes on: trunk/Master/texmf-dist/doc/support/expltools/e300-03.tex
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: trunk/Master/texmf-dist/doc/support/expltools/e301.tex
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/e301.tex (rev 0)
+++ trunk/Master/texmf-dist/doc/support/expltools/e301.tex 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,2 @@
+\cs_new:Nn % error on this line
+ \example_foo:n
Property changes on: trunk/Master/texmf-dist/doc/support/expltools/e301.tex
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Modified: trunk/Master/texmf-dist/doc/support/expltools/project-proposal.pdf
===================================================================
(Binary files differ)
Added: trunk/Master/texmf-dist/doc/support/expltools/w302.tex
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/w302.tex (rev 0)
+++ trunk/Master/texmf-dist/doc/support/expltools/w302.tex 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,3 @@
+\tl_set:No
+ \l_tmpa_tl
+ \l_tmpb_tl % error on this line
Property changes on: trunk/Master/texmf-dist/doc/support/expltools/w302.tex
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: trunk/Master/texmf-dist/doc/support/expltools/w303.tex
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/w303.tex (rev 0)
+++ trunk/Master/texmf-dist/doc/support/expltools/w303.tex 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,3 @@
+\cs_new:Nn
+ { \example_foo:n } % warning on this line
+ { bar }
Property changes on: trunk/Master/texmf-dist/doc/support/expltools/w303.tex
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Modified: trunk/Master/texmf-dist/doc/support/expltools/warnings-and-errors-03-syntactic-analysis.md
===================================================================
--- trunk/Master/texmf-dist/doc/support/expltools/warnings-and-errors-03-syntactic-analysis.md 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/doc/support/expltools/warnings-and-errors-03-syntactic-analysis.md 2025-03-28 21:25:19 UTC (rev 74771)
@@ -1,26 +1,31 @@
# Syntactic analysis
In the syntactic analysis step, the expl3 analysis tool converts the list of `\TeX`{=tex} tokens into a tree of function calls.
-## Unexpected function call argument {.e}
-A function is called with an unexpected argument. Partial applications are detected by analysing closing braces (`}`) and do not produce an error.
+## Unexpected function call argument {.e label=e300}
+A function is called with an unexpected argument.
-``` tex
-\cs_new:Nn
- \example_foo:n
- { foo~#1 }
-\cs_new:Nn
- \example_bar:
- { \example_foo:n }
-\cs_new:Nn
- \example_baz:
- {
- \example_bar:
- { bar }
- }
-```
+ /e300-02.tex
+ /e300-03.tex
-``` tex
-\cs_new:Nn
- { unexpected } % error on this line
- \l_tmpa_tl % error on this line
-```
+Partial applications are detected by analysing closing braces (`}`) and do not produce an error:
+
+ /e300-01.tex
+
+## End of expl3 part within function call {.e label=e301}
+A function call is cut off by the end of a file or an expl3 part of a file:
+
+ /e301.tex
+
+## Unbraced n-type function call argument {.w label=w302}
+An n-type function call argument is unbraced:
+
+ /w302.tex
+
+Depending on the specific function, this may or may not be an error.
+
+## Braced N-type function call argument {.w label=w303}
+An N-type function call argument is braced:
+
+ /w302.tex
+
+Depending on the specific function, this may or may not be an error.
Modified: trunk/Master/texmf-dist/doc/support/expltools/warnings-and-errors.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-cli.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-cli.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-cli.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -1,13 +1,17 @@
-- A command-line interface for the static analyzer explcheck.
+local evaluation = require("explcheck-evaluation")
local format = require("explcheck-format")
local get_option = require("explcheck-config")
local new_issues = require("explcheck-issues")
local utils = require("explcheck-utils")
+local new_file_results = evaluation.new_file_results
+local new_aggregate_results = evaluation.new_aggregate_results
+
local preprocessing = require("explcheck-preprocessing")
local lexical_analysis = require("explcheck-lexical-analysis")
--- local syntactic_analysis = require("explcheck-syntactic-analysis")
+local syntactic_analysis = require("explcheck-syntactic-analysis")
-- local semantic_analysis = require("explcheck-semantic-analysis")
-- local pseudo_flow_analysis = require("explcheck-pseudo-flow-analysis")
@@ -66,13 +70,11 @@
-- Process all input files.
local function main(pathnames, options)
- local num_warnings = 0
- local num_errors = 0
-
if not options.porcelain then
print("Checking " .. #pathnames .. " " .. format.pluralize("file", #pathnames))
end
+ local aggregate_evaluation_results = new_aggregate_results()
for pathname_number, pathname in ipairs(pathnames) do
local is_ok, error_message = xpcall(function()
@@ -87,26 +89,22 @@
local content = assert(file:read("*a"))
assert(file:close())
- -- Run all processing steps.
- local line_starting_byte_numbers, expl_ranges, seems_like_latex_style_file, tokens -- luacheck: ignore tokens
-
- line_starting_byte_numbers, expl_ranges, seems_like_latex_style_file = preprocessing(issues, pathname, content, options)
-
- if #issues.errors > 0 then
- goto continue
+ -- Run all steps.
+ local analysis_results = {}
+ for _, step in ipairs({preprocessing, lexical_analysis, syntactic_analysis}) do
+ step.process(pathname, content, issues, analysis_results, options)
+ -- If a processing step ended with error, skip all following steps.
+ if #issues.errors > 0 then
+ goto skip_remaining_steps
+ end
end
- tokens = lexical_analysis(issues, pathname, content, expl_ranges, seems_like_latex_style_file, options)
-
- -- syntactic_analysis(issues)
- -- semantic_analysis(issues)
- -- pseudo_flow_analysis(issues)
-
-- Print warnings and errors.
- ::continue::
- num_warnings = num_warnings + #issues.warnings
- num_errors = num_errors + #issues.errors
- format.print_results(pathname, issues, line_starting_byte_numbers, pathname_number == #pathnames, options)
+ ::skip_remaining_steps::
+ local file_evaluation_results = new_file_results(content, analysis_results, issues)
+ aggregate_evaluation_results:add(file_evaluation_results)
+ local is_last_file = pathname_number == #pathnames
+ format.print_results(pathname, issues, analysis_results, options, file_evaluation_results, is_last_file)
end, debug.traceback)
if not is_ok then
error("Failed to process " .. pathname .. ": " .. tostring(error_message), 0)
@@ -113,11 +111,10 @@
end
end
- -- Print a summary.
- if not options.porcelain then
- format.print_summary(#pathnames, num_warnings, num_errors, options.porcelain)
- end
+ format.print_summary(options, aggregate_evaluation_results)
+ local num_errors = aggregate_evaluation_results.num_errors
+ local num_warnings = aggregate_evaluation_results.num_warnings
if(num_errors > 0) then
return 1
elseif(get_option("warnings_are_errors", options) and num_warnings > 0) then
@@ -158,6 +155,7 @@
.. "\t--max-line-length=N The maximum line length before the warning S103 (Line too long) is produced.\n"
.. "\t The default maximum line length is N=" .. max_line_length .. " characters.\n\n"
.. "\t--porcelain, -p Produce machine-readable output. See also --error-format.\n\n"
+ .. "\t--verbose Print additional information in non-machine-readable output. See also --porcelain.\n\n"
.. "\t--warnings-are-errors Produce a non-zero exit code if any warnings are produced by the analysis.\n"
)
print("The options are provisional and may be changed or removed before version 1.0.0.")
@@ -164,7 +162,7 @@
end
local function print_version()
- print("explcheck (expltools 2025-02-25) v0.7.1")
+ print("explcheck (expltools 2025-03-27) v0.8.0")
print("Copyright (c) 2024-2025 Vít Starý Novotný")
print("Licenses: LPPL 1.3 or later, GNU GPL v2 or later")
end
@@ -215,6 +213,8 @@
options.max_line_length = tonumber(argument:sub(19))
elseif argument == "--porcelain" or argument == "-p" then
options.porcelain = true
+ elseif argument == "--verbose" then
+ options.verbose = true
elseif argument == "--warnings-are-errors" then
options.warnings_are_errors = true
elseif argument:sub(1, 2) == "--" then
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-config.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-config.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-config.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -32,13 +32,13 @@
if pathname ~= nil then
-- If a pathname is provided and the current configuration specifies the option for this filename, use it.
local filename = utils.get_basename(pathname)
- if config["filename"] and config["filename"][filename] ~= nil and config["filename"][filename][key] ~= nil then
- return config["filename"][filename][key]
+ if config.filename and config.filename[filename] ~= nil and config.filename[filename][key] ~= nil then
+ return config.filename[filename][key]
end
-- If a pathname is provided and the current configuration specifies the option for this package, use it.
local package = utils.get_basename(utils.get_parent(pathname))
- if config["package"] and config["package"][package] ~= nil and config["package"][package][key] ~= nil then
- return config["package"][package][key]
+ if config.package and config.package[package] ~= nil and config.package[package][key] ~= nil then
+ return config.package[package][key]
end
end
-- If the current configuration specifies the option in the defaults, use it.
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-config.toml
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-config.toml 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-config.toml 2025-03-28 21:25:19 UTC (rev 74771)
@@ -5,11 +5,13 @@
make_at_letter = "auto"
max_line_length = 80
porcelain = false
+terminal_width = 80
+verbose = false
warnings_are_errors = false
[package.l3kernel]
expl3_detection_strategy = "always"
-ignored_issues = ["e208", "e209"]
+ignored_issues = ["e208", "e209", "e300", "e301", "w302"]
max_line_length = 140
[filename."tex4ht.sty"]
@@ -54,7 +56,6 @@
max_line_length = 100
[filename."chemmacros.sty"]
-expl3_detection_strategy = "always"
max_line_length = 100
[filename."interlinear.sty"]
@@ -63,6 +64,7 @@
[filename."skrapport.cls"]
expl3_detection_strategy = "always"
max_line_length = 150
+ignored_issues = ["e300"]
[filename."namedef.sty"]
expl3_detection_strategy = "always"
@@ -75,7 +77,7 @@
max_line_length = 100
[filename."knowledge.sty"]
-ignored_issues = ["e201"]
+ignored_issues = ["e201", "e300", "e301", "w302"]
[filename."grading-scheme.sty"]
ignored_issues = ["e201"]
@@ -133,6 +135,7 @@
[filename."interchar.sty"]
expl3_detection_strategy = "always"
+ignored_issues = ["e300"]
max_line_length = 90
[filename."bangla.sty"]
@@ -141,8 +144,8 @@
[filename."simurgh-empheq.sty"]
expl3_detection_strategy = "precision"
-[filename."exam-zh.cls"]
-ignored_issues = ["e209"]
+[package.exam-zh]
+ignored_issues = ["e209", "e301"]
max_line_length = 150
[filename."lwarp-statistics.sty"]
@@ -157,7 +160,7 @@
[package.xecjk]
expl3_detection_strategy = "always"
-ignored_issues = ["e209"]
+ignored_issues = ["e209", "e300"]
max_line_length = 100
[filename."luaprogtable.sty"]
@@ -240,3 +243,25 @@
[filename."ufrgscca.cls"]
expl3_detection_strategy = "always"
+
+[filename."keythms-amsart-support.tex"]
+expl3_detection_strategy = "always"
+ignored_issues = ["e301", "w302"]
+
+[package.leadsheets]
+ignored_issues = ["e301"]
+
+[package.mathcommand]
+ignored_issues = ["e301"]
+
+[package.media4svg]
+ignored_issues = ["e301"]
+
+[package.temporal-logic]
+ignored_issues = ["e300"]
+
+[package.xsim]
+ignored_issues = ["e301"]
+
+[package.zitie]
+ignored_issues = ["e300"]
Added: trunk/Master/texmf-dist/scripts/expltools/explcheck-evaluation.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-evaluation.lua (rev 0)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-evaluation.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,107 @@
+-- Evaluation the analysis results, both for individual files and in aggregate.
+
+local call_types = require("explcheck-syntactic-analysis").call_types
+
+local CALL = call_types.CALL
+
+local FileEvaluationResults = {}
+local AggregateEvaluationResults = {}
+
+-- Create a new evaluation results for the analysis results of an individual file.
+function FileEvaluationResults.new(cls, content, analysis_results, issues)
+ -- Instantiate the class.
+ local self = {}
+ setmetatable(self, cls)
+ cls.__index = cls
+ -- Evaluate the pre-analysis information.
+ local num_total_bytes = #content
+ -- Evaluate the issues.
+ local num_warnings = #issues.warnings
+ local num_errors = #issues.errors
+ -- Evaluate the results of the preprocessing.
+ local num_expl_bytes
+ if analysis_results.expl_ranges ~= nil then
+ num_expl_bytes = 0
+ for _, range in ipairs(analysis_results.expl_ranges) do
+ num_expl_bytes = num_expl_bytes + #range
+ end
+ end
+ -- Evaluate the results of the lexical analysis.
+ local num_tokens
+ if analysis_results.tokens ~= nil then
+ num_tokens = 0
+ for _, part_tokens in ipairs(analysis_results.tokens) do
+ num_tokens = num_tokens + #part_tokens
+ end
+ end
+ -- Evaluate the results of the syntactic analysis.
+ local num_calls, num_call_tokens
+ if analysis_results.calls ~= nil then
+ num_calls, num_call_tokens = 0, 0
+ for _, part_calls in ipairs(analysis_results.calls) do
+ for _, call in ipairs(part_calls) do
+ local call_type, call_tokens, _, _ = table.unpack(call)
+ if call_type == CALL then
+ num_calls = num_calls + 1
+ num_call_tokens = num_call_tokens + #call_tokens
+ end
+ end
+ end
+ end
+ -- Initialize the class.
+ self.num_total_bytes = num_total_bytes
+ self.num_warnings = num_warnings
+ self.num_errors = num_errors
+ self.num_expl_bytes = num_expl_bytes
+ self.num_tokens = num_tokens
+ self.num_calls = num_calls
+ self.num_call_tokens = num_call_tokens
+ return self
+end
+
+-- Create an aggregate evaluation results.
+function AggregateEvaluationResults.new(cls)
+ -- Instantiate the class.
+ local self = {}
+ setmetatable(self, cls)
+ cls.__index = cls
+ -- Initialize the class.
+ self.num_files = 0
+ self.num_total_bytes = 0
+ self.num_warnings = 0
+ self.num_errors = 0
+ self.num_expl_bytes = 0
+ self.num_tokens = 0
+ self.num_calls = 0
+ self.num_call_tokens = 0
+ return self
+end
+
+-- Add evaluation results of an individual file to the aggregate.
+function AggregateEvaluationResults:add(evaluation_results)
+ self.num_files = self.num_files + 1
+ self.num_total_bytes = self.num_total_bytes + evaluation_results.num_total_bytes
+ self.num_warnings = self.num_warnings + evaluation_results.num_warnings
+ self.num_errors = self.num_errors + evaluation_results.num_errors
+ if evaluation_results.num_expl_bytes ~= nil then
+ self.num_expl_bytes = self.num_expl_bytes + evaluation_results.num_expl_bytes
+ end
+ if evaluation_results.num_tokens ~= nil then
+ self.num_tokens = self.num_tokens + evaluation_results.num_tokens
+ end
+ if evaluation_results.num_calls ~= nil then
+ self.num_calls = self.num_calls + evaluation_results.num_calls
+ end
+ if evaluation_results.num_call_tokens ~= nil then
+ self.num_call_tokens = self.num_call_tokens + evaluation_results.num_call_tokens
+ end
+end
+
+return {
+ new_file_results = function(...)
+ return FileEvaluationResults:new(...)
+ end,
+ new_aggregate_results = function(...)
+ return AggregateEvaluationResults:new(...)
+ end
+}
Property changes on: trunk/Master/texmf-dist/scripts/expltools/explcheck-evaluation.lua
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-format.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-format.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-format.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -3,6 +3,18 @@
local get_option = require("explcheck-config")
local utils = require("explcheck-utils")
+local color_codes = {
+ BOLD = 1,
+ RED = 31,
+ GREEN = 32,
+ YELLOW = 33,
+}
+
+local BOLD = color_codes.BOLD
+local RED = color_codes.RED
+local GREEN = color_codes.GREEN
+local YELLOW = color_codes.YELLOW
+
-- Transform a singular into plural if the count is zero or greater than two.
local function pluralize(singular, count)
if count == 1 then
@@ -12,6 +24,45 @@
end
end
+-- Upper-case the initial letter of a word.
+local function titlecase(word)
+ assert(#word > 0)
+ return string.format("%s%s", word:sub(1, 1):upper(), word:sub(2))
+end
+
+-- Convert a number to a string with thousand separators.
+local function separate_thousands(number)
+ local initial_digit, following_digits = string.match(tostring(number), '^(%d)(%d*)$')
+ return initial_digit .. following_digits:reverse():gsub('(%d%d%d)', '%1,'):reverse()
+end
+
+-- Transform short numbers to words and make long numbers more readable using thousand separators.
+local function humanize(number)
+ if number == 1 then
+ return "one"
+ elseif number == 2 then
+ return "two"
+ elseif number == 3 then
+ return "three"
+ elseif number == 4 then
+ return "four"
+ elseif number == 5 then
+ return "five"
+ elseif number == 6 then
+ return "six"
+ elseif number == 7 then
+ return "seven"
+ elseif number == 8 then
+ return "eight"
+ elseif number == 9 then
+ return "nine"
+ elseif number == 10 then
+ return "ten"
+ else
+ return separate_thousands(number)
+ end
+end
+
-- Shorten a pathname, so that it does not exceed maximum length.
local function format_pathname(pathname, max_length)
-- First, replace path segments with `/.../`, keeping other segments.
@@ -69,52 +120,142 @@
return text:gsub("\27%[[0-9]+m", "")
end
+-- Format a ratio as a percentage.
+local function format_ratio(numerator, denominator)
+ assert(numerator <= denominator)
+ if numerator == denominator then
+ return "100%"
+ else
+ assert(denominator > 0)
+ return string.format("%.0f%%", 100.0 * numerator / denominator)
+ end
+end
+
+-- Print the summary results of analyzing multiple files.
+local function print_summary(options, evaluation_results)
+ local porcelain, verbose = get_option('porcelain', options), get_option('verbose', options)
+
+ if porcelain then
+ return
+ end
+
+ local num_files = evaluation_results.num_files
+ local num_warnings = evaluation_results.num_warnings
+ local num_errors = evaluation_results.num_errors
+
+ io.write(string.format("\n\n%s ", colorize("Total:", BOLD)))
+
+ local errors_message = tostring(num_errors) .. " " .. pluralize("error", num_errors)
+ errors_message = colorize(errors_message, BOLD, (num_errors > 0 and RED) or GREEN)
+ io.write(errors_message .. ", ")
+
+ local warnings_message = tostring(num_warnings) .. " " .. pluralize("warning", num_warnings)
+ warnings_message = colorize(warnings_message, BOLD, (num_warnings > 0 and YELLOW) or GREEN)
+ io.write(warnings_message .. " in ")
+
+ io.write(tostring(num_files) .. " " .. pluralize("file", num_files))
+
+ -- Display additional information.
+ if verbose then
+ print()
+ io.write(string.format("\n%s", colorize("Aggregate statistics:", BOLD)))
+ -- Display pre-evaluation information.
+ local num_total_bytes = evaluation_results.num_total_bytes
+ io.write(string.format("\n- %s total %s", titlecase(humanize(num_total_bytes)), pluralize("byte", num_total_bytes)))
+ -- Evaluate the evalution results of the preprocessing.
+ local num_expl_bytes = evaluation_results.num_expl_bytes
+ if num_expl_bytes == 0 then
+ goto skip_remaining_additional_information
+ end
+ io.write(string.format("\n- %s expl3 %s ", titlecase(humanize(num_expl_bytes)), pluralize("byte", num_expl_bytes)))
+ io.write(string.format("(%s of %s)", format_ratio(num_expl_bytes, num_total_bytes), pluralize("file", num_files)))
+ -- Evaluate the evalution results of the lexical analysis.
+ local num_tokens = evaluation_results.num_tokens
+ if num_tokens == 0 then
+ goto skip_remaining_additional_information
+ end
+ io.write(string.format(" containing %s %s", humanize(num_tokens), pluralize("TeX token", num_tokens)))
+ -- Evaluate the evalution results of the syntactic analysis.
+ local num_calls = evaluation_results.num_calls
+ local num_call_tokens = evaluation_results.num_call_tokens
+ if num_calls == 0 then
+ goto skip_remaining_additional_information
+ end
+ assert(num_call_tokens > 0)
+ io.write(string.format("\n- %s top-level expl3 %s spanning ", titlecase(humanize(num_calls)), pluralize("call", num_calls)))
+ if num_call_tokens == num_tokens then
+ io.write("all tokens")
+ else
+ io.write(string.format("%s %s ", humanize(num_call_tokens), pluralize("token", num_call_tokens)))
+ local formatted_token_ratio = format_ratio(num_call_tokens, num_tokens)
+ local formatted_byte_ratio = format_ratio(num_expl_bytes * num_call_tokens, num_total_bytes * num_tokens)
+ io.write(string.format("(%s of tokens, ~%s of %s)", formatted_token_ratio, formatted_byte_ratio, pluralize("file", num_files)))
+ end
+ end
+
+ ::skip_remaining_additional_information::
+
+ print()
+end
+
-- Print the results of analyzing a file.
-local function print_results(pathname, issues, line_starting_byte_numbers, is_last_file, options)
+local function print_results(pathname, issues, analysis_results, options, evaluation_results, is_last_file)
+ local porcelain, verbose = get_option('porcelain', options), get_option('verbose', options)
+ local line_starting_byte_numbers = analysis_results.line_starting_byte_numbers
+ assert(line_starting_byte_numbers ~= nil)
-- Display an overview.
local all_issues = {}
local status
- local porcelain = get_option('porcelain', options, pathname)
if(#issues.errors > 0) then
- status = (
- colorize(
- (
- tostring(#issues.errors)
- .. " "
- .. pluralize("error", #issues.errors)
- ), 1, 31
+ if not porcelain then
+ status = (
+ colorize(
+ (
+ tostring(#issues.errors)
+ .. " "
+ .. pluralize("error", #issues.errors)
+ ), BOLD, RED
+ )
)
- )
+ end
table.insert(all_issues, issues.errors)
if(#issues.warnings > 0) then
- status = (
- status
- .. ", "
- .. colorize(
+ if not porcelain then
+ status = (
+ status
+ .. ", "
+ .. colorize(
+ (
+ tostring(#issues.warnings)
+ .. " "
+ .. pluralize("warning", #issues.warnings)
+ ), BOLD, YELLOW
+ )
+ )
+ end
+ table.insert(all_issues, issues.warnings)
+ end
+ else
+ if(#issues.warnings > 0) then
+ if not porcelain then
+ status = colorize(
(
tostring(#issues.warnings)
.. " "
.. pluralize("warning", #issues.warnings)
- ), 1, 33
+ ), BOLD, YELLOW
)
- )
+ end
table.insert(all_issues, issues.warnings)
+ else
+ if not porcelain then
+ status = colorize("OK", BOLD, GREEN)
+ end
end
- elseif(#issues.warnings > 0) then
- status = colorize(
- (
- tostring(#issues.warnings)
- .. " "
- .. pluralize("warning", #issues.warnings)
- ), 1, 33
- )
- table.insert(all_issues, issues.warnings)
- else
- status = colorize("OK", 1, 32)
end
if not porcelain then
- local max_overview_length = 72
+ local max_overview_length = get_option('terminal_width', options, pathname)
local prefix = "Checking "
local formatted_pathname = format_pathname(
pathname,
@@ -124,7 +265,7 @@
- #prefix
- #(" ")
- #decolorize(status)
- ), 1
+ ), BOLD
)
)
local overview = (
@@ -137,7 +278,7 @@
- #prefix
- #decolorize(status)
- #formatted_pathname
- ), 1
+ ), BOLD
)
)
.. status
@@ -164,7 +305,8 @@
end_column_number = end_column_number
end
local position = ":" .. tostring(start_line_number) .. ":" .. tostring(start_column_number) .. ":"
- local max_line_length = 88
+ local terminal_width = get_option('terminal_width', options, pathname)
+ local max_line_length = math.max(math.min(88, terminal_width), terminal_width - 16)
local reserved_position_length = 10
local reserved_suffix_length = 30
local label_indent = (" "):rep(4)
@@ -231,25 +373,92 @@
end
end
end
- if not is_last_file and not porcelain then
- print()
+ end
+
+ -- Display additional information.
+ if verbose and not porcelain then
+ local line_indent = (" "):rep(4)
+ print()
+ -- Display pre-evaluation information.
+ local num_total_bytes = evaluation_results.num_total_bytes
+ if num_total_bytes == 0 then
+ io.write(string.format("\n%sEmpty file", line_indent))
+ goto skip_remaining_additional_information
end
+ local formatted_file_size = string.format("%s %s", titlecase(humanize(num_total_bytes)), pluralize("byte", num_total_bytes))
+ io.write(string.format("\n%s%s %s", line_indent, colorize("File size:", BOLD), formatted_file_size))
+ -- Evaluate the evalution results of the preprocessing.
+ io.write(string.format("\n\n%s%s", line_indent, colorize("Preprocessing results:", BOLD)))
+ local seems_like_latex_style_file = analysis_results.seems_like_latex_style_file
+ if seems_like_latex_style_file ~= nil then
+ if seems_like_latex_style_file then
+ io.write(string.format("\n%s- Seems like a LaTeX style file", line_indent))
+ else
+ io.write(string.format("\n%s- Doesn't seem like a LaTeX style file", line_indent))
+ end
+ end
+ local num_expl_bytes = evaluation_results.num_expl_bytes
+ if num_expl_bytes == 0 or num_expl_bytes == nil then
+ io.write(string.format("\n%s- No expl3 material", line_indent))
+ goto skip_remaining_additional_information
+ end
+ local expl_ranges = analysis_results.expl_ranges
+ assert(expl_ranges ~= nil)
+ assert(#expl_ranges > 0)
+ io.write(string.format("\n%s- %s %s spanning ", line_indent, titlecase(humanize(#expl_ranges)), pluralize("expl3 part", #expl_ranges)))
+ if num_expl_bytes == num_total_bytes then
+ io.write("the whole file")
+ else
+ local formatted_expl_bytes = string.format("%s %s", humanize(num_expl_bytes), pluralize("byte", num_expl_bytes))
+ local formatted_expl_ratio = format_ratio(num_expl_bytes, num_total_bytes)
+ io.write(string.format("%s (%s of file)", formatted_expl_bytes, formatted_expl_ratio))
+ end
+ if not (#expl_ranges == 1 and #expl_ranges[1] == num_total_bytes) then
+ io.write(":")
+ for part_number, range in ipairs(expl_ranges) do
+ local start_line_number, start_column_number = utils.convert_byte_to_line_and_column(line_starting_byte_numbers, range:start())
+ local end_line_number, end_column_number = utils.convert_byte_to_line_and_column(line_starting_byte_numbers, range:stop())
+ local formatted_range_start = string.format("%d:%d", start_line_number, start_column_number)
+ local formatted_range_end = string.format("%d:%d", end_line_number, end_column_number)
+ io.write(string.format("\n%s%d. Between ", line_indent:rep(2), part_number))
+ io.write(string.format("%s and %s", formatted_range_start, formatted_range_end))
+ end
+ end
+ -- Evaluate the evalution results of the lexical analysis.
+ io.write(string.format("\n\n%s%s", line_indent, colorize("Lexical analysis results:", BOLD)))
+ local num_tokens = evaluation_results.num_tokens
+ if num_tokens == 0 or num_tokens == nil then
+ io.write(string.format("\n%s- No TeX tokens in expl3 parts", line_indent))
+ goto skip_remaining_additional_information
+ end
+ io.write(string.format("\n%s- %s %s in expl3 parts", line_indent, titlecase(humanize(num_tokens)), pluralize("TeX token", num_tokens)))
+ -- Evaluate the evalution results of the syntactic analysis.
+ io.write(string.format("\n\n%s%s", line_indent, colorize("Syntactic analysis results:", BOLD)))
+ local num_calls = evaluation_results.num_calls
+ local num_call_tokens = evaluation_results.num_call_tokens
+ if num_calls == 0 or num_calls == nil then
+ io.write(string.format("\n%s- No top-level expl3 calls", line_indent))
+ goto skip_remaining_additional_information
+ end
+ assert(num_calls ~= nil)
+ assert(num_calls > 0)
+ io.write(string.format("\n%s- %s %s ", line_indent, titlecase(humanize(num_calls)), pluralize("top-level expl3 call", num_calls)))
+ io.write("spanning ")
+ if num_call_tokens == num_tokens then
+ io.write("all tokens")
+ else
+ local formatted_call_tokens = string.format("%s %s", humanize(num_call_tokens), pluralize("token", num_call_tokens))
+ local formatted_token_ratio = format_ratio(num_call_tokens, num_tokens)
+ local formatted_byte_ratio = format_ratio(num_expl_bytes * num_call_tokens, num_total_bytes * num_tokens)
+ io.write(string.format("%s (%s of tokens, ~%s of file)", formatted_call_tokens, formatted_token_ratio, formatted_byte_ratio))
+ end
end
-end
--- Print the summary results of analyzing multiple files.
-local function print_summary(num_pathnames, num_warnings, num_errors)
- io.write("\n\nTotal: ")
+ ::skip_remaining_additional_information::
- local errors_message = tostring(num_errors) .. " " .. pluralize("error", num_errors)
- errors_message = colorize(errors_message, 1, (num_errors > 0 and 31) or 32)
- io.write(errors_message .. ", ")
-
- local warnings_message = tostring(num_warnings) .. " " .. pluralize("warning", num_warnings)
- warnings_message = colorize(warnings_message, 1, (num_warnings > 0 and 33) or 32)
- io.write(warnings_message .. " in ")
-
- print(tostring(num_pathnames) .. " " .. pluralize("file", num_pathnames))
+ if not porcelain and not is_last_file and (#all_issues > 0 or verbose) then
+ print()
+ end
end
return {
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-issues.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-issues.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-issues.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -4,14 +4,14 @@
function Issues.new(cls)
-- Instantiate the class.
- local new_object = {}
- setmetatable(new_object, cls)
+ local self = {}
+ setmetatable(self, cls)
cls.__index = cls
-- Initialize the class.
- new_object.errors = {}
- new_object.warnings = {}
- new_object.ignored_issues = {}
- return new_object
+ self.errors = {}
+ self.warnings = {}
+ self.ignored_issues = {}
+ return self
end
-- Normalize an issue identifier.
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-lexical-analysis.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-lexical-analysis.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-lexical-analysis.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -1,14 +1,28 @@
-- The lexical analysis step of static analysis converts expl3 parts of the input files into TeX tokens.
local get_option = require("explcheck-config")
-local new_range = require("explcheck-ranges")
+local ranges = require("explcheck-ranges")
local obsolete = require("explcheck-obsolete")
local parsers = require("explcheck-parsers")
+local new_range = ranges.new_range
+local range_flags = ranges.range_flags
+
+local EXCLUSIVE = range_flags.EXCLUSIVE
+local INCLUSIVE = range_flags.INCLUSIVE
+
local lpeg = require("lpeg")
+local token_types = {
+ CONTROL_SEQUENCE = "control sequence",
+ CHARACTER = "character",
+}
+
+local CONTROL_SEQUENCE = token_types.CONTROL_SEQUENCE
+local CHARACTER = token_types.CHARACTER
+
-- Tokenize the content and register any issues.
-local function lexical_analysis(issues, pathname, all_content, expl_ranges, seems_like_latex_style_file, options)
+local function lexical_analysis(pathname, content, issues, results, options)
-- Process bytes within a given range similarly to TeX's input processor (TeX's "eyes" [1]) and produce lines.
--
@@ -23,10 +37,10 @@
-- https://petr.olsak.net/ftp/olsak/tbn/tbn.pdf
--
local function get_lines(range)
- local content = all_content:sub(range:start(), range:stop())
- for _, line in ipairs(lpeg.match(parsers.tex_lines, content)) do
+ local range_content = content:sub(range:start(), range:stop())
+ for _, line in ipairs(lpeg.match(parsers.tex_lines, range_content)) do
local line_start, line_text, line_end = table.unpack(line)
- local line_range = new_range(line_start, line_end, "exclusive", #all_content)
+ local line_range = new_range(line_start, line_end, EXCLUSIVE, #content)
local map_back = (function(line_text, line_range) -- luacheck: ignore line_text line_range
return function (index)
assert(index > 0)
@@ -33,7 +47,7 @@
assert(index <= #line_text + #parsers.expl3_endlinechar)
if index <= #line_text then
local mapped_index = range:start() + line_range:start() + index - 2 -- a line character
- assert(line_text[index] == content[mapped_index])
+ assert(line_text[index] == range_content[mapped_index])
return mapped_index
elseif index > #line_text and index <= #line_text + #parsers.expl3_endlinechar then
return math.max(1, range:start() + line_range:start() + #line_text - 2) -- an \endlinechar
@@ -46,7 +60,7 @@
end
end
- -- Tokenize a line, similarly to TeX's token processor (TeX's "mouth" [1]).
+ -- Process lines similarly to TeX's token processor (TeX's "mouth" [1]) and produce tokens and a tree of apparent TeX groupings.
--
-- See also:
-- - Section 303 on page 122 of Knuth (1986) [1]
@@ -60,13 +74,17 @@
--
local function get_tokens(lines)
local tokens = {}
+
+ local groupings = {}
+ local current_grouping = groupings
+ local parent_grouping
+
local state
- local num_open_groups_upper_estimate = 0
-- Determine the category code of the at sign ("@").
local make_at_letter = get_option("make_at_letter", options, pathname)
if make_at_letter == "auto" then
- make_at_letter = seems_like_latex_style_file
+ make_at_letter = results.seems_like_latex_style_file
end
for line_text, map_back in lines do
@@ -104,7 +122,7 @@
local previous_catcode, previous_csname = 9, nil
while character_index <= #line_text do
local character, catcode, character_index_increment = get_character_and_catcode(character_index)
- local range = new_range(character_index, character_index, "inclusive", #line_text, map_back, #all_content)
+ local range = new_range(character_index, character_index, INCLUSIVE, #line_text, map_back, #content)
if (
catcode ~= 9 and catcode ~= 10 -- a potential missing stylistic whitespace
and (
@@ -154,8 +172,8 @@
end
end
local csname = table.concat(csname_table)
- range = new_range(character_index, previous_csname_index, "inclusive", #line_text, map_back, #all_content)
- table.insert(tokens, {"control sequence", csname, 0, range})
+ range = new_range(character_index, previous_csname_index, INCLUSIVE, #line_text, map_back, #content)
+ table.insert(tokens, {CONTROL_SEQUENCE, csname, 0, range})
if (
previous_catcode ~= 9 and previous_catcode ~= 10 -- a potential missing stylistic whitespace
-- do not require whitespace before non-expl3 control sequences or control sequences with empty or one-character names
@@ -167,9 +185,9 @@
character_index = csname_index
elseif catcode == 5 then -- end of line
if state == "N" then
- table.insert(tokens, {"control sequence", "par", range})
+ table.insert(tokens, {CONTROL_SEQUENCE, "par", 0, range})
elseif state == "M" then
- table.insert(tokens, {"character", " ", 10, range})
+ table.insert(tokens, {CHARACTER, " ", 10, range})
end
character_index = character_index + character_index_increment
elseif catcode == 9 then -- ignored character
@@ -177,7 +195,7 @@
character_index = character_index + character_index_increment
elseif catcode == 10 then -- space
if state == "M" then
- table.insert(tokens, {"character", " ", 10, range})
+ table.insert(tokens, {CHARACTER, " ", 10, range})
end
previous_catcode = catcode
character_index = character_index + character_index_increment
@@ -188,11 +206,19 @@
character_index = character_index + character_index_increment
else
if catcode == 1 or catcode == 2 then -- begin/end grouping
- if catcode == 1 then
- num_open_groups_upper_estimate = num_open_groups_upper_estimate + 1
- elseif catcode == 2 then
- if num_open_groups_upper_estimate > 0 then
- num_open_groups_upper_estimate = num_open_groups_upper_estimate - 1
+ if catcode == 1 then -- begin grouping
+ current_grouping = {parent = current_grouping, start = #tokens + 1}
+ assert(groupings[current_grouping.start] == nil)
+ assert(current_grouping.parent[current_grouping.start] == nil)
+ groupings[current_grouping.start] = current_grouping -- provide flat access to groupings
+ current_grouping.parent[current_grouping.start] = current_grouping -- provide recursive access to groupings
+ elseif catcode == 2 then -- end grouping
+ if current_grouping.parent ~= nil then
+ current_grouping.stop = #tokens + 1
+ assert(current_grouping.start ~= nil and current_grouping.start < current_grouping.stop)
+ parent_grouping = current_grouping.parent
+ current_grouping.parent = nil -- remove a circular reference for the current grouping
+ current_grouping = parent_grouping
else
issues:add('e208', 'too many closing braces', range)
end
@@ -215,18 +241,24 @@
else -- some other character
previous_catcode = catcode
end
- table.insert(tokens, {"character", character, catcode, range})
+ table.insert(tokens, {CHARACTER, character, catcode, range})
state = "M"
character_index = character_index + character_index_increment
end
end
end
- return tokens
+ -- Remove circular references for all unclosed groupings.
+ while current_grouping.parent ~= nil do
+ parent_grouping = current_grouping.parent
+ current_grouping.parent = nil
+ current_grouping = parent_grouping
+ end
+ return tokens, groupings
end
-- Tokenize the content.
- local all_tokens = {}
- for _, range in ipairs(expl_ranges) do
+ local tokens, groupings = {}, {}
+ for _, range in ipairs(results.expl_ranges) do
local lines = (function()
local co = coroutine.create(function()
get_lines(range)
@@ -236,14 +268,15 @@
return line_text, map_back
end
end)()
- local tokens = get_tokens(lines)
- table.insert(all_tokens, tokens)
+ local part_tokens, part_groupings = get_tokens(lines)
+ table.insert(tokens, part_tokens)
+ table.insert(groupings, part_groupings)
end
- for _, tokens in ipairs(all_tokens) do
- for token_index, token in ipairs(tokens) do
+ for _, part_tokens in ipairs(tokens) do
+ for token_index, token in ipairs(part_tokens) do
local token_type, payload, catcode, range = table.unpack(token) -- luacheck: ignore catcode
- if token_type == "control sequence" then
+ if token_type == CONTROL_SEQUENCE then
local csname = payload
local _, _, argument_specifiers = csname:find(":([^:]*)")
if argument_specifiers ~= nil then
@@ -257,10 +290,10 @@
if lpeg.match(obsolete.deprecated_csname, csname) ~= nil then
issues:add('w202', 'deprecated control sequences', range)
end
- if token_index + 1 <= #tokens then
- local next_token = tokens[token_index + 1]
+ if token_index + 1 <= #part_tokens then
+ local next_token = part_tokens[token_index + 1]
local next_token_type, next_csname, _, next_range = table.unpack(next_token)
- if next_token_type == "control sequence" then
+ if next_token_type == CONTROL_SEQUENCE then
if (
lpeg.match(parsers.expl3_function_assignment_csname, csname) ~= nil
and lpeg.match(parsers.non_expl3_csname, next_csname) == nil
@@ -288,7 +321,12 @@
end
end
- return all_tokens
+ -- Store the intermediate results of the analysis.
+ results.tokens = tokens
+ results.groupings = groupings
end
-return lexical_analysis
+return {
+ process = lexical_analysis,
+ token_types = token_types,
+}
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-obsolete.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-obsolete.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-obsolete.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -9,7 +9,7 @@
-- luacheck: push no max line length
local obsolete = {}
-obsolete.deprecated_csname = (P("k") * (P("e") * (P("y") * (P("s") * (P("_") * (P("s") * (P("e") * (P("t") * (P("_") * (P("f") * (P("i") * (P("l") * (P("t") * (P("e") * (P("r") * (P(":") * (P("n") * (P("n") * (P("o") * (P("nN") + P("N")) + P("v") * (P("nN") + P("N")) + P("V") * (P("nN") + P("N")) + P("n") * (P("nN") + P("N")) + P("n") + P("V") + P("v") + P("o"))))))))))))))))))) + P("l") * (P("_") * (P("k") * (P("e") * (P("y") * (P("s") * (P("_") * (P("key_tl") + P("path_tl")))))) + P("t") * (P("e") * (P("x") * (P("t") * (P("_") * (P("accents_tl") + P("letterlike_tl")))))))) + P("i") * (P("o") * (P("w") * (P("_") * (P("s") * (P("h") * (P("i") * (P("p") * (P("o") * (P("u") * (P("t") * (P("_") * (P("x") * (P(":") * (P("c") * (P("n") + P("x")) + P("N") * (P("n") + P("x")))))))))))))))) + P("t") * (P("e") * (P("x") * (P("t") * (P("_") * (P("t") * (P("i") * (P("t") * (P("l") * (P("e") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n"))))))))))))))) + P("l") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("c") * (P("n") * (P("TF") + P("F") + P("T")) + P("n")) + P("N") * (P("n") * (P("TF") + P("F") + P("T")) + P("n"))))))) + P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n")))))))))))) + P("b") * (P("u") * (P("i") * (P("l") * (P("d") * (P("_") * (P("clear:N") + P("g") * (P("clear:N") + P("et:NN")))))))) + P("m") * (P("i") * (P("x") * (P("e") * (P("d") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n")))))))))))) + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n"))))))))))))))) + P("s") * (P("t") * (P("r") * (P("_") * (P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("f") + P("n")))))))))))) + P("declare_eight_bit_encoding:nnn") + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":")!
* (P("f") + P("n")))))))))))) + P("f") * (P("o") * (P("l") * (P("d") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("n") + P("V"))))))) + P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("n") + P("V"))))))))))))) + P("e") * (P("q") * (P("_") * (P("gset_map_x:NNn") + P("i") * (P("n") * (P("d") * (P("e") * (P("x") * (P("e") * (P("d") * (P("_") * (P("m") * (P("a") * (P("p") * (P("_") * (P("inline:Nn") + P("function:NN"))))))))))))) + P("set_map_x:NNn")))) + P("ys_load_deprecation:")) + P("p") * (P("d") * (P("f") * (P("_") * (P("o") * (P("b") * (P("j") * (P("e") * (P("c") * (P("t") * (P("_") * (P("w") * (P("r") * (P("i") * (P("t") * (P("e") * (P(":") * (P("n") * (P("n") + P("x")))))))) + P("new:nn"))))))))))) + P("e") * (P("e") * (P("k") * (P("_") * (P("c") * (P("a") * (P("t") * (P("c") * (P("o") * (P("d") * (P("e") * (P("_") * (P("ignore_spaces:N") + P("remove_ignore_spaces:N")))))))) + P("h") * (P("a") * (P("r") * (P("c") * (P("o") * (P("d") * (P("e") * (P("_") * (P("ignore_spaces:N") + P("remove_ignore_spaces:N")))))))))) + P("m") * (P("e") * (P("a") * (P("n") * (P("i") * (P("n") * (P("g") * (P("_") * (P("ignore_spaces:N") + P("remove_ignore_spaces:N"))))))))))))) + P("r") * (P("o") * (P("p") * (P("_") * (P("g") * (P("p") * (P("u") * (P("t") * (P("_") * (P("i") * (P("f") * (P("_") * (P("n") * (P("e") * (P("w") * (P(":") * (P("c") * (P("Vn") + P("n") * (P("n") + P("V"))) + P("N") * (P("Vn") + P("n") * (P("n") + P("V"))))))))))))))) + P("p") * (P("u") * (P("t") * (P("_") * (P("i") * (P("f") * (P("_") * (P("n") * (P("e") * (P("w") * (P(":") * (P("c") * (P("Vn") + P("n") * (P("n") + P("V"))) + P("N") * (P("Vn") + P("n") * (P("n") + P("V"))))))))))))))))))) + P("m") * (P("s") * (P("g") * (P("_") * (P("g") * (P("s") * (P("e") * (P("t") * (P(":") * (P("n") * (P("n") * (P("nn") + P("n")))))))))))) + P("c") * (P("s_argument_spec:N") + P("h") * (P("a") * (P("r") * (P("_") * (P("s") * (P("t") * (P("r") * (P("_") * (P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("_case:N") + P("case:N")))))) + P("u!
") * (P("p") * (P("p") * (P("e") * (P("r") * (P("_case:N") + P("case:N")))))) + P("titlecase:N") + P("mixed_case:N") + P("f") * (P("o") * (P("l") * (P("d") * (P("_case:N") + P("case:N"))))))))) + P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("_case:N") + P("case:N")))))) + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("_case:N") + P("case:N")))))) + P("t") * (P("itlecase:N") + P("o") * (P("_") * (P("utfviii_bytes:n") + P("nfd:N")))) + P("mixed_case:N") + P("f") * (P("o") * (P("l") * (P("d") * (P("_case:N") + P("case:N"))))))))))) * eof
+obsolete.deprecated_csname = (P("l") * (P("_") * (P("t") * (P("e") * (P("x") * (P("t") * (P("_") * (P("letterlike_tl") + P("accents_tl")))))) + P("k") * (P("e") * (P("y") * (P("s") * (P("_") * (P("path_tl") + P("key_tl")))))))) + P("m") * (P("s") * (P("g") * (P("_") * (P("g") * (P("s") * (P("e") * (P("t") * (P(":") * (P("n") * (P("n") * (P("nn") + P("n")))))))))))) + P("t") * (P("l") * (P("_") * (P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n")))))))))))) + P("m") * (P("i") * (P("x") * (P("e") * (P("d") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n")))))))))))) + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n")))))))))))) + P("b") * (P("u") * (P("i") * (P("l") * (P("d") * (P("_") * (P("g") * (P("et:NN") + P("clear:N")) + P("clear:N"))))))) + P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("N") * (P("n") * (P("TF") + P("F") + P("T")) + P("n")) + P("c") * (P("n") * (P("TF") + P("F") + P("T")) + P("n"))))))))) + P("e") * (P("x") * (P("t") * (P("_") * (P("t") * (P("i") * (P("t") * (P("l") * (P("e") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("nn") + P("n")))))))))))))))) + P("s") * (P("t") * (P("r") * (P("_") * (P("declare_eight_bit_encoding:nnn") + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("f") + P("n")))))))))))) + P("f") * (P("o") * (P("l") * (P("d") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("n") + P("V")))))) + P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("n") + P("V"))))))))))) + P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("_") * (P("c") * (P("a") * (P("s") * (P("e") * (P(":") * (P("f") + P("n"))))))))))))))) + P("e") * (P("q") * (P("_") * (P("i") * (P("n") * (P("d") * (P("e") * (P("x") * (P("e") * (P("d") * (P("_") * (P("m") * (P("a") * (P("p") * (P("_") * (P("function:NN") + P("inline:Nn")))))))!
)))))) + P("set_map_x:NNn") + P("gset_map_x:NNn")))) + P("ys_load_deprecation:")) + P("p") * (P("d") * (P("f") * (P("_") * (P("o") * (P("b") * (P("j") * (P("e") * (P("c") * (P("t") * (P("_") * (P("new:nn") + P("w") * (P("r") * (P("i") * (P("t") * (P("e") * (P(":") * (P("n") * (P("n") + P("x")))))))))))))))))) + P("e") * (P("e") * (P("k") * (P("_") * (P("m") * (P("e") * (P("a") * (P("n") * (P("i") * (P("n") * (P("g") * (P("_") * (P("remove_ignore_spaces:N") + P("ignore_spaces:N"))))))))) + P("c") * (P("h") * (P("a") * (P("r") * (P("c") * (P("o") * (P("d") * (P("e") * (P("_") * (P("remove_ignore_spaces:N") + P("ignore_spaces:N"))))))))) + P("a") * (P("t") * (P("c") * (P("o") * (P("d") * (P("e") * (P("_") * (P("remove_ignore_spaces:N") + P("ignore_spaces:N"))))))))))))) + P("r") * (P("o") * (P("p") * (P("_") * (P("p") * (P("u") * (P("t") * (P("_") * (P("i") * (P("f") * (P("_") * (P("n") * (P("e") * (P("w") * (P(":") * (P("N") * (P("n") * (P("n") + P("V")) + P("Vn")) + P("c") * (P("n") * (P("n") + P("V")) + P("Vn"))))))))))))) + P("g") * (P("p") * (P("u") * (P("t") * (P("_") * (P("i") * (P("f") * (P("_") * (P("n") * (P("e") * (P("w") * (P(":") * (P("N") * (P("n") * (P("n") + P("V")) + P("Vn")) + P("c") * (P("n") * (P("n") + P("V")) + P("Vn"))))))))))))))))))) + P("i") * (P("o") * (P("w") * (P("_") * (P("s") * (P("h") * (P("i") * (P("p") * (P("o") * (P("u") * (P("t") * (P("_") * (P("x") * (P(":") * (P("N") * (P("n") + P("x")) + P("c") * (P("n") + P("x")))))))))))))))) + P("k") * (P("e") * (P("y") * (P("s") * (P("_") * (P("s") * (P("e") * (P("t") * (P("_") * (P("f") * (P("i") * (P("l") * (P("t") * (P("e") * (P("r") * (P(":") * (P("n") * (P("n") * (P("n") * (P("nN") + P("N")) + P("v") * (P("nN") + P("N")) + P("V") * (P("nN") + P("N")) + P("o") * (P("nN") + P("N")) + P("n") + P("V") + P("v") + P("o"))))))))))))))))))) + P("c") * (P("h") * (P("a") * (P("r") * (P("_") * (P("t") * (P("o") * (P("_") * (P("nfd:N") + P("utfviii_bytes:n"))) + P("itlecase:N")) + P("mixed_case:N") + P("f") * (P("o") * (P("l") * (P("d") * (P("cas!
e:N") + P("_case:N"))))) + P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("case:N") + P("_case:N")))))) + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("case:N") + P("_case:N")))))) + P("s") * (P("t") * (P("r") * (P("_") * (P("titlecase:N") + P("mixed_case:N") + P("f") * (P("o") * (P("l") * (P("d") * (P("case:N") + P("_case:N"))))) + P("u") * (P("p") * (P("p") * (P("e") * (P("r") * (P("case:N") + P("_case:N")))))) + P("l") * (P("o") * (P("w") * (P("e") * (P("r") * (P("case:N") + P("_case:N")))))))))))))) + P("s_argument_spec:N"))) * eof
-- luacheck: pop
return obsolete
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-parsers.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-parsers.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-parsers.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -8,6 +8,7 @@
local any = P(1)
local eof = -any
local fail = P(false)
+local success = P(true)
---- Tokens
local ampersand = P("&")
@@ -59,10 +60,11 @@
[11] = letter + colon + underscore, -- letter
[13] = form_feed, -- active character
[14] = percent_sign, -- comment character
- [15] = control_character, -- invalid character
+ [15] = control_character - newline, -- invalid character
}
expl3_catcodes[12] = any -- other
-for catcode, parser in pairs(expl3_catcodes) do
+for catcode = 0, 15 do
+ local parser = expl3_catcodes[catcode]
if catcode ~= 12 then
expl3_catcodes[12] = expl3_catcodes[12] - parser
end
@@ -69,10 +71,14 @@
end
local determine_expl3_catcode = fail
-for catcode, parser in pairs(expl3_catcodes) do
+for catcode = 0, 15 do
+ local parser = expl3_catcodes[catcode]
determine_expl3_catcode = (
determine_expl3_catcode
- + parser / function() return catcode end
+ + parser
+ / function()
+ return catcode
+ end
)
end
@@ -158,10 +164,15 @@
* expl3_catcodes[2]
)
+local N_type_argument_specifier = S("NV")
+local n_type_argument_specifier = S("ncvoxefTF")
+local parameter_argument_specifier = S("p")
local weird_argument_specifier = S("w")
local do_not_use_argument_specifier = S("D")
local argument_specifier = (
- S("NncVvoxefTFp")
+ N_type_argument_specifier
+ + n_type_argument_specifier
+ + parameter_argument_specifier
+ weird_argument_specifier
+ do_not_use_argument_specifier
)
@@ -354,7 +365,7 @@
---- Standard delimiters
local provides = (
expl3_catcodes[0]
- * P([[ProvidesExpl]])
+ * P("ProvidesExpl")
* (
P("Package")
+ P("Class")
@@ -369,8 +380,16 @@
* optional_spaces_and_newline
* argument
)
-local expl_syntax_on = expl3_catcodes[0] * P([[ExplSyntaxOn]])
-local expl_syntax_off = expl3_catcodes[0] * P([[ExplSyntaxOff]])
+local expl_syntax_on = expl3_catcodes[0] * P("ExplSyntaxOn")
+local expl_syntax_off = expl3_catcodes[0] * P("ExplSyntaxOff")
+local endinput = (
+ expl3_catcodes[0]
+ * (
+ P("tex_endinput:D")
+ + P("endinput")
+ + P("file_input_stop:")
+ )
+)
---- Commands from LaTeX style files
local latex_style_file_csname =
@@ -511,26 +530,36 @@
commented_lines = commented_lines,
decimal_digit = decimal_digit,
determine_expl3_catcode = determine_expl3_catcode,
+ do_not_use_argument_specifier = do_not_use_argument_specifier,
do_not_use_argument_specifiers = do_not_use_argument_specifiers,
double_superscript_convention = double_superscript_convention,
+ endinput = endinput,
eof = eof,
- fail = fail,
- expl3like_material = expl3like_material,
+ expl3_catcodes = expl3_catcodes,
expl3_endlinechar = expl3_endlinechar,
expl3_function_assignment_csname = expl3_function_assignment_csname,
expl3_function_csname = expl3_function_csname,
+ expl3like_material = expl3like_material,
+ expl3_quark_or_scan_mark_csname = expl3_quark_or_scan_mark_csname,
+ expl3_quark_or_scan_mark_definition_csname = expl3_quark_or_scan_mark_definition_csname,
expl3_scratch_variable_csname = expl3_scratch_variable_csname,
expl3_variable_or_constant_csname = expl3_variable_or_constant_csname,
expl3_variable_or_constant_use_csname = expl3_variable_or_constant_use_csname,
- expl3_quark_or_scan_mark_csname = expl3_quark_or_scan_mark_csname,
- expl3_quark_or_scan_mark_definition_csname = expl3_quark_or_scan_mark_definition_csname,
expl_syntax_off = expl_syntax_off,
expl_syntax_on = expl_syntax_on,
+ fail = fail,
ignored_issues = ignored_issues,
latex_style_file_content = latex_style_file_content,
linechar = linechar,
newline = newline,
non_expl3_csname = non_expl3_csname,
+ n_type_argument_specifier = n_type_argument_specifier,
+ N_type_argument_specifier = N_type_argument_specifier,
+ parameter_argument_specifier = parameter_argument_specifier,
provides = provides,
+ space = space,
+ success = success,
+ tab = tab,
tex_lines = tex_lines,
+ weird_argument_specifier = weird_argument_specifier,
}
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-preprocessing.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-preprocessing.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-preprocessing.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -1,15 +1,21 @@
-- The preprocessing step of static analysis determines which parts of the input files contain expl3 code.
local get_option = require("explcheck-config")
-local new_range = require("explcheck-ranges")
+local ranges = require("explcheck-ranges")
local parsers = require("explcheck-parsers")
local utils = require("explcheck-utils")
+local new_range = ranges.new_range
+local range_flags = ranges.range_flags
+
+local EXCLUSIVE = range_flags.EXCLUSIVE
+local INCLUSIVE = range_flags.INCLUSIVE
+
local lpeg = require("lpeg")
-local Cp, P, V = lpeg.Cp, lpeg.P, lpeg.V
+local B, Cmt, Cp, Ct, Cc, P, V = lpeg.B, lpeg.Cmt, lpeg.Cp, lpeg.Ct, lpeg.Cc, lpeg.P, lpeg.V
-- Preprocess the content and register any issues.
-local function preprocessing(issues, pathname, content, options)
+local function preprocessing(pathname, content, issues, results, options)
-- Determine the bytes where lines begin.
local line_starting_byte_numbers = {}
@@ -51,10 +57,10 @@
local comment_range_end, comment_range
if(comment_line_number + 1 <= #line_starting_byte_numbers) then
comment_range_end = line_starting_byte_numbers[comment_line_number + 1]
- comment_range = new_range(comment_range_start, comment_range_end, "exclusive", #content)
+ comment_range = new_range(comment_range_start, comment_range_end, EXCLUSIVE, #content)
else
comment_range_end = #content
- comment_range = new_range(comment_range_start, comment_range_end, "inclusive", #content)
+ comment_range = new_range(comment_range_start, comment_range_end, INCLUSIVE, #content)
end
if #ignored_issues == 0 then -- ignore all issues on this line
issues:ignore(nil, comment_range)
@@ -95,16 +101,20 @@
-- Determine which parts of the input files contain expl3 code.
local expl_ranges = {}
+ local input_ended = false
- local function capture_range(range_start, range_end)
- local range = new_range(range_start, range_end, "exclusive", #transformed_content, map_back, #content)
- table.insert(expl_ranges, range)
+ local function capture_range(should_skip, range_start, range_end)
+ if not should_skip then
+ local range = new_range(range_start, range_end, EXCLUSIVE, #transformed_content, map_back, #content)
+ table.insert(expl_ranges, range)
+ end
end
local function unexpected_pattern(pattern, code, message, test)
- return Cp() * pattern * Cp() / function(range_start, range_end)
- local range = new_range(range_start, range_end, "exclusive", #transformed_content, map_back, #content)
- if test == nil or test() then
+ return Ct(Cp() * pattern * Cp()) / function(range_table)
+ if not input_ended and (test == nil or test()) then
+ local range_start, range_end = range_table[#range_table - 1], range_table[#range_table]
+ local range = new_range(range_start, range_end, EXCLUSIVE, #transformed_content, map_back, #content)
issues:add(code, message, range)
end
end
@@ -111,22 +121,59 @@
end
local num_provides = 0
- local Opener, Closer = parsers.fail, parsers.fail
+ local FirstLineProvides, FirstLineExplSyntaxOn, HeadlessCloser, Head, Any =
+ parsers.fail, parsers.fail, parsers.fail, parsers.fail, parsers.any
local expl3_detection_strategy = get_option('expl3_detection_strategy', options, pathname)
if expl3_detection_strategy ~= 'never' and expl3_detection_strategy ~= 'always' then
- Opener = (
- parsers.expl_syntax_on
- + unexpected_pattern(
- parsers.provides,
- "e104",
- [[multiple delimiters `\ProvidesExpl*` in a single file]],
+ FirstLineProvides = unexpected_pattern(
+ parsers.provides,
+ "e104",
+ [[multiple delimiters `\ProvidesExpl*` in a single file]],
+ function()
+ num_provides = num_provides + 1
+ return num_provides > 1
+ end
+ )
+ FirstLineExplSyntaxOn = parsers.expl_syntax_on
+ HeadlessCloser = (
+ parsers.expl_syntax_off
+ + parsers.endinput
+ / function()
+ input_ended = true
+ end
+ )
+ -- (Under)estimate the current TeX grouping level.
+ local estimated_grouping_level = 0
+ Any = (
+ -B(parsers.expl3_catcodes[0]) -- no preceding backslash
+ * parsers.expl3_catcodes[1] -- begin grouping
+ * Cmt(
+ parsers.success,
function()
- num_provides = num_provides + 1
- return num_provides > 1
+ estimated_grouping_level = estimated_grouping_level + 1
+ return true
end
)
+ + parsers.expl3_catcodes[2] -- end grouping
+ * Cmt(
+ parsers.success,
+ function()
+ estimated_grouping_level = math.max(0, estimated_grouping_level - 1)
+ return true
+ end
+ )
+ + parsers.any
)
- Closer = parsers.expl_syntax_off
+ -- Allow indent before a standard delimiter outside a TeX grouping.
+ Head = (
+ parsers.newline
+ + Cmt(
+ parsers.success,
+ function()
+ return estimated_grouping_level == 0
+ end
+ )
+ )
end
local has_expl3like_material = false
@@ -134,6 +181,9 @@
"Root";
Root = (
(
+ V"FirstLineExplPart" / capture_range
+ )^-1
+ * (
V"NonExplPart"
* V"ExplPart" / capture_range
)^0
@@ -142,7 +192,11 @@
NonExplPart = (
(
unexpected_pattern(
- V"Closer",
+ (
+ V"Head"
+ * Cp()
+ * V"HeadlessCloser"
+ ),
"w101",
"unexpected delimiters"
)
@@ -155,25 +209,64 @@
return true
end
)
- + (parsers.any - V"Opener")
+ + (
+ V"Any"
+ - V"Opener"
+ )
)^0
),
- ExplPart = (
- V"Opener"
+ FirstLineExplPart = (
+ Cc(input_ended)
+ * V"FirstLineOpener"
* Cp()
* (
- unexpected_pattern(
- V"Opener",
+ V"Provides"
+ + unexpected_pattern(
+ (
+ V"Head"
+ * Cp()
+ * V"FirstLineOpener"
+ ),
"w101",
"unexpected delimiters"
)
- + (parsers.any - V"Closer")
+ + (
+ V"Any"
+ - V"Closer"
+ )
)^0
- * Cp()
- * (V"Closer" + parsers.eof)
+ * (
+ V"Head"
+ * Cp()
+ * V"HeadlessCloser"
+ + Cp()
+ * parsers.eof
+ )
),
- Opener = Opener,
- Closer = Closer,
+ ExplPart = (
+ V"Head"
+ * V"FirstLineExplPart"
+ ),
+ FirstLineProvides = FirstLineProvides,
+ Provides = (
+ V"Head"
+ * V"FirstLineProvides"
+ ),
+ FirstLineOpener = (
+ FirstLineExplSyntaxOn
+ + V"FirstLineProvides"
+ ),
+ Opener = (
+ V"Head"
+ * V"FirstLineOpener"
+ ),
+ HeadlessCloser = HeadlessCloser,
+ Closer = (
+ V"Head"
+ * V"HeadlessCloser"
+ ),
+ Head = Head,
+ Any = Any,
}
lpeg.match(analysis_grammar, transformed_content)
@@ -183,7 +276,7 @@
if suffix == ".cls" or suffix == ".opt" or suffix == ".sty" then
seems_like_latex_style_file = true
else
- seems_like_latex_style_file = lpeg.match(parsers.latex_style_file_content, transformed_content)
+ seems_like_latex_style_file = lpeg.match(parsers.latex_style_file_content, transformed_content) ~= nil
end
-- If no expl3 parts were detected, decide whether no part or the whole input file is in expl3.
@@ -196,7 +289,7 @@
if expl3_detection_strategy == "recall" then
issues:add('w100', 'no standard delimiters')
end
- local range = new_range(1, #content, "inclusive", #content)
+ local range = new_range(1, #content, INCLUSIVE, #content)
table.insert(expl_ranges, range)
elseif expl3_detection_strategy == "auto" then
-- Use context clues to determine whether no part or the whole
@@ -203,7 +296,7 @@
-- input file is in expl3.
if has_expl3like_material then
issues:add('w100', 'no standard delimiters')
- local range = new_range(1, #content, "inclusive", #content)
+ local range = new_range(1, #content, INCLUSIVE, #content)
table.insert(expl_ranges, range)
end
else
@@ -216,22 +309,27 @@
local offset = expl_range:start() - 1
local function line_too_long(range_start, range_end)
- local range = new_range(offset + range_start, offset + range_end, "exclusive", #transformed_content, map_back, #content)
- issues:add('s103', 'line too long', range)
+ local range = new_range(offset + range_start, offset + range_end, EXCLUSIVE, #transformed_content, map_back, #content)
+ issues:add('s103', 'line too long', range)
+ end
+
+ local overline_lines_grammar = (
+ (
+ Cp() * parsers.linechar^(get_option('max_line_length', options, pathname) + 1) * Cp() / line_too_long
+ + parsers.linechar^0
+ )
+ * parsers.newline
+ )^0
+
+ lpeg.match(overline_lines_grammar, transformed_content:sub(expl_range:start(), expl_range:stop()))
end
- local overline_lines_grammar = (
- (
- Cp() * parsers.linechar^(get_option('max_line_length', options, pathname) + 1) * Cp() / line_too_long
- + parsers.linechar^0
- )
- * parsers.newline
- )^0
-
- lpeg.match(overline_lines_grammar, transformed_content:sub(expl_range:start(), expl_range:stop()))
+ -- Store the intermediate results of the analysis.
+ results.line_starting_byte_numbers = line_starting_byte_numbers
+ results.expl_ranges = expl_ranges
+ results.seems_like_latex_style_file = seems_like_latex_style_file
end
- return line_starting_byte_numbers, expl_ranges, seems_like_latex_style_file
-end
-
-return preprocessing
+ return {
+ process = preprocessing
+}
Modified: trunk/Master/texmf-dist/scripts/expltools/explcheck-ranges.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-ranges.lua 2025-03-28 00:42:54 UTC (rev 74770)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-ranges.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -2,54 +2,98 @@
local Range = {}
+local range_flags = {
+ EXCLUSIVE = 0,
+ INCLUSIVE = 1,
+ MAYBE_EMPTY = 2,
+}
+
+local EXCLUSIVE = range_flags.EXCLUSIVE
+local INCLUSIVE = range_flags.INCLUSIVE
+local MAYBE_EMPTY = range_flags.MAYBE_EMPTY
+
-- Create a new range based on the start/end indices, the type of the end index
--- (inclusive/exclusive), the size of the array that contains the range, and an
--- optional nondecreasing map-back function from indices in the array to
--- indices in an original array and the size of the original array.
+-- (INCLUSIVE/EXCLUSIVE, MAYBE_EMPTY), the size of the array that contains the
+-- range, and an optional nondecreasing map-back function from indices in the
+-- array to indices in an original array and the size of the original array.
--
--- Since empty ranges are usually a mistake, they are not allowed.
--- For example, `Range:new(index, index, "exclusive", #array)` will cause an
--- assertion error. The exception to this rule are empty arrays, which permit
--- the empty range `Range:new(0, 0, "inclusive", 0)`.
+-- Since empty ranges are usually a mistake, they are not allowed unless MAYBE_EMPTY
+-- is specified. For example, `Range:new(index, index, EXCLUSIVE, #array)` is not
+-- allowed but `Range:new(index, index, EXCLUSIVE + MAYBE_EMPTY, #array)` is.
+-- The exception to this rule are empty arrays, which always produce an empty range.
function Range.new(cls, range_start, range_end, end_type, transformed_array_size, map_back, original_array_size)
-- Instantiate the class.
- local new_object = {}
- setmetatable(new_object, cls)
+ local self = {}
+ setmetatable(self, cls)
cls.__index = cls
-- Check pre-conditions.
if transformed_array_size == 0 then
- assert(range_start == 0)
+ -- If the transformed array is empty, produce an empty range, encoded as [0; 0].
+ range_start = 0
+ range_end = 0
+ end_type = INCLUSIVE + MAYBE_EMPTY
else
+ -- Otherwise, check that the range start is not out of bounds.
assert(range_start >= 1)
assert(range_start <= transformed_array_size)
end
- assert(end_type == "inclusive" or end_type == "exclusive")
- if end_type == "exclusive" then
+ local exclusive_end = end_type % 2 == EXCLUSIVE
+ local maybe_empty = end_type - (end_type % 2) == MAYBE_EMPTY
+ if exclusive_end then
-- Convert exclusive range end to inclusive.
range_end = range_end - 1
end
if transformed_array_size == 0 then
+ -- If the transformed array is empty, only allow empty ranges, encoded as [0; 0].
+ assert(range_start == 0)
assert(range_end == 0)
else
- assert(range_end >= range_start)
+ -- Otherwise:
+ if maybe_empty then
+ -- If MAYBE_EMPTY is specified, allow empty ranges [x, x).
+ assert(range_end >= range_start - 1)
+ else
+ -- Otherwise, only allow non-empty ranges [x, y].
+ assert(range_end >= range_start)
+ end
+ -- Check that the range end is not out of bounds.
assert(range_end <= transformed_array_size)
end
+ -- Apply the map-back function.
local mapped_range_start, mapped_range_end
if map_back ~= nil then
- -- Apply the map-back function to the endpoints of the range.
+ -- Apply the map-back function to the range start.
assert(original_array_size ~= nil)
mapped_range_start = map_back(range_start)
if original_array_size == 0 then
+ -- If the original array is empty, check that the range start has stayed at 0.
assert(mapped_range_start == 0)
else
+ -- Otherwise, check that the range start is not out of bounds.
assert(mapped_range_start >= 1)
assert(mapped_range_start <= original_array_size)
end
- mapped_range_end = map_back(range_end)
+ if range_end < range_start then
+ -- If the range is supposed to be empty, set the range end to the range start - 1.
+ assert(maybe_empty)
+ mapped_range_end = mapped_range_start - 1
+ else
+ -- Otherwise, apply the map-back function to the range end as well.
+ mapped_range_end = map_back(range_end)
+ end
if original_array_size == 0 then
+ -- If the original array is empty, check that the range end has also stayed at 0.
assert(mapped_range_end == 0)
else
- assert(mapped_range_end >= mapped_range_start)
+ -- Otherwise:
+ if maybe_empty then
+ -- If MAYBE_EMPTY is specified, allow empty ranges [x, x).
+ assert(mapped_range_end >= mapped_range_start - 1)
+ else
+ -- Otherwise, only allow non-empty ranges [x, y].
+ assert(mapped_range_end >= mapped_range_start)
+ end
+ -- Check that the range end is not out of bounds.
assert(mapped_range_end <= original_array_size)
end
else
@@ -57,9 +101,9 @@
mapped_range_end = range_end
end
-- Initialize the class.
- new_object.range_start = mapped_range_start
- new_object.range_end = mapped_range_end
- return new_object
+ self.range_start = mapped_range_start
+ self.range_end = mapped_range_end
+ return self
end
-- Get the inclusive start of the range, optionally mapped back to the original array.
@@ -72,6 +116,55 @@
return self.range_end
end
-return function(...)
- return Range:new(...)
+-- Get the length of the range.
+function Range:__len()
+ if self:start() == 0 then
+ assert(self:stop() == 0)
+ return 0 -- empty range
+ elseif self:stop() < self:start() then
+ assert(self:stop() == self:start() - 1)
+ return 0 -- empty range
+ else
+ return self:stop() - self:start() + 1 -- non-empty range
+ end
end
+
+-- Get an iterator over the range.
+function Range:iter()
+ if #self == 0 then
+ return function() -- empty range
+ return nil
+ end
+ else
+ local i = self:start() - 1
+ return function() -- non-empty range
+ i = i + 1
+ if i <= self:stop() then
+ return i
+ else
+ return nil
+ end
+ end
+ end
+end
+
+-- Get a string representation of the range.
+function Range:__tostring()
+ if #self == 0 then
+ if self:start() == 0 then
+ assert(self:stop() == 0)
+ return "(0, 0)" -- empty range in empty array
+ else
+ return string.format("[%d, %d)", self:start(), self:stop() + 1) -- empty range in non-empty array
+ end
+ else
+ return string.format("[%d, %d]", self:start(), self:stop()) -- non-empty range
+ end
+end
+
+return {
+ new_range = function(...)
+ return Range:new(...)
+ end,
+ range_flags = range_flags,
+}
Added: trunk/Master/texmf-dist/scripts/expltools/explcheck-syntactic-analysis.lua
===================================================================
--- trunk/Master/texmf-dist/scripts/expltools/explcheck-syntactic-analysis.lua (rev 0)
+++ trunk/Master/texmf-dist/scripts/expltools/explcheck-syntactic-analysis.lua 2025-03-28 21:25:19 UTC (rev 74771)
@@ -0,0 +1,365 @@
+-- The syntactic analysis step of static analysis converts TeX tokens into a tree of function calls.
+
+local ranges = require("explcheck-ranges")
+local parsers = require("explcheck-parsers")
+
+local new_range = ranges.new_range
+local range_flags = ranges.range_flags
+
+local EXCLUSIVE = range_flags.EXCLUSIVE
+local INCLUSIVE = range_flags.INCLUSIVE
+local MAYBE_EMPTY = range_flags.MAYBE_EMPTY
+
+local lpeg = require("lpeg")
+
+local call_types = {
+ CALL = "call",
+ OTHER_TOKENS = "other tokens",
+}
+
+local CALL = call_types.CALL
+local OTHER_TOKENS = call_types.OTHER_TOKENS
+
+-- Convert the content to a tree of function calls an register any issues.
+local function syntactic_analysis(pathname, content, issues, results, options) -- luacheck: ignore pathname content options
+
+ local token_types = require("explcheck-lexical-analysis").token_types
+
+ local CONTROL_SEQUENCE = token_types.CONTROL_SEQUENCE
+ local CHARACTER = token_types.CHARACTER
+
+ -- Extract function calls from TeX tokens and groupings.
+ local function get_calls(tokens, token_range, groupings)
+ local calls = {}
+ if #token_range == 0 then
+ return calls
+ end
+
+ local token_number = token_range:start()
+
+ -- Record a range of unrecognized tokens.
+ local function record_other_tokens(other_token_range)
+ local previous_call = #calls > 0 and calls[#calls] or nil
+ if previous_call == nil or previous_call[1] ~= OTHER_TOKENS then -- record a new span of other tokens between calls
+ table.insert(calls, {OTHER_TOKENS, other_token_range})
+ else -- extend the previous span of other tokens
+ assert(previous_call[1] == OTHER_TOKENS)
+ assert(previous_call[2]:stop() == other_token_range:start() - 1)
+ previous_call[2] = new_range(previous_call[2]:start(), other_token_range:stop(), INCLUSIVE, #tokens)
+ end
+ end
+
+ -- Normalize common non-expl3 commands to expl3 equivalents.
+ local function normalize_csname(csname)
+ local next_token_number = token_number + 1
+ local normalized_csname = csname
+ local ignored_token_number
+
+ if csname == "let" then -- \let
+ if token_number + 1 <= token_range:stop() then
+ if tokens[token_number + 1][1] == CONTROL_SEQUENCE then -- followed by a control sequence
+ if token_number + 2 <= token_range:stop() then
+ if tokens[token_number + 2][1] == CONTROL_SEQUENCE then -- followed by another control sequence
+ normalized_csname = "cs_set_eq:NN" -- \let \csname \csname
+ elseif tokens[token_number + 2][1] == CHARACTER then -- followed by a character
+ if tokens[token_number + 2][2] == "=" then -- that is an equal sign
+ if token_number + 3 <= token_range:stop() then
+ if tokens[token_number + 3][1] == CONTROL_SEQUENCE then -- followed by another control sequence
+ ignored_token_number = token_number + 2
+ normalized_csname = "cs_set_eq:NN" -- \let \csname = \csname
+ end
+ end
+ end
+ end
+ end
+ end
+ end
+ elseif csname == "def" or csname == "gdef" or csname == "edef" or csname == "xdef" then -- \?def
+ if token_number + 1 <= token_range:stop() then
+ if tokens[token_number + 1][1] == CONTROL_SEQUENCE then -- followed by a control sequence
+ if csname == "def" then -- \def \csname
+ normalized_csname = "cs_set:Npn"
+ elseif csname == "gdef" then -- \gdef \csname
+ normalized_csname = "cs_gset:Npn"
+ elseif csname == "edef" then -- \edef \csname
+ normalized_csname = "cs_set:Npe"
+ elseif csname == "xdef" then -- \xdef \csname
+ normalized_csname = "cs_set:Npx"
+ else
+ assert(false, csname)
+ end
+ end
+ end
+ elseif csname == "global" then -- \global
+ next_token_number = next_token_number + 1
+ assert(next_token_number == token_number + 2)
+ if token_number + 1 <= token_range:stop() then
+ if tokens[token_number + 1][1] == CONTROL_SEQUENCE then -- followed by a control sequence
+ csname = tokens[token_number + 1][2]
+ if csname == "let" then -- \global \let
+ if token_number + 2 <= token_range:stop() then
+ if tokens[token_number + 2][1] == CONTROL_SEQUENCE then -- followed by another control sequence
+ if token_number + 3 <= token_range:stop() then
+ if tokens[token_number + 3][1] == CONTROL_SEQUENCE then -- followed by another control sequence
+ normalized_csname = "cs_gset_eq:NN" -- \global \let \csname \csname
+ goto skip_decrement
+ elseif tokens[token_number + 3][1] == CHARACTER then -- followed by a character
+ if tokens[token_number + 3][2] == "=" then -- that is an equal sign
+ if token_number + 4 <= token_range:stop() then
+ if tokens[token_number + 4][1] == CONTROL_SEQUENCE then -- followed by another control sequence
+ ignored_token_number = token_number + 3
+ normalized_csname = "cs_gset_eq:NN" -- \global \let \csname = \csname
+ goto skip_decrement
+ end
+ end
+ end
+ end
+ end
+ end
+ end
+ elseif csname == "def" or csname == "gdef" or csname == "edef" or csname == "xdef" then -- \global \?def
+ if token_number + 2 <= token_range:stop() then
+ if tokens[token_number + 2][1] == CONTROL_SEQUENCE then -- followed by another control sequence
+ if csname == "def" then -- \global \def \csname
+ normalized_csname = "cs_gset:Npn"
+ elseif csname == "gdef" then -- \global \gdef \csname
+ normalized_csname = "cs_gset:Npn"
+ elseif csname == "edef" then -- \global \edef \csname
+ normalized_csname = "cs_gset:Npe"
+ elseif csname == "xdef" then -- \global \xdef \csname
+ normalized_csname = "cs_gset:Npx"
+ else
+ assert(false)
+ end
+ goto skip_decrement
+ end
+ end
+ end
+ end
+ end
+ next_token_number = next_token_number - 1
+ assert(next_token_number == token_number + 1)
+ ::skip_decrement::
+ end
+ return normalized_csname, next_token_number, ignored_token_number
+ end
+
+ while token_number <= token_range:stop() do
+ local token = tokens[token_number]
+ local token_type, payload, _, byte_range = table.unpack(token)
+ if token_type == CONTROL_SEQUENCE then -- a control sequence
+ local original_csname = payload
+ local csname, next_token_number, ignored_token_number = normalize_csname(original_csname)
+ ::retry_control_sequence::
+ local _, _, argument_specifiers = csname:find(":([^:]*)") -- try to extract a call
+ if argument_specifiers ~= nil and lpeg.match(parsers.argument_specifiers, argument_specifiers) ~= nil then
+ local arguments = {}
+ local next_token, next_token_range
+ local next_token_type, _, next_catcode, next_byte_range
+ local next_grouping, parameter_text_start_token_number
+ for argument_specifier in argument_specifiers:gmatch(".") do -- an expl3 control sequence, try to collect the arguments
+ if lpeg.match(parsers.weird_argument_specifier, argument_specifier) then
+ goto skip_other_token -- a "weird" argument specifier, skip the control sequence
+ elseif lpeg.match(parsers.do_not_use_argument_specifier, argument_specifier) then
+ goto skip_other_token -- a "do not use" argument specifier, skip the control sequence
+ end
+ ::check_token::
+ if next_token_number > token_range:stop() then -- missing argument (partial application?), skip all remaining tokens
+ if token_range:stop() == #tokens then
+ if csname ~= original_csname then -- before recording an error, retry without trying to understand non-expl3
+ csname, next_token_number, ignored_token_number = original_csname, token_number + 1, nil
+ goto retry_control_sequence
+ else
+ issues:add('e301', 'end of expl3 part within function call', byte_range)
+ end
+ end
+ record_other_tokens(new_range(token_number, token_range:stop(), INCLUSIVE, #tokens))
+ token_number = next_token_number
+ goto continue
+ end
+ next_token = tokens[next_token_number]
+ next_token_type, _, next_catcode, next_byte_range = table.unpack(next_token)
+ if ignored_token_number ~= nil and next_token_number == ignored_token_number then
+ next_token_number = next_token_number + 1
+ goto check_token
+ end
+ if lpeg.match(parsers.parameter_argument_specifier, argument_specifier) then
+ parameter_text_start_token_number = next_token_number -- a "TeX parameter" argument specifier, try to collect parameter text
+ while next_token_number <= token_range:stop() do
+ next_token = tokens[next_token_number]
+ next_token_type, _, next_catcode, next_byte_range = table.unpack(next_token)
+ if next_token_type == CHARACTER and next_catcode == 2 then -- end grouping, skip the control sequence
+ if csname ~= original_csname then -- before recording an error, retry without trying to understand non-expl3
+ csname, next_token_number, ignored_token_number = original_csname, token_number + 1, nil
+ goto retry_control_sequence
+ else
+ issues:add('e300', 'unexpected function call argument', next_byte_range)
+ goto skip_other_token
+ end
+ elseif next_token_type == CHARACTER and next_catcode == 1 then -- begin grouping, record the parameter text
+ next_token_number = next_token_number - 1
+ table.insert(arguments, new_range(parameter_text_start_token_number, next_token_number, INCLUSIVE + MAYBE_EMPTY, #tokens))
+ break
+ end
+ next_token_number = next_token_number + 1
+ end
+ if next_token_number > token_range:stop() then -- missing begin grouping (partial application?), skip all remaining tokens
+ if token_range:stop() == #tokens then
+ if csname ~= original_csname then -- before recording an error, retry without trying to understand non-expl3
+ csname, next_token_number, ignored_token_number = original_csname, token_number + 1, nil
+ goto retry_control_sequence
+ else
+ issues:add('e301', 'end of expl3 part within function call', next_byte_range)
+ end
+ end
+ record_other_tokens(new_range(token_number, token_range:stop(), INCLUSIVE, #tokens))
+ token_number = next_token_number
+ goto continue
+ end
+ elseif lpeg.match(parsers.N_type_argument_specifier, argument_specifier) then -- an N-type argument specifier
+ if next_token_type == CHARACTER and next_catcode == 1 then -- begin grouping, try to collect the balanced text
+ next_grouping = groupings[next_token_number]
+ assert(next_grouping ~= nil)
+ assert(next_grouping.start == next_token_number)
+ if next_grouping.stop == nil then -- an unclosed grouping, skip the control sequence
+ if token_range:stop() == #tokens then
+ if csname ~= original_csname then -- before recording an error, retry without trying to understand non-expl3
+ csname, next_token_number, ignored_token_number = original_csname, token_number + 1, nil
+ goto retry_control_sequence
+ else
+ issues:add('e301', 'end of expl3 part within function call', next_byte_range)
+ end
+ end
+ goto skip_other_token
+ else -- a balanced text
+ next_token_range = new_range(next_grouping.start + 1, next_grouping.stop - 1, INCLUSIVE + MAYBE_EMPTY, #tokens)
+ if #next_token_range == 1 then -- a single token, record it
+ issues:add('w303', 'braced N-type function call argument', next_byte_range)
+ table.insert(arguments, next_token_range)
+ next_token_number = next_grouping.stop
+ elseif #next_token_range == 2 and -- two tokens
+ tokens[next_token_range:start()][1] == CHARACTER and tokens[next_token_range:start()][3] == 6 and -- a parameter
+ tokens[next_token_range:stop()][1] == CHARACTER and -- followed by a digit (unrecognized parameter/replacement text?)
+ lpeg.match(parsers.decimal_digit, tokens[next_token_range:stop()][2]) then
+ record_other_tokens(new_range(token_number, next_grouping.stop, INCLUSIVE, #tokens))
+ token_number = next_grouping.stop + 1
+ goto continue
+ else -- no token / more than one token, skip the control sequence
+ if csname ~= original_csname then -- before recording an error, retry without trying to understand non-expl3
+ csname, next_token_number, ignored_token_number = original_csname, token_number + 1, nil
+ goto retry_control_sequence
+ else
+ issues:add('e300', 'unexpected function call argument', next_byte_range)
+ goto skip_other_token
+ end
+ end
+ end
+ elseif next_token_type == CHARACTER and next_catcode == 2 then -- end grouping (partial application?), skip all tokens
+ record_other_tokens(new_range(token_number, next_token_number, EXCLUSIVE, #tokens))
+ token_number = next_token_number
+ goto continue
+ else
+ if next_token_type == CHARACTER and next_catcode == 6 then -- a parameter
+ if next_token_number + 1 <= token_range:stop() then -- followed by one other token
+ if tokens[next_token_number + 1][1] == CHARACTER and -- that is a digit (unrecognized parameter/replacement text?)
+ lpeg.match(parsers.decimal_digit, tokens[next_token_number + 1][2]) then -- skip all tokens
+ record_other_tokens(new_range(token_number, next_token_number + 1, INCLUSIVE, #tokens))
+ token_number = next_token_number + 2
+ goto continue
+ end
+ end
+ end
+ -- an N-type argument, record it
+ table.insert(arguments, new_range(next_token_number, next_token_number, INCLUSIVE, #tokens))
+ end
+ elseif lpeg.match(parsers.n_type_argument_specifier, argument_specifier) then -- an n-type argument specifier
+ if next_token_type == CHARACTER and next_catcode == 1 then -- begin grouping, try to collect the balanced text
+ next_grouping = groupings[next_token_number]
+ assert(next_grouping ~= nil)
+ assert(next_grouping.start == next_token_number)
+ if next_grouping.stop == nil then -- an unclosed grouping, skip the control sequence
+ if token_range:stop() == #tokens then
+ if csname ~= original_csname then -- before recording an error, retry without trying to understand non-expl3
+ csname, next_token_number, ignored_token_number = original_csname, token_number + 1, nil
+ goto retry_control_sequence
+ else
+ issues:add('e301', 'end of expl3 part within function call', next_byte_range)
+ end
+ end
+ goto skip_other_token
+ else -- a balanced text, record it
+ table.insert(arguments, new_range(next_grouping.start + 1, next_grouping.stop - 1, INCLUSIVE + MAYBE_EMPTY, #tokens))
+ next_token_number = next_grouping.stop
+ end
+ elseif next_token_type == CHARACTER and next_catcode == 2 then -- end grouping (partial application?), skip all tokens
+ record_other_tokens(new_range(token_number, next_token_number, EXCLUSIVE, #tokens))
+ token_number = next_token_number
+ goto continue
+ else -- not begin grouping
+ if next_token_type == CHARACTER and next_catcode == 6 then -- a parameter
+ if next_token_number + 1 <= token_range:stop() then -- followed by one other token
+ if tokens[next_token_number + 1][1] == CHARACTER and -- that is a digit (unrecognized parameter/replacement text?)
+ lpeg.match(parsers.decimal_digit, tokens[next_token_number + 1][2]) then -- skip all tokens
+ record_other_tokens(new_range(token_number, next_token_number + 1, INCLUSIVE, #tokens))
+ token_number = next_token_number + 2
+ goto continue
+ end
+ end
+ end
+ -- an unbraced n-type argument, record it
+ issues:add('w302', 'unbraced n-type function call argument', next_byte_range)
+ table.insert(arguments, new_range(next_token_number, next_token_number, INCLUSIVE, #tokens))
+ end
+ else
+ error('Unexpected argument specifier "' .. argument_specifier .. '"')
+ end
+ next_token_number = next_token_number + 1
+ end
+ table.insert(calls, {CALL, new_range(token_number, next_token_number, EXCLUSIVE, #tokens), csname, arguments})
+ token_number = next_token_number
+ goto continue
+ else -- a non-expl3 control sequence, skip it
+ goto skip_other_token
+ end
+ elseif token_type == CHARACTER then -- an ordinary character
+ if payload == "=" then -- an equal sign
+ if token_number + 2 <= token_range:stop() then -- followed by two other tokens
+ if tokens[token_number + 1][1] == CONTROL_SEQUENCE then -- the first being a control sequence
+ if tokens[token_number + 2][1] == CHARACTER and tokens[token_number + 2][2] == "," then -- and the second being a comma
+ -- (probably l3keys definition?), skip all three tokens
+ record_other_tokens(new_range(token_number, token_number + 2, INCLUSIVE, #tokens))
+ token_number = token_number + 3
+ goto continue
+ end
+ end
+ end
+ end
+ -- an ordinary character, skip it
+ goto skip_other_token
+ else
+ error('Unexpected token type "' .. token_type .. '"')
+ end
+ ::skip_other_token::
+ record_other_tokens(new_range(token_number, token_number, INCLUSIVE, #tokens))
+ token_number = token_number + 1
+ ::continue::
+ end
+ return calls
+ end
+
+ local calls = {}
+ for part_number, part_tokens in ipairs(results.tokens) do
+ local part_groupings = results.groupings[part_number]
+ local part_token_range = new_range(1, #part_tokens, INCLUSIVE, #part_tokens)
+ local part_calls = get_calls(part_tokens, part_token_range, part_groupings)
+ table.insert(calls, part_calls)
+ end
+
+ -- Store the intermediate results of the analysis.
+ results.calls = calls
+end
+
+return {
+ process = syntactic_analysis,
+ call_types = call_types
+}
Property changes on: trunk/Master/texmf-dist/scripts/expltools/explcheck-syntactic-analysis.lua
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
More information about the tex-live-commits
mailing list.