help-smalltalk
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Help-smalltalk] [PATCH] stinst: Fix the scanner for compile time consta


From: Holger Hans Peter Freyther
Subject: [Help-smalltalk] [PATCH] stinst: Fix the scanner for compile time constants in literal arrays
Date: Sat, 21 Dec 2013 18:14:02 +0100

The scanner didn't tokenize ##(1/2)) correctly. After scanning the
last $) has not been processed yet. Count the number of optimized
tokens and consume an equal amount of $). >>#scanLiteralArrayParts
will only parse the $) when there is an unclosed compile time
constant.

2013-12-21  Holger Hans Peter Freyther  <address@hidden>

        * RBScannerTests.st: Add new tests.
        * RBParser.st: Change parsing of optimized tokens.
---
 packages/stinst/parser/ChangeLog         |  5 ++
 packages/stinst/parser/Makefile.frag     |  2 +-
 packages/stinst/parser/RBParser.st       | 12 +++--
 packages/stinst/parser/RBScannerTests.st | 79 ++++++++++++++++++++++++++++++++
 packages/stinst/parser/package.xml       |  2 +
 5 files changed, 96 insertions(+), 4 deletions(-)
 create mode 100644 packages/stinst/parser/RBScannerTests.st

diff --git a/packages/stinst/parser/ChangeLog b/packages/stinst/parser/ChangeLog
index 2bd7eeb..d32a0cc 100644
--- a/packages/stinst/parser/ChangeLog
+++ b/packages/stinst/parser/ChangeLog
@@ -1,3 +1,8 @@
+2013-12-21  Holger Hans Peter Freyther  <address@hidden>
+
+       * RBScannerTests.st: Add new tests.
+       * RBParser.st: Change parsing of optimized tokens.
+
 2013-12-19  Holger Hans Peter Freyther  <address@hidden>
 
        * STCompiler.st: Rename STCompiler>>#addPool: to
diff --git a/packages/stinst/parser/Makefile.frag 
b/packages/stinst/parser/Makefile.frag
index b70055b..f86626f 100644
--- a/packages/stinst/parser/Makefile.frag
+++ b/packages/stinst/parser/Makefile.frag
@@ -1,5 +1,5 @@
 Parser_FILES = \
-packages/stinst/parser/ChangeLog 
packages/stinst/parser/DebugInformationTests.st 
packages/stinst/parser/Exporter.st packages/stinst/parser/Extensions.st 
packages/stinst/parser/GSTParser.st packages/stinst/parser/GSTParserTests.st 
packages/stinst/parser/NewSyntaxExporter.st 
packages/stinst/parser/OldSyntaxExporter.st 
packages/stinst/parser/OrderedSet.st 
packages/stinst/parser/ParseTreeSearcher.st 
packages/stinst/parser/PoolResolutionTests.st 
packages/stinst/parser/RBFormatter.st packages/stinst/parser/RBParseNodes.st 
packages/stinst/parser/RBParser.st packages/stinst/parser/RBToken.st 
packages/stinst/parser/RewriteTests.st packages/stinst/parser/SIFParser.st 
packages/stinst/parser/SqueakExporter.st packages/stinst/parser/SqueakParser.st 
packages/stinst/parser/STCompiler.st packages/stinst/parser/STCompilerTests.st 
packages/stinst/parser/STCompLit.st packages/stinst/parser/STDecompiler.st 
packages/stinst/parser/STEvaluationDriver.st 
packages/stinst/parser/STFileParser.st packages/stin
 st/parser/STLoaderObjs.st packages/stinst/parser/STLoaderObjsTests.st 
packages/stinst/parser/STLoader.st packages/stinst/parser/STSymTable.st 
+packages/stinst/parser/ChangeLog 
packages/stinst/parser/DebugInformationTests.st 
packages/stinst/parser/Exporter.st packages/stinst/parser/Extensions.st 
packages/stinst/parser/GSTParser.st packages/stinst/parser/GSTParserTests.st 
packages/stinst/parser/NewSyntaxExporter.st 
packages/stinst/parser/OldSyntaxExporter.st 
packages/stinst/parser/OrderedSet.st 
packages/stinst/parser/ParseTreeSearcher.st 
packages/stinst/parser/PoolResolutionTests.st 
packages/stinst/parser/RBFormatter.st packages/stinst/parser/RBParseNodes.st 
packages/stinst/parser/RBParser.st packages/stinst/parser/RBScannerTests.st 
packages/stinst/parser/RBToken.st packages/stinst/parser/RewriteTests.st 
packages/stinst/parser/SIFParser.st packages/stinst/parser/SqueakExporter.st 
packages/stinst/parser/SqueakParser.st packages/stinst/parser/STCompiler.st 
packages/stinst/parser/STCompilerTests.st packages/stinst/parser/STCompLit.st 
packages/stinst/parser/STDecompiler.st 
packages/stinst/parser/STEvaluationDriver.st packages/st
 inst/parser/STFileParser.st packages/stinst/parser/STLoaderObjs.st 
packages/stinst/parser/STLoaderObjsTests.st packages/stinst/parser/STLoader.st 
packages/stinst/parser/STSymTable.st 
 $(Parser_FILES):
 $(srcdir)/packages/stinst/parser/stamp-classes: $(Parser_FILES)
        touch $(srcdir)/packages/stinst/parser/stamp-classes
diff --git a/packages/stinst/parser/RBParser.st 
b/packages/stinst/parser/RBParser.st
index b953762..77e4832 100644
--- a/packages/stinst/parser/RBParser.st
+++ b/packages/stinst/parser/RBParser.st
@@ -1215,16 +1215,20 @@ Stream subclass: RBScanner [
 
     scanLiteralArray [
        <category: 'private-scanning'>
-       | arrayStream start |
+       | arrayStream start optimizedNodes |
        arrayStream := WriteStream on: (Array new: 10).
        self step.
        start := tokenStart.
+       optimizedNodes := 0.
        
        [self stripSeparators.
        tokenStart := stream position.
-       currentCharacter == $)] 
+       currentCharacter == $) and: [optimizedNodes = 0]]
                whileFalse: 
-                   [arrayStream nextPut: self scanLiteralArrayParts.
+                   [ |res|
+                   res := arrayStream nextPut: self scanLiteralArrayParts.
+                   res isOptimized ifTrue: [optimizedNodes := optimizedNodes + 
1].
+                   res isSpecial ifTrue: [optimizedNodes := optimizedNodes - 
1].
                    buffer reset].
        self step.
        ^RBLiteralToken 
@@ -1254,6 +1258,8 @@ Stream subclass: RBScanner [
        currentCharacter == $$ ifTrue: [^self scanLiteralCharacter].
        currentCharacter == $( ifTrue: [^self scanLiteralArray].
        currentCharacter == $[ ifTrue: [^self scanByteArray].
+       "In the case of RBOptimizedToken we scan for $) as well"
+       currentCharacter == $) ifTrue: [^self scanSpecialCharacter].
        ^self scannerError: 'Unknown character in literal array'
     ]
 
diff --git a/packages/stinst/parser/RBScannerTests.st 
b/packages/stinst/parser/RBScannerTests.st
new file mode 100644
index 0000000..ba84813
--- /dev/null
+++ b/packages/stinst/parser/RBScannerTests.st
@@ -0,0 +1,79 @@
+"======================================================================
+|
+|   Smalltalk in Smalltalk RBScanner tests
+|
+|
+ ======================================================================"
+
+"======================================================================
+|
+| Copyright 2013 Free Software Foundation, Inc.
+| Written by Holger Hans Peter Freyther.
+|
+| This file is part of GNU Smalltalk.
+|
+| GNU Smalltalk is free software; you can redistribute it and/or modify it
+| under the terms of the GNU General Public License as published by the Free
+| Software Foundation; either version 2, or (at your option) any later version.
+| 
+| GNU Smalltalk is distributed in the hope that it will be useful, but WITHOUT
+| ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+| FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+| details.
+| 
+| You should have received a copy of the GNU General Public License along with
+| GNU Smalltalk; see the file COPYING.  If not, write to the Free Software
+| Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  
+|
+ ======================================================================"
+
+
+TestCase subclass: TestRBScanner [
+    <comment: 'I do test the RBScanner/Tokenizer.'>
+
+    testEmbeddedCompiletimeConstant [
+        | scanner token value |
+        scanner := RBScanner on: '#(##(1))' readStream.
+
+        "Token is the literal"
+        token := scanner next.
+        self assert: token isLiteral.
+
+        "And the literal should be RBOptimizedToken RBNumberLiteralToken 
RBSpecialCharacterToken"
+        value := token value.
+        value inspect.
+        self assert: value size equals: 3.
+        self assert: value first isOptimized.
+        self assert: value second isLiteral.
+        self assert: value second value equals: 1.
+        self assert: value third isSpecial.
+        self assert: value third value equals: $).
+
+        "And we are at the end"
+        scanner atEnd printNl.
+        self assert: scanner atEnd.
+    ]
+
+    testDirectCompiletimeConstant [
+        | scanner token |
+        scanner := RBScanner on: '##(1)' readStream.
+
+        "Token is for optimized code"
+        token := scanner next.
+        self assert: token isOptimized.
+
+        "Token is the '1'"
+        token := scanner next.
+        self assert: token isLiteral.
+        self assert: token value equals: 1.
+
+        "Token is the ')'"
+        token := scanner next.
+        self assert: token isSpecial.
+        self assert: token value equals: $).
+
+        "And we are at the end"
+        self assert: scanner atEnd.
+    ]
+]
+
diff --git a/packages/stinst/parser/package.xml 
b/packages/stinst/parser/package.xml
index 6102af7..1bbe77c 100644
--- a/packages/stinst/parser/package.xml
+++ b/packages/stinst/parser/package.xml
@@ -38,12 +38,14 @@
    <sunit>STInST.Tests.TestSTLoaderObjects</sunit>
    <sunit>STInST.Tests.TestGSTParser</sunit>
    <sunit>STInST.Tests.TestCompiler</sunit>
+   <sunit>STInST.Tests.TestRBScanner</sunit>
    <filein>RewriteTests.st</filein>
    <filein>PoolResolutionTests.st</filein>
    <filein>DebugInformationTests.st</filein>
    <filein>STLoaderObjsTests.st</filein>
    <filein>GSTParserTests.st</filein>
    <filein>STCompilerTests.st</filein>
+   <filein>RBScannerTests.st</filein>
   </test>
 
   <file>ChangeLog</file>
-- 
1.8.5.1




reply via email to

[Prev in Thread] Current Thread [Next in Thread]