Re: [Help-smalltalk] RBScanner and compile time constants

[Holger Hans Peter Freyther]

On Fri, Dec 20, 2013 at 11:11:16PM +0100, Paolo Bonzini wrote:
> Il 20/12/2013 22:21, Holger Hans Peter Freyther ha scritto:
> > Now my question is if the optimized token should recursively scan the
> > literals? Or should the RBScanner count $( and $) to know that it needs
> > to read one more item?
> 
> Keeping track of the number of parentheses, and exiting scanLiteralArray
> when it matches the count at entry, sounds like the easiest solution.
> 
> Only #scanLiteralArray and #scanSpecialCharacter deal with parentheses.


Before reading your mail I came up with that. I can change it to add
counting of currentCharacter==$( and currentCharacter==$) in RBScanner>>#step
instead?

With the below diff I can tokenize/parse #(##(1/2)) the only issue is that
compiling it doesn't work...

Works:
        res := STEvaluationDriver new
            parseSmalltalkStream: '##(1/2)' readStream
            with: GSTFileInParser.
        self assert: res equals: 1/2.

Doesn't:

        res := STEvaluationDriver new
            parseSmalltalkStream: '#(##(1/2))' readStream
            with: GSTFileInParser.
        res inspect.

STInST.RBOptimizedToken(Object)>>doesNotUnderstand: #realValue 
(SysExcept.st:1408)
optimized [] in STInST.RBLiteralToken>>realValue (RBToken.st:227)
Array(ArrayedCollection)>>collect: (ArrayColl.st:307)
STInST.RBLiteralToken>>realValue (RBToken.st:227)
STInST.RBLiteralNode>>value (RBParseNodes.st:2616)
STInST.STCompiler>>acceptLiteralNode: (STCompiler.st:626)
STInST.RBLiteralNode>>acceptVisitor: (RBParseNodes.st:2654)
STInST.STCompiler>>acceptReturnNode: (STCompiler.st:896)
STInST.RBReturnNode>>acceptVisitor: (RBParseNodes.st:1594)
optimized [] in STInST.STCompiler>>compileStatements: (STCompiler.st:570)
Array(SequenceableCollection)>>keysAndValuesDo: (SeqCollect.st:886)
STInST.STCompiler>>compileStatements: (STCompiler.st:571)
STInST.STCompiler>>acceptSequenceNode: (STCompiler.st:447)
STInST.RBSequenceNode>>acceptVisitor: (RBParseNodes.st:981)
STInST.STCompiler(STInST.RBProgramNodeVisitor)>>visitNode: (RBParseNodes.st:49)
STInST.STCompiler class>>compile:asMethodOf:classified:parser:environment: 
(STCompiler.st:161)
STInST.STEvaluationDriver>>evaluate: (STEvaluationDriver.st:209)




diff --git a/packages/stinst/parser/RBParser.st 
b/packages/stinst/parser/RBParser.st
index b953762..3314ff9 100644
--- a/packages/stinst/parser/RBParser.st
+++ b/packages/stinst/parser/RBParser.st
@@ -1215,16 +1215,20 @@ Stream subclass: RBScanner [
 
     scanLiteralArray [
        <category: 'private-scanning'>
-       | arrayStream start |
+       | arrayStream start optimizedNodes |
        arrayStream := WriteStream on: (Array new: 10).
        self step.
        start := tokenStart.
+       optimizedNodes := 0.
        
        [self stripSeparators.
        tokenStart := stream position.
-       currentCharacter == $)] 
+       currentCharacter == $) and: [optimizedNodes = 0]] 
                whileFalse: 
-                   [arrayStream nextPut: self scanLiteralArrayParts.
+                   [ |res|
+                   res := arrayStream nextPut: self scanLiteralArrayParts.
+                   res isOptimized ifTrue: [optimizedNodes := optimizedNodes + 
1].
+                   res isSpecial ifTrue: [optimizedNodes := optimizedNodes - 
1].
                    buffer reset].
        self step.
        ^RBLiteralToken 
@@ -1254,6 +1258,8 @@ Stream subclass: RBScanner [
        currentCharacter == $$ ifTrue: [^self scanLiteralCharacter].
        currentCharacter == $( ifTrue: [^self scanLiteralArray].
        currentCharacter == $[ ifTrue: [^self scanByteArray].
+       "In the case of RBOptimizedToken we scan for $) as well"
+       currentCharacter == $) ifTrue: [^self scanSpecialCharacter].
        ^self scannerError: 'Unknown character in literal array'
     ]