In this concatenation form, a line vector may be written with coma or space separated values: octave> [1 2] ans =
1 2
octave> [1, 2] ans =
1 2
Here I suppose the parser splits the _expression_ between the braquets using space (and ;) separator, and sees an invalid Octave _expression_ : (1, 1). You have to remove the space between "rand" and "(1, 1)" or add paranthesis if you want to keep the separation between function and arguments : [1; (rand (1, 1))]