There are multiple problems.
There are errors in the code. You seem to mix up X and Y variables in plots and fits.
You read data as integers(?) -- why? You want doubles.
Octave (and matlab) is primarily an interactive programming/developing environment.
When your program crashes you still can inspect the variables left and see if the values
of those variables give you some hints.
You say "Using `;` instead of `,` doesn't work either", but the error is different!
It should have given you a clue.
I found in general the curve fitting in "optim" package are more finicky than they should be.
Out of those "leasqr" seems to work better for me than "nonlin_curvefit".
The parameters in the fit function are usually not the same parameters you would use in
human definition of the function. Usually you want to make them be more similar to each other.
You should avoid divisions if possible.
In you case the function becomes:
exp(a*t)*(b*sin(w*t)+c*cos(w*t))
(a,b,c,w -- fit parameters; w = 2*pi*frequency)
It looks to me it fits better if you add a small offset as well.
Having said that, attached are a modified script and the output it produced for me.
Dmitri.