Re: Installation on windows (fwd)

[Sheldon Gill]

Richard Frith-Macdonald wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 2005-03-16 08:13:42 +0000 Sheldon Gill <address@hidden> wrote:
>
> > I disagree that Unicode to the file system isn't part of OpenStep. 
> > Mostly,
> > the OSAPI uses NSStrings passed to NSFileManager or NSFileHandle. It is
> > Unicode clean as far as possible.
>
>
> Hmm ... While I guess I can see why you might think that, I don't really 
> see
> that as anything to do with the filesystem. OpenStep uses NSString objects
> internally, so inevitably the API to those classes passes NSString objects
> ... to me that implies nothing about the underlying filesystem support.
> Plainly having a unicode filesystem is a good thing, and I feel that 
> support
> for unicode filesystems is a natural extension, but the support of unicode
> strings doesn't mean that unicode filesystems were supported.
>
> > There is even a specific method to transform from an NSString to a path
> > string which the OS system routines can handle to provide compatibility
> > between the OpenStep fully unicode API and the underlying OS 
> > requirements.
>
>
> Specifically, that converts FROM and NSString (which may be thought of as
> unicode. even if it is oftern stored as 8byte characters) TO an 8-bit per
> character filesystem representation.  This says that the internal format 
> may
> be 16bit unicode, but the external one isn't ... OpenStep anticipates that
> the filesystem will be using CStrings rather than unicode.

My take on that goes a little differently. It recognises that file
system calls on the underlying OS probably require a CString. Exactly
what type of CString can differ. On some platforms you'll need to
convert to a specific codepage. On others it'd be preferred UTF-8.
Having one call to convert makes it easy.

I don't think it is mean as a convert to/from "internal" format
mechanism. Its a convenient support mechanism when the underlying OS
uses CString parameters.

In the case of Windows, it can handle UTF-16 directly. In this case
there are no code page or other encoding transforms that need to take
place so you can always pass a UTF-16 string.

> > There are string methods to help manipulating strings which represent 
> > paths
> > and many of them are rather POSIX focused but that is a different thing.
> > Mostly they don't care path separator etc and are done in a sufficiently
> > abstract way so it doesn't matter.
>
> Yes ... I think we should try to keep things as abstract as possible and
> OpenStep goes a long way towards this.

Agreed. Nice and abstract so mostly applications don't need to care too
much.

> > Umm.. no. I really don't want to break compatibility. In fact, I want to
> > enhance it and make it easier. That doesn't mean, though, that I advocate
> > doing everything to try and support existing code as written where it 
> > makes
> > sense to have a better approach.
> >
> > I'd actually say that the current changes do more to break 
> > compatibility in
> > porting code than my goal to do things cleanly "under the hood".
> >
> > The current situation forces localToOpenStep and openStepToLocal. 
> > These are
> > extensions to the API and complicate things. In my view, they are
> > unnecessary. We can do better without them.
>
> I'm not really happy with them ... but I'm not sure there is a better
> approach, and they seem to make things simpler than any alternative.  A 
> more
> efficient api might take 16bit character strings directly ... but then what
> about 32bit unicode strings etc etc.
> This API may require two stages (one to convert between external data 
> format
> and NSString,  and one to convert between external path format and internal
> path format), but it keeps additional methods to a minimum given that the

This is the real point between us. I am sure there is a better approach 
and that is to do without them.

> > Under MacOS-X, *nix and even YellowBox I could:
> >    * get an NSString from an environment variable via NSProcessInfo
> >    * Use that string as a path to NSFileManager
>
> I believe that's also the case in GNUstep (unless recently broken).  The
> interesting question is to what extent path manipulation can be 
> performed on
> such a string rather than whether it can be used literally.  In GNUstep
> that's minimal.
> However, while it way work, I've not seen it recommended anywhere..
>
> > under gnustep-CVS I have to add an intermediate step:
> >    * get an NSString from an environment variable via NSProcessInfo
> >    * openStepToLocal the string
>
>
> You mean openStepPathFromLocal: ?

Yes, I do. I often shorthand method/function names when I think its 
reasonably unambiguous. Must be the Aussie influence ;)

The issue I have is this:

You need to call the local/OpenStep conversion routines on Win32. You 
don't on other platforms. So it is actually making porting *more* difficult.

These routines add semantic complexity. You have to keep track of when a 
string is in "internal" format and when it is in "external" format.

> >    * Use that string as a path to NSFileManager and friends
> >
> > > OpenStep was designed for/from a unix worldview and has an API 
> > > reflecting this.
> > >
> > > As such it's my opinion that failing to transform strings (for 
> > > instance to
> > > use '/' as the path separator internally) would break things and 
> > > would force
> > > users (by which I mean people writing applications) to write more 
> > > code with
> > > larger overheads because they would end up writing code to allow for
> > > differences in path semantics at the application level rather than the
> > > library level.
> >
> >
> > Don't you mean "...use '\' as the path separator..."?
>
>
> Yes, sorry.

No worries, mate.

I'm interested, though. Under what use cases which don't involve 
archiving paths and moving them between platforms?  I can't think of an 
internal breakage while remaining on the one platform.

As I noted before, I think that differences in path semantics would 
force you to code at the application level when the situation dictates.

Basically when path semantics are sufficiently between the platforms it 
is going to be an issue. I have the view that trying to do it at the 
library level will ultimately be more problematic.

The reason is the implication and impact of the semantic differences is 
difficult to foresee and the best response will probably vary from 
application to application. Having an emulation layer in the library 
would simply make it more complex and annoying to handle these cases.

I think we can handle the general case cleanly now without needing 
"internal/external" transformation.

> > If you consider my example above the current CVS solution adds code and
> > semantic complexity at the application level. Precisely what you are
> > advocating shouldn't happen.
>
>
> Not really ...
>
> 1. You are picking a special (though not unusual) case and you have to
> consider what the overall behavior is.
> 2. GNUstep is providing a helper method to make it easier, rather than
> leaving the app write to code their own.

How is this helper method making it easier?

> OPENSTEP/MacOS-x say you should use
> - -stringWithFileSystemRepresentation:length: and
> - -fileSystemRepresentationWithPath: when importing/exporting strings which
> are expected to be paths.

I don't read it that way. I read those functions as a convenient way to 
move from an NSString to an 8-bit encoding suitable for handing to OS 
file system calls. Its an important difference. There is a need for it 
to be this way, rather than just CString because many platforms required 
encoding to a specific code page.

In particular, I point to Win95 which was a supported OpenStep platform. 
 Many *nix systems were also tied to a specific 8-bit encoding.

I also point to the telling line from Apple's Foundation documentation:

  Raises an NSCharacterConversionException if the receiver can’t be
  represented in the file system’s encoding.

Its about encoding, not internal/external transformation.

> I think the MacOS-X/OPENSTEP way would be to call
> path = [fileManager stringWithFileSystemRepresentation: [path cString]
> length: [path cStringLength]];
> to convert a string from the environment into a path.
> Certainly the documentation says these methods are to convert to/from
> CStrings used by native system calls, and if the paths were held internally
> by openstep in the same format as the external paths, then you could just
> use -cString and +stringWithCString: to perform such conversions.

No, as I said above simply using -cString wasn't a sufficient solution. 
Under MacOS-X you could probably use -cString and get away with it 
because it'd be UTF-8 which the FS handles.

Anyway, my example doesn't require using any native system calls. It is 
entirely at the OpenStep API level. For MacOS-X I don't think you would 
call stringWithFileSystemRepresentation:length: for that code fragment.

> These methods do not work on windows with unicode filesystems ... as the
> designers of  OPENSTEP  didn't anticipate unicode filesystems arriving
> on windows and needing a whole new API working with 16bit characters rather
> than CStrings.  That's really the only reason that -openStepPathFromLocal:
> and -localFromOpenStepPath: were introduced.

Yes. My argument is that you don't need a whole new API for working with 
16bit characters. You've got all you need already.

The local/OpenStep methods aren't needed for working with the file 
system at all. Their current implementation and use is actually to 
"munge the string to/from a POSIX path" which is a completely different 
thing to "making the string suitable for use with the OS system calls".

They are a POSIX emulation layer. Precisely the thing we're trying to 
get away from by using MinGW.

> Normally, when interacting with users, I think the expectation is that 
> paths
> will be selected by browsing the filesystem etc, so the open/save pannel
> returns a valid path string already in internal format.  This means that
> conversions between internal and external format paths outside the base
> library are rarely necessary.
>
> Using a standardised format internally means that an application can
> effectively manipulate paths for multiple filesystems on multiple machines
> and can do so in a consistent manner.

Yes. I feel effective path manipulation can be done with the current 
string methods. Possibly with the addition of a category or two to 
handle certain things which are semantically very different between the 
two platforms.

I don't think that we need to convert to an "internal" format.

> > In the common situation that I get a path string from a user or other
> > external input source (like a file with references) all gnustep 
> > compatible
> > portable code has to call localToOpen and vice versa. That is adding
> > complexity and code at the application level. We don't need to do it 
> > and I
> > suggest we don't.
> >
> > > So, while it would indeed be 'safe, sane, future compatible, least code,
> > > smallest overheads' internally withing the base library, it would be the
> > > exact opposite for users of the library who would have to write more 
> > > code to
> > > make their apps portable between unix/windows, would have to 
> > > understand the
> > > assumptions of both filesystems, and would need to know about new
> > > filesystems in future... this is not the sort of programming 
> > > environment we
> > > want to provide.
> >
> >
> > A certain level of knowledge about the assumptions and layout of both
> > filesystems is, IMHO, unavoidable. We can, though, provide a good way of
> > dealing with the differences and moving from one to another with minimal
> > impact.
>
>
> I didn't mean to imply that we could avoid having to know about 
> filesystems,
> just that the extent of such knowledge (and certainly the impact on
> application code) should be minimised.

We agree on this. As you're well aware by now, I'm advocating further 
minimisation :)

I know that you have spent quite a bit of time and effort on the Unicode 
support issue and the local/OpenStep conversion methods. I do appreciate 
that.

Recall that these methods, though, we suggested as a better way than 
Roland's initial patch which was to add to NSString:

  - (unichar *) wFileSystemRepresentationForPath:

My point is that this was approaching the problem from the wrong angle. 
He wanted to use native calls and could have done that simply with:

  [path getCharacters: buffer]
  OSCall(buffer);

> > By way of example, under *nix it'd be perfectly acceptable for my 
> > application
> > to create a document using NSHomeDirectory(). The preferred place under
> > Windows is "My Documents". Extensions to NSSearchPath() will help us cope
> > with this sort of situation but the underlying semantic variations 
> > remain.
>
> And the preferred place under MacOS-X is 'Documents' (and I guess other
> conventions exist within other environment such as gnome or kde).

Yup. Starts getting complicated, doesn't it?

> Do you mean extending NSSearchPathForDirectoriesInDomains() to add 
> something
> to return the preferred directory for storing documents (rather than
> resource files etc) on the current system, then encouraging developers to
> use it for all documents?

Yes, I do.

Or rather, I did. For Cocoa compatibility reasons I'm beginning to think 
it should be done slightly differently. Discussion for a different thread...

> That sounds a reasonable idea.  Though I don't like using that API myself
> ... I think Apple would have done better to make it an NSFileManager method
> instead ... so people could alter behavior in subclasses or categories.
> Generally I don't approve of functions in the public API except for really
> low level operations where extreme performance is critical.

I have the same view about the API. Thing is, I think that this *is* a 
fairly low level operation which should be very fast. We want developers 
to use it. We want them to take the time to first obtain a location, 
then do the file operations. The faster it is, the better for all.

In this case I also don't think that altering the behaviour is a good 
thing. The routine really is there to serve up the FHS dogma for the 
platform so the application can be quite agnostic about it all. Being 
able to change that isn't a good thing IMHO.

If you're looking for something like extensible folder routing, 
NSSearch() isn't the place for it. That's best done at the desktop level 
and I've design to get that going, too.

> > In many cases the code can be written cleanly portable. Some cases will
> > require knowing which OS you're on and behaving accordingly.
> >
> > Developers need to be aware when porting. We can only achieve partial OS
> > isolation. We can try to reduce the pain and provide nice docs to help 
> > people
> > along, though.
>
> Yes.

See, we agree on quite a lot really.


Regards,
Sheldon