file names, directories and case sensitivity

Discussion of all aspects of the game engine, including development of new and existing features.

Moderator: Forum Moderators

User avatar
norbert
Posts: 368
Joined: June 14th, 2009, 6:57 pm
Location: The Netherlands

file names, directories and case sensitivity

Post by norbert »

I don't know if Windows uses case sensitivity for file names and directories, but it's annoying that quite frequently users upload packages to the add-on server that simply don't work and require me to rename file names and/or directories because their names don't match those used in the code. This really happens quite frequently, I'm no longer surprised to see "Error" messages when I select scenarios or campaigns. Can the devs make sure those users also get an error message?

[Edit: A similar problem is that Windows users frequently type the path separator characters for, for example, web icons with backslashes instead of slashes (which gives them an icons/missing-image.png).]
User avatar
zookeeper
WML Wizard
Posts: 9742
Joined: September 11th, 2004, 10:40 pm
Location: Finland

Re: file names, directories and case sensitivity

Post by zookeeper »

Well, we delete broken add-ons from the server from time to time anyway, so if you find a broken one you can just PM the author and/or ask on #wesnoth-dev for someone to delete it.
User avatar
norbert
Posts: 368
Joined: June 14th, 2009, 6:57 pm
Location: The Netherlands

Re: file names, directories and case sensitivity

Post by norbert »

zookeeper wrote:Well, we delete broken add-ons from the server from time to time anyway
Okay, but this means work for humans and possibly throwing away good content. If the application would use case sensitivity on all platforms, this would help prevent broken add-ons being uploaded and give the users more (possibly good) content. An add-on developer might even give up on his/her work when nobody takes the time to inform the dev (why) the add-ons isn't working and it's not being used - or because people complain about errors that cannot be replicated on certain platforms.
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: file names, directories and case sensitivity

Post by silene »

norbert wrote:If the application would use case sensitivity on all platforms,
Please enlighten us. I for one have no clue on how to force Windows to forget about case insensivity. Even Cygwin can't do it, unless you enable case sensitivity system-wide. (There is a registry setting for that.) Breaking every other Windows applications just for the sake of Wesnoth seems a bit overkill.
User avatar
norbert
Posts: 368
Joined: June 14th, 2009, 6:57 pm
Location: The Netherlands

Re: file names, directories and case sensitivity

Post by norbert »

silene wrote:Please enlighten us.
I can't, if the problem is Windows...
norbert wrote:I don't know if Windows uses case sensitivity for file names and directories, but it's annoying that [...]
I guess no solution for this problem for now.
User avatar
Sapient
Inactive Developer
Posts: 4453
Joined: November 26th, 2005, 7:41 am
Contact:

Re: file names, directories and case sensitivity

Post by Sapient »

Or you could "fix" the Linux version to be case insensitive. ;)
http://www.wesnoth.org/wiki/User:Sapient... "Looks like your skills saved us again. Uh, well at least, they saved Soarin's apple pie."
tsr
Posts: 790
Joined: May 24th, 2006, 1:05 pm

Re: file names, directories and case sensitivity

Post by tsr »

Hm, couldn't you force it from wesnoths side anyway?

When reading WML-paths require that path and file names are in lowercase and using forward slashes as separators?

Raise an error if either (or both) of the following occur:
- the wml-preprocessor encounters a path like "~/add-ons/Blaha/UtIls.cfg"
- the wml-preprocessor checks the actual path and file names for uppercase characters and finds one

Whit an instructive error-message in the form of something like:
For compatability reasons (across several plattforms) Wesnoth has decided that:
- All files and directories must only use lowercase letters, numbers and underscores in their names.
- All references to files must only use lowercase letters, numbers and underscores in their values.
- All references to paths must use the forward slash '/' as the directory-separator

Sure, it will make the preprocessor slower, but it might be worth it...

Of course I have no idea how wesnoth reads files and/or file-references and if it is even possible to get the real (I mean considering case) name of a file in windows.

Another option would be to translate all names and references when uploading to the add-ons server, but that might be confusing for UMC-developers and would still allow breakage in un-uploaded content (like forum previews).

/tsr
Momar
Posts: 4
Joined: June 1st, 2010, 9:20 am
Location: Free State of Madrid, Dicks Kingdom (Spain)

Re: file names, directories and case sensitivity

Post by Momar »

I didn't program C for a very long time (I mean "seriously") but may be it can use "findfirst" functions. [acronym="if I remember correctly]IIRC[/acronym], the "findfirst" structure stores the actual name of the file in its actual case even in DOS/Windows systems. Of course it will make file opening slower as it "finds" the file and checks the name.

Something like this:

Code: Select all

#include <dir.h>
#include <stdio.h>

FILE* MyWesnothFileOpen (char *FileName, char *FileMode)
{
  struct ffblk FileFind;

  if (findfirst (FileName, &FileFind, FA_ARCH)
     || strcmp (FileFind.ff_name, FileName) != 0)
  /* Didn't find the file or the name isn't "the same". */
    return NULL;
  return fopen (FileName, FileMode);
}
User avatar
Sapient
Inactive Developer
Posts: 4453
Joined: November 26th, 2005, 7:41 am
Contact:

Re: file names, directories and case sensitivity

Post by Sapient »

Rather than slowing down all valid file lookups on Windows to allow them to artificially fail, wouldn't it make a lot more sense to handle the (presumably far less frequent) case of failed file lookups on Linux systems? To handle it, you can simply unwind the path until it is valid then load each section of the path by checking for a differently-cased version. Any add-ons with files or directories differing only by case would be invalid for cross-OS compatibility purposes, regardless.
http://www.wesnoth.org/wiki/User:Sapient... "Looks like your skills saved us again. Uh, well at least, they saved Soarin's apple pie."
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: file names, directories and case sensitivity

Post by silene »

Sapient wrote:To handle it, you can simply unwind the path until it is valid then load each section of the path by checking for a differently-cased version.
Please define "differently-cased" version, from an implementation point of view. For instance, are "i" and "I" the same letter with difference cases? Depending on the locale, they aren't! Case insensivity is a can of worm; I would much prefer slowing Wesnoth on Windows to check for case sensitivity rather than reimplementing in Linux the whole codepage mechanism of Windows.

Note that, in most cases, I don't even think there would be a slowdown on Windows. Indeed, Wesnoth is already visiting the directories (e.g. binary_path) in order to find the corresponding file. I'm sure someone could come with an implementation that doesn't incur more system calls. So Windows developers, how do you change the following Wesnoth function so that it tests for both existence and case? (Usage of Windows API is allowed, obviously.)

Code: Select all

bool file_exists(const std::string& name)
{
       struct stat st;
       return ::stat(name.c_str(), &st) == 0;
}
User avatar
Iris
Site Administrator
Posts: 6744
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: file names, directories and case sensitivity

Post by Iris »

norbert wrote:[Edit: A similar problem is that Windows users frequently type the path separator characters for, for example, web icons with backslashes instead of slashes (which gives them an icons/missing-image.png).]
Disregarding the "web" part (which I have no idea what it exactly means), I'd suggest two solutions:

1) Disallowing backslashes in file system method arguments.
2) Translating backslashes in file system method arguments.

Of course (2) would mean even more string replacements at runtime and would encourage laziness on part of the content authors. It could also make bogus paths such as "C:\Users\Foobar\Documents\blah.png" less obvious — and it would also look out of place without universal case-insensitivity. So I'd go for (1).

(Also, I seem to recall that FAT32 filesystems are not case-preserving on Windows 9x, but since we don't support those anymore they'd be out of the question. After skimming the Wikipedia article on FAT it seems that FAT32 is case-preserving on the Windows NT family. I also remember that NTFS is case-preserving.)

Another solution to consider would be a upload-time check from the add-ons server that took advantage of the add-on packages' format used for transmitting over the network — WML data which could be searched for strings that look like filenames. This would probably impose a lot of extra load in the add-ons server though, since not only the add-ons' own directories and files would need to be checked for case matches, but also Wesnoth's core datadir.

A different solution would be a periodic (daily, weekly, etc.) check run on all the add-ons in the server (maybe using wmllint*) that delivered results to be viewed by everyone who wanted to take action on them.

(* I don't like hate wmllint. ESR has proposed making wmllint-awareness a requirement for keeping content on the add-ons server before, though.)

EDIT: in case you are desperate and want to be fancy, you can also create a VFAT filesystem image on your case-sensitive Linux-compatible filesystem and mount it as your .wesnoth/data/add-ons directory, by the way. I know it sounds ridiculous, but I once did that for running a program that ran on Linux but used data files that were only tested on Windows. No, it won't solve the problem for mismatched case on references to core data files. You could create another image for /usr/share/wesnoth though!
Last edited by Iris on June 5th, 2010, 6:23 am, edited 1 time in total.
Reason: Inserted mandatory non-serious-looking addendum.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm (now available for Wesnoth 1.14.x and 1.15.4+).
User avatar
Sapient
Inactive Developer
Posts: 4453
Joined: November 26th, 2005, 7:41 am
Contact:

Re: file names, directories and case sensitivity

Post by Sapient »

silene wrote:
Sapient wrote:To handle it, you can simply unwind the path until it is valid then load each section of the path by checking for a differently-cased version.
Please define "differently-cased" version, from an implementation point of view. For instance, are "i" and "I" the same letter with difference cases? Depending on the locale, they aren't! Case insensivity is a can of worm; I would much prefer slowing Wesnoth on Windows to check for case sensitivity rather than reimplementing in Linux the whole codepage mechanism of Windows.
By "differently cased" I mean of course that ASCII a-z can be interchangeable with A-Z. That , by the way, is one of the weakest straw man arguments I have ever heard and has no absolutely no bearing on any usermade content I am aware of.
http://www.wesnoth.org/wiki/User:Sapient... "Looks like your skills saved us again. Uh, well at least, they saved Soarin's apple pie."
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: file names, directories and case sensitivity

Post by silene »

Sapient wrote:By "differently cased" I mean of course that ASCII a-z can be interchangeable with A-Z.
You really missed my point, didn't you? Windows on a latin5 system doesn't interchange i and I! Perhaps your Windows is ASCII only, but this is not the case for anyone out there!
User avatar
Sapient
Inactive Developer
Posts: 4453
Joined: November 26th, 2005, 7:41 am
Contact:

Re: file names, directories and case sensitivity

Post by Sapient »

The code would not even be active on Windows, but on case sensitive OS when a file was not found, so the point whatever it was is meaningless. Also, any add-ons with files or directories differing only by case would be invalid for cross-OS compatibility purposes.

Remember the original problem being discussed is getting the usermade content to work for everyone, so it just needs to work for all the practical cases -- without annoying content creators or needlessly slowing down normal file operations.
http://www.wesnoth.org/wiki/User:Sapient... "Looks like your skills saved us again. Uh, well at least, they saved Soarin's apple pie."
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: file names, directories and case sensitivity

Post by silene »

Sapient wrote:The code would not even be active on Windows, but on case sensitive OS when a file was not found, so the point whatever it was is meaningless. Also, any add-ons with files or directories differing only by case would be invalid for cross-OS compatibility purposes.
Again, you are missing my point; are you even trying? Let's suppose there is a UMC containing a file with "I" in its name but it is referred with "i" in WML (or the contrary). Then there are some Windows systems where the file won't be found. So we want to detect this kind of issue, and the only way I can imagine is to enforce case sensitivity.
Post Reply