Restricted Python

[allefant]

Recently, I found out about another project which uses Python scripting: Phil Hassey is working on public python bots for his game Galcon, and he does it by simply severely restricting the standard Python interpreter, to a degree that it seems 100% safe. The method is similar to the mentioned restricted python from the Zope project, but much simpler (and much more restricted). Anyway, I want to try using his code for Wesnoth.

Some of the restrictions (yes, they are severe):

no import (but possibility to make external stuff available, e.g. the wesnoth module)
no exceptions (scripts need to check return values, no big problem)
no builtin functions (can't use setattr or type or eval or file..)
no access to internals (no access to .__dict__ or .__class__ or .__name__..)

The result of course is not really Python anymore, only a very simple scripting language with Python syntax. But it means, the current Python code can be kept (also can continue to allow use of the full Python interpreter for signed scripts). And this restricted scripting language still is very nice I think, nicer than e.g. Lua.

Of course, there is no 100% guarantee for security. But then, there isn't for WML either, a real hacker might find ways to execute arbitrary code from a WML script.

Anyway, since some more python users besides me are reading this forum, I wanted to hear some opinions about it, or if someone even can see an obvious attack point in Phil Hasseys's code (which is attached).

Just put safe.py, test.py and script.py in a directory, and run it with "python test.py". Then see if you can do something unsafe inside script.py (it already has some hello world code which can be deleted). E.g. adding an "import", "file" or "eval" statement will immediately refuse to execute with an exception from the safe.py module.. and so should any other malicious use.

[rrenaud]

Maybe you should post this to comp.lang.python and see what they say?

[singalen]

If it makes game notably faster and lighter on memory, it's great.
Though, "no exceptions and internals" give some doubts.

[allefant]

Exceptions are allowed now in the code this is from, I'll try updating it, so Python AIs will be able to use them again (and the API could throw them again).

And this makes no difference in performance or memory at all, it's just to ensure the code can not touch anything outside of Wesnoth, which is a requirement for executing it when downloaded from the campaign server.

[H3g3m0n]

It might be worth looking at pypy-c which has seems to have sandboxing support. http://codespeak.net/pypy/dist/pypy/doc/sandbox.html

Haven't used it myself so I'm not sure how fast/complete it is but its seems safe.

[allefant]

This sounds very nice. I have no idea about PyPy though.. like, how can I replace cpython with it? Do I still compile and link with the normal python-dev, and only have to use a different runtime? Would we still need to re-distribute cpython for it to work?

On their site, it also says "PyPy requires a 32-bit machine or OS for now." which means I couldn't even use it here yet. But looks very promising for sure.

[shining]

Hi,

I'm a Java developer not familiar with C#/C++. I want to "refresh" a bit of my python know-how by writing an ai for the game which appeals me most, Wesnoth. I managed to compile it under Windows, and wonder if you could give me a quick hint how I can remove the python restriction over safe.py from my client to have full python support? I find it not appealing to develop this script under such restrictive conditions neither do I want to configure it for every new library I want to use. Thanks for your help.

Greetz,
shining

[allefant]

Look at data/ais/safe.py, look for the function "safe_exec", and replace it by something like (untested):

Code: Select all

def safe_exec(code, context):
    exec code in context

Which restriction in particular did you find limiting? If it's just missing standard modules, they can be easily added to safe.py unless they are unsafe (like os or sys). I also want to re-allow exceptions as they should be safe enough, just needs a bit of looking into.

[shining]

Hi allefant,

thanks for your help. Just wanted to test to include sys, but I still get caught by parse.py. The pathes array was not defined, but I want to try to include a module of the std lib so I declared it and left it empty. I added sys to the whitelist, but now it's complaining

Code: Select all

Python version: 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)]
Traceback (most recent call last):
  File "<string>", line 9, in ?
  File "./data/ais\safe.py", line 131, in safe_exec
    exec code in context
  File "<string>", line 395, in ?
  File "<string>", line 28, in __init__
NameError: global name 'sys' is not defined

I see in parse.py it does work, can you explain me how to include things from the std lib in my ai script? Is this because it is not compiled but interpreted at runtime?

I tried dynamical import

Code: Select all

sys = __import__('sys')

but now when I want to test write sth. to stderr I get

Code: Select all

Python version: 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)]
Traceback (most recent call last):
  File "<string>", line 9, in ?
  File "./data/ais\safe.py", line 131, in safe_exec
    exec code in context
  File "<string>", line 396, in ?
  File "<string>", line 23, in __init__
  File "<string>", line 255, in recruit
IOError: [Errno 9] Bad file descriptor

is this because you write stderr to a file and I somehow confused the path?
Thanks for your help. I think this is not the right place to acquire your help, where should I put my request to?

[allefant]

This is the right place I'd say.

Hm, I completely forgot about parse.py - so in addition to safe.py, I also do some custom parsing..

Ok, then best to bypass this all directly from C++. In ai_python.cpp (btw, you should get the current SVN version, as I fixed a rather huge memory leak recently due to a wrong ref count - directly using the C-Python API just is no good idea, swig/pyrex/boost-python and so on are way better choices), look for where there is a string which looks like this:

Code: Select all

        python_code +=
		"err = \"unknown error\"\n"
		"try:\n"
		"\timport sys, traceback\n"
		"\tsys.stderr = file(\"pyerr.txt\", \"wb\")\n"
		"\tbackup = sys.path[:]\n"
		"\tsys.path.append(\"" + path + "/data/ais\")\n"
		"\ttry:\n"
		"\t\timport wesnoth, parse, safe, heapq, random\n"
		"\t\tcode = parse.parse(\"" + script + "\")\n"
		"\t\tsafe.safe_exec(code, {\n"
		"\t\t\"wesnoth\" : wesnoth,\n"
		"\t\t\"heapq\" : heapq,\n"
		"\t\t\"random\" : random})\n"
		"\texcept:\n"
		"\t\terr = str(traceback.format_exc())\n"
		"\t\traise\n"
		"finally:\n"
		"\tsys.path = backup\n";
	PyObject *ret = PyRun_String(python_code.c_str(), Py_file_input,
		globals, globals);

and (untested) replace it with:

Code: Select all

		"err = \"unknown error\"\n"
		"try:\n"
		"\timport sys, traceback\n"
		"\tsys.stderr = file(\"pyerr.txt\", \"wb\")\n"
		"\tbackup = sys.path[:]\n"
		"\tsys.path.append(\"" + path + "/data/ais\")\n"
		"\ttry:\n"
		"\t\timport \"" + script + "\")\n"
		"\texcept:\n"
		"\t\terr = str(traceback.format_exc())\n"
		"\t\traise\n"
		"finally:\n"
		"\tsys.path = backup\n";
		
	PyObject *ret = PyRun_String(python_code.c_str(), Py_file_input,
		globals, globals);

So now, neither parse() nor safe_exec() are used.

[shining]

Hi Allefant,

Thx for your help, it took me a while to checkout the stuff from svn and compile it. I still can't get it to work, now Wesnoth is complaining about wrong python version:

Code: Select all

Battle for Wesnoth v1.3.10+svn
Started on Wed Nov 14 23:44:34 2007

loadscreen: Logo image is too big.
WARNING: setlocale() failed for ''.
set locale to ''
loadscreen: filesystem counter = 231
loadscreen: binarywml counter = 21960
loadscreen: setconfig counter = 240
loadscreen: parser counter = 214
counted sides: 1
counted sides: 2
Python version: 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)]
internal error (wrong python version?)

That's the same version I used to compile the project tagged with 1.3.10 with. This is what I changed (only the String) as you suggested.

Code: Select all

python_code +=
      "err = \"unknown error\"\n"
      "try:\n"
      "\timport sys, traceback\n"
      "\tsys.stderr = file(\"pyerr.txt\", \"wb\")\n"
      "\tbackup = sys.path[:]\n"
      "\tsys.path.append(\"" + path + "/data/ais\")\n"
      "\ttry:\n"
      "\t\timport \"" + script + "\")\n"
      "\texcept:\n"
      "\t\terr = str(traceback.format_exc())\n"
      "\t\traise\n"
      "finally:\n"
      "\tsys.path = backup\n";
	PyObject *ret = PyRun_String(python_code.c_str(), Py_file_input,
		globals, globals);

	if (PyErr_Occurred()) {
		// RuntimeError is the game-won exception, no need to print it.
		// Anything else likely is a mistake by the script author.
		if (!PyErr_ExceptionMatches(PyExc_RuntimeError)) {
			LOG_AI << "Python script has crashed.\n";
			std::cerr << "Python version: " << Py_GetVersion() << "\n";
			PyObject *err = PyDict_GetItemString(globals, "err");
			std::cerr << (err ? PyString_AsString(err) :
				std::string("internal error (wrong python version?)")) <<
				std::endl;
		}
		// Otherwise, re-throw the exception here,
		// so it will get handled properly further up.
		else {
			LOG_AI << "Python script has been interrupted.\n";
			Py_XDECREF(ret);
			Py_DECREF(globals);
			throw exception;
		}
	}
	Py_XDECREF(ret);
	Py_DECREF(globals);

I'm not yet good in reading this C++ stuff but could this be that this is another Python runtime exception?

[allefant]

Ah, this

"\t\timport \"" + script + "\")\n"

shuld be

"\t\timport " + script + "\n"

or something.. so yes, it's of course just a syntax error, not the wrong python version. Basically, script has the name of the python file to execute, and you want to execute it - there's likely other ways as well, like that one Python-C-API function to execute a .py file.. but the above should work fine once the string has correct Python code in it.

[mrk]

Hi shining, Allefant,

I've been working on a Python AI for Wesnoth for a while now and have managed to get by, by simply disabling the safe.py checks as Allefant suggested, and occasionally adding modules (string, math) to the parse whitelist.

This doesn't work now that I want to persist a complex graph structure between turns! I'd love to use pickle to serialize into a string and then wesnoth.set_variable, but whitelisting pickle doesn't seem to help. After looking at the code posted above, I'm surprised string and math worked - I guess they're conveniently already imported by something else.

I'd really like to see the python API get_variable and set_variable calls taking objects and using pickle internally to convert them to and from strings - I think this'd help a lot of people out.

In the meantime, any suggestions for how I might persist these objects without checking out the wesnoth source and setting up a CPP dev environment here would be really appreciated!

Mark

[mrk]

Actually shining already showed me a good workaround:

Code: Select all

pickle = __import__('pickle')

Unfortunately, pickle "can't pickle location objects". Shame, but I'm sure I can work my way around that. Makes auto-pickling in the API less attractive, of course.

I also wanted to give a bit of thanks for the Python API - it's excellent to be able to just open an editor and write some AI for your favourite game! If I'd had to download and compile the C++ source, I'd probably never have bothered... so, thanks

[allefant]

I'll make string and math get always imported, they should be safe, and make sense to be available.

And I'll see about extending set_variable and get_variable as you suggest. Just must think a bit about it. pickle itself is very unsafe, since a malicious user could just unpickle system("rm -rf /") from a binary string he pickled onto the campaign server. But only allowing set_variable to pickle, and get_variable to unpickle, is different.

For set_variable, since you can only pass an object to it which already is allowed in the safe execution context, it shouldn't be possible to inject malicious code into the safegame.

And as there should be no way to put malicious code into a safegame, calling get_variable should not be able to unpickle any malicious objects either. Of course, I'm not entirely sure, maybe if you are some evil hacker, you somehow can manage to pickle a string in a way so it gets unpickled as malicious code..

Do you need to pickle classes and functions? If not, I could simply use http://docs.python.org/api/marshalling-utils.html from the C-Python-API to extend set/get_string. It works only for "None, integers, long integers, floating point numbers, strings, Unicode objects, tuples, lists, dictionaries, and code objects, where it should be understood that tuples, lists and dictionaries are only supported as long as the values contained therein are themselves supported". But, I could add that very easily, and without much security concerns.