Alexey Solofnenko blog

Recently we have been hit with a new error in our multi platform builds with files stored on NAS - different build commands could fail on Unix with "not owner" error message. At first it was Solaris linker: "ild: Not owner: chmod() failed for file ...", then we switched to another machine and got:

java.io.IOException: java.io.IOException: Not owner
  at java.io.UnixFileSystem.createFileExclusively(Native Method)
  at java.io.File.createNewFile(File.java:883)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)

And there were no obvious changes in the build that could cause it. So we set together with IT and tried to diagnose the problem. What was even more surprising, the same commands could sometimes succeed when executed manually. After few attempts (the whole process took few hours) we found that we cannot change a permission on just created folders. Even "truss" was showing EPERM error code. When waited for about few (<30) seconds the same chmod would succeed. We tried Solaris 10 and Linux with the same result. At least we got something easily reproducible.

So the working theory became that NAS assigns some different permissions to just created folders and then changes them to correct permissions later. To remove NIS server out of the picture we tried a local account and got just created directory owned by root(!). Of cause after that chmod failed with the same "Not owner" error. We went back and forth thinking why it happened and decided that the test did not show what we needed - it is possible that NAS did not know the account and use the account we use for mount.

Google suggested changing directory permissions to 777 and it, of cause, worked. Unfortunately, sticky bits and umask did not seem working on NAS, so all new folders were created with 755 permissions. After few more experiments and cups of coffee we noticed that the build root directory is owned by root. That was odd.

And we changed its ownership to the correct account and group. We also changed its permission to 777. And it helped. It looks like our NAS uses the parent folder for initial ownership settings and then changes it to the correct once and uses parent directory permissions as-is. Now initial owner became correct (or 777 helped) and the builds are running correctly. I will try to set the builds root folder permissions to 755 to actually see what change fixed the build, but not now - now we need the builds working.