Alexey Solofnenko blog

Recently we have been hit with a new error in our multi platform builds with files stored on NAS - different build commands could fail on Unix with "not owner" error message. At first it was Solaris linker: "ild: Not owner: chmod() failed for file ...", then we switched to another machine and got:

java.io.IOException: java.io.IOException: Not owner
  at java.io.UnixFileSystem.createFileExclusively(Native Method)
  at java.io.File.createNewFile(File.java:883)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)

And there were no obvious changes in the build that could cause it. So we set together with IT and tried to diagnose the problem. What was even more surprising, the same commands could sometimes succeed when executed manually. After few attempts (the whole process took few hours) we found that we cannot change a permission on just created folders. Even "truss" was showing EPERM error code. When waited for about few (<30) seconds the same chmod would succeed. We tried Solaris 10 and Linux with the same result. At least we got something easily reproducible.

So the working theory became that NAS assigns some different permissions to just created folders and then changes them to correct permissions later. To remove NIS server out of the picture we tried a local account and got just created directory owned by root(!). Of cause after that chmod failed with the same "Not owner" error. We went back and forth thinking why it happened and decided that the test did not show what we needed - it is possible that NAS did not know the account and use the account we use for mount.

Google suggested changing directory permissions to 777 and it, of cause, worked. Unfortunately, sticky bits and umask did not seem working on NAS, so all new folders were created with 755 permissions. After few more experiments and cups of coffee we noticed that the build root directory is owned by root. That was odd.

And we changed its ownership to the correct account and group. We also changed its permission to 777. And it helped. It looks like our NAS uses the parent folder for initial ownership settings and then changes it to the correct once and uses parent directory permissions as-is. Now initial owner became correct (or 777 helped) and the builds are running correctly. I will try to set the builds root folder permissions to 755 to actually see what change fixed the build, but not now - now we need the builds working.

I have had a problem that Google did not have an answer for - how to search for an attribute value, if the attribute name is not case sensitive? Usually the situation arises when the original file is not case sensitive (for example, HTML files are not). In my situation the XML was processed by another code that treated it in a case-insensitive way. Surprisingly the only good answer in Google I got that XML is case sensitive and live with it. After reading that I was about to replace one search with two searches using two different cases that can actually happen in my files, but decided to look further, if it is a possible to write a condition with a node name. Of cause there was - name() function returns the current node name and it can be tested. Surprise, surprise, the query did not work at first - I used '=='instead of '=', but eventually I got it right. This is it:

/Path/To/element/@*[translate(name(), 'abcdefghijklmnopqrstuvwxyz', 'ABCDEFGHIJKLMNOPQRSTUVWXYZ')='UPPERCASE_ATTR_NAME']

While our installer developer was busy on another project, I created a simple patch installer with WIX that just places few assemblies with their policies into GAC. It required some tweaking. My first approach of reading instructions only after all other possibilities failed did not really work. There was one positive outcome - Wix 2.0 was unusable (what is up with 8.3 names!?). I wonder how other people are using it. So I quickly downloaded Wix 3.0 beta and decided to use VisualStudio integration to create my installer. VisualStudio helped a little, but soon I was back editing the file in UltraEdit. After few minutes the installer was ready and it was installing the patch without any user interaction. That was the problem - I needed a question "do you really want to install this thing or you clicked on it by mistake?". Fortunately there were pre-built UIs and I selected WixUI_Minimal. Again it worked great, but what is up with EULA? I did not need it for a patch installer. Searching on internet did not return anything useful besides putting a readme instead of a licence. There were comments about changing standard UIs, so I downloaded WIX sources and started tweaking. Simple replacing WelcomeEulaDlg with WilcomeDlg did not work - there were messages about "next" button left without any action, but what is the "install" action? After few tries I broke my perfectly working installer into something unusable. Fortunately I foresaw that problem and put the script into ClearCase to be able to track my progress. I restored the last working revision and started gradually copying pre-built UI out of WixUI_Minimal into my script. First the UI itself, then the WelcomeEulaDlg. After each iteration I checked that the installer was still working. The last step was to update the dialog with content similar to WelcomeDlg dialog. The whole process actually was a trivial task. By the end the installer worked correctly without any funny EULA business.

This is what I have right now. I guess it can be cleaned up a little, but it should be done with care, each time checking that the installer is still working, so I am not going to do it now:

<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
  <Product Id="...." Name="... HotFix 3" Language="1033" Version="3.1.0.3" Manufacturer="..." UpgradeCode="...">
    <Package InstallerVersion="200" Compressed="yes" />

    <Media Id="1" Cabinet="hotfix3.cab" EmbedCab="yes" />

    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFilesFolder">
        <Directory Id="INSTALLLOCATION" Name="hotfix3">
          <Component Id="....dll" Guid="..." DiskId="1" Feature="ProductFeature">
            ....
          </Component>
          ...
        </Directory>
      </Directory>
    </Directory>

    <Feature Id="ProductFeature" Title="Hotfix 3" Level="1">
    </Feature>
    <Property Id="WIXUI_INSTALLDIR" Value="TARGETDIR" />
    <UI>
        <TextStyle Id="WixUI_Font_Normal" FaceName="Tahoma" Size="8" />
        <TextStyle Id="WixUI_Font_Bigger" FaceName="Tahoma" Size="12" />
        <TextStyle Id="WixUI_Font_Title" FaceName="Tahoma" Size="9" Bold="yes" />

        <Property Id="DefaultUIFont" Value="WixUI_Font_Normal" />
        <Property Id="WixUI_Mode" Value="Minimal" />

        <DialogRef Id="ErrorDlg" />
        <DialogRef Id="FatalError" />
        <DialogRef Id="FilesInUse" />
        <DialogRef Id="MsiRMFilesInUse" />
        <DialogRef Id="PrepareDlg" />
        <DialogRef Id="ProgressDlg" />
        <DialogRef Id="ResumeDlg" />
        <DialogRef Id="UserExit" />

        <Dialog Id="WelcomeEulaDlg2" Width="370" Height="270" Title="!(loc.WelcomeEulaDlg_Title)">
            <Control Id="Bitmap" Type="Bitmap" X="0" Y="0" Width="370" Height="234" TabSkip="no" Text="!(loc.WelcomeEulaDlgBitmap)" />
            <Control Id="Description" Type="Text" X="135" Y="80" Width="220" Height="60" Transparent="yes" NoPrefix="yes" Text="!(loc.WelcomeDlgDescription)" />
            <Control Id="Back" Type="PushButton" X="180" Y="243" Width="56" Height="17" Disabled="yes" Text="!(loc.WixUIBack)" />
            <Control Id="Install" Type="PushButton" ElevationShield="yes" X="236" Y="243" Width="56" Height="17" Default="yes" Text="!(loc.WelcomeEulaDlgInstall)">
                <Publish Event="SpawnWaitDialog" Value="WaitForCostingDlg">CostingComplete = 1</Publish>
                <Publish Event="EndDialog" Value="Return"><![CDATA[OutOfDiskSpace <> 1]]></Publish>
                <Publish Event="SpawnDialog" Value="OutOfRbDiskDlg">OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 0 AND (PROMPTROLLBACKCOST="P" OR NOT PROMPTROLLBACKCOST)</Publish>
                <Publish Event="EndDialog" Value="Return">OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 0 AND PROMPTROLLBACKCOST="D"</Publish>
                <Publish Event="EnableRollback" Value="False">OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 0 AND PROMPTROLLBACKCOST="D"</Publish>
                <Publish Event="SpawnDialog" Value="OutOfDiskDlg">(OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 1) OR (OutOfDiskSpace = 1 AND PROMPTROLLBACKCOST="F")</Publish>
            </Control>
            <Control Id="InstallNoShield" Type="PushButton" ElevationShield="no" X="236" Y="243" Width="56" Height="17" Default="yes" Text="!(loc.WelcomeEulaDlgInstall)" Hidden="yes">
                <Publish Event="SpawnWaitDialog" Value="WaitForCostingDlg">CostingComplete = 1</Publish>
                <Publish Event="EndDialog" Value="Return"><![CDATA[OutOfDiskSpace <> 1]]></Publish>
                <Publish Event="SpawnDialog" Value="OutOfRbDiskDlg">OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 0 AND (PROMPTROLLBACKCOST="P" OR NOT PROMPTROLLBACKCOST)</Publish>
                <Publish Event="EndDialog" Value="Return">OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 0 AND PROMPTROLLBACKCOST="D"</Publish>
                <Publish Event="EnableRollback" Value="False">OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 0 AND PROMPTROLLBACKCOST="D"</Publish>
                <Publish Event="SpawnDialog" Value="OutOfDiskDlg">(OutOfDiskSpace = 1 AND OutOfNoRbDiskSpace = 1) OR (OutOfDiskSpace = 1 AND PROMPTROLLBACKCOST="F")</Publish>
            </Control>
            <Control Id="Cancel" Type="PushButton" X="304" Y="243" Width="56" Height="17" Cancel="yes" Text="!(loc.WixUICancel)">
                <Publish Event="SpawnDialog" Value="CancelDlg">1</Publish>
            </Control>
            <Control Id="Title" Type="Text" X="135" Y="20" Width="220" Height="60" Transparent="yes" NoPrefix="yes" Text="!(loc.WelcomeDlgTitle)" />
            <Control Id="BottomLine" Type="Line" X="0" Y="234" Width="370" Height="0" />
        </Dialog>

        <Publish Dialog="ExitDialog" Control="Finish" Event="EndDialog" Value="Return" Order="999">1</Publish>

        <Publish Dialog="VerifyReadyDlg" Control="Back" Event="NewDialog" Value="MaintenanceTypeDlg">1</Publish>

        <Publish Dialog="MaintenanceWelcomeDlg" Control="Next" Event="NewDialog" Value="MaintenanceTypeDlg">1</Publish>

        <Publish Dialog="MaintenanceTypeDlg" Control="RepairButton" Event="NewDialog" Value="VerifyReadyDlg">1</Publish>
        <Publish Dialog="MaintenanceTypeDlg" Control="RemoveButton" Event="NewDialog" Value="VerifyReadyDlg">1</Publish>
        <Publish Dialog="MaintenanceTypeDlg" Control="Back" Event="NewDialog" Value="MaintenanceWelcomeDlg">1</Publish>

        <Property Id="ARPNOMODIFY" Value="1" />
    </UI>
        <InstallUISequence>
            <Show Dialog="WelcomeEulaDlg2" Before="ProgressDlg">NOT Installed</Show>
        </InstallUISequence>

    <UIRef Id="WixUI_Common" />
  </Product>
</Wix>

Ever wondered if your policy assemblies work?

.Net assemblies remember what versions of referenced assemblies they were built with. In general it is a good thing, but it creates its own problems when building a product that consists of many separate parts. In the interest of QA we often build mismatched package builds with some of the application products built with earlier versions of core libraries. Fortunately our build scripts (directly from VisualStudio) generate all necessary policy assemblies for previous and the current releases and installer puts them into GAC.

At first there were rumors that one of applications (developed in a different office and built on its own schedule) does not start in mismatched builds. In the beginning the warnings were dismissed because the APIs were not stable, but later it became apparent that something is wrong. And finally the problem came to me...

After enabling assembly loader log, it showed that policy assembly was recognized and the redirect actually happened, but the target assembly was not found even it was right there in the GAC. However there was a message that the platform was locked to the policy assembly platform. This particular assembly had native DLLs attached to it, so its platform was not a generic "MSIL", but "x86" and this is why it could not be found.

To test this idea I regenerated policy file with matching platform, but how to install it into GAC? Removing old policy was not possible - it was locked by installer. Fortunately Cygwin`s Midnight Commander sees real file system and ignores all Microsoft crap and I was able to remove the assembly directly from hard drive and install a new one into GAC. And it helped - the application started and worked without problems.

The policy generation (and some other "black magic" staff, including support for linked resources) is done via an include in our project files and is a part of another utility project. After that project rebuild, our automatic build picked it up and certified all dependent projects to pass unit tests. Now all our policy assembly platform match their assembly's platform and everything quiet in our kingdom.

Being a mathematician (and a little bit a physicist) I always felt something is wrong with hydrogen cars idea. Besides the promise of higher cruising range, each additional transformation reduces overall efficiency factor causing even more energy to be used per unit of work (or per mile). I would prefer that all that money would be used on development of better batteries, so cars could drive about 400 miles without recharging.

The idea of using VMWare to save energy is somewhat similar - reducing overall computer efficiency to be able to pack more services on less hardware. Lets review theoretical benefits of computer virtualization:

• automatic distribution of computer resources between VMs
• automatic starting and stopping VMs
• automatic VM relocation between computers
• services isolation

And the cost - less VM performance.

However, if you look closer at the benefits, you will notice they are similar to benefits of Java. Instead of virtualizing the whole OS, Java virtualizes just processes while generating code that executes close to native speeds (sometimes faster, sometimes slower, depending on what you are doing) and Java code has much better portability between computers with different architectures. I know it first hand - after we moved some parts of our build processes from slow Windows VMs to a single Solaris box, we saved hours!

So what is the conclusion - use Java to save energy and adding one more level of indirection is usually not the best solution.

Today I was fixing few Windows forms. There few problems that usually should be easy to fix. However in this case none of direct approaches worked. After few attempts I found that there is a logic in a form Closing event, also in OnHandleDescroyed(). As if it was not enough, there was a secret order of calls to initialize the form. After direct approach failed, I reverted all changes and started refactoring the code to simplify internal relations, move code out of the form into custom controls... At the end the all main problems were fixed and I am preaching everybody not to program like that.

There are few tools that help you with synchronizing several computers (I use excellent Google Browser Sync for Firefox, for example), but none of them can include some logic when the files are supposed to be a little bit different on different computers.

I have had this problem with Thunderbird email rules. I run Thunderbird on my laptop or on a Linux server. Unfortunately historically I specified mail server slightly different (full and short domain names) on both computers, so just file copying does not work. Instead I wrote a simple Cygwin script that I run when I boot my computer in my office (in addition to some file synchronization, it also cleans temporary files up, stops beep service and synchronizes time with the company time (using cmdtime3 that I also run before any automatic test build - automatically updating time during test build can cause the build failure) , which is supposed to be the same as the real time, but why take any chances - some unit tests are very time sensitive).

The idea is simple - on each platform I save old rules file (it could be just a timestamp, but I wanted to be safe and compare file contents) and then check, if rules file was updated on one platform (Linux), and then on another - no three way merges for now. This is the script (uppercase values - parts that are supposed to be machine specific). If you do not see full script with your current skin, just copy/paste it into any editor.

export WIN_FILE="/cygdrive/c/Documents and Settings/USER-NAME/Application Data/Thunderbird/Profiles/PROFILE/ImapMail/SERVER-NAME/msgFilterRules.dat"
export LIN_FILE=//LINUX-SERVER/HOME-SHARE/.thunderbird/LINUX-PROFILE/ImapMail/SERVER-FULL-NAME/msgFilterRules.dat
if ! cmp -s $LIN_FILE ${LIN_FILE}.saved; then
echo Linux updated
cat $LIN_FILE|perl -pe 's^\@SERVER-FULL-NAME/^\@SERVER-NAME/^sg' > "$WIN_FILE"
cp "$WIN_FILE" "${WIN_FILE}.saved"
cp $LIN_FILE ${LIN_FILE}.saved
elif ! cmp -s "$WIN_FILE" "${WIN_FILE}.saved"; then
echo Windows updated
cat "$WIN_FILE"|perl -pe 's^\@SERVER-NAME/^\@SERVER-FULL-NAME/^sg' > "$LIN_FILE"
cp $LIN_FILE ${LIN_FILE}.saved
cp "$WIN_FILE" "${WIN_FILE}.saved"
else
echo Not updated
fi

Everybody knows that metrics can be misleading, but they are still interesting. My boss has came to me and asked how many lines of code do we have in some projects. At first I used Cygwin's wc, but then I needed to filter empty lines and lines with just comments. It was easier to write a Perl script:

find . -name '*.cs' -not -name '*.Designer.cs' -print0|xargs -0 perl -e 'my $num=0; while(<>) { s/^\s+//s; unless (length($_)==0 or m!^#! or m!^//! or m!/\*!) {$num+=1;} } print "$num\n";'

The script does not handle multiline comments correctly, but it provides a reasonable approximation of the number of manually written lines of code.

Just learned about a way of embedding images into HTML pages without using extra files. There are some limitations, for example, IE (including IE7), Outlook and other MS products do not support data: URLs (who cares?!). I tried it right a way with Thunderbird and it worked... except Thunderbird extracted the image and put it as an attachment. This is not what I expected. There are however places, when I send build report messages myself using JavaMail. Sending them without attachments would simplify the sending greatly. Microsoft users will not be at great lost - the important information will be still there.

After all manipulations, I still found that JAlbum cannot sort directories right. The only way to fix that would be sorting everything myself. That would mean generating albumfiles.txt in every folder. The task is sorting root album folder {newest first}, and all child folders {directories first, directories oldest first, files alphabetically}. I have to sort files alphabetically, because their date is not reliable. This is new code in Perl;

sub compareFiles($$) {
  my ($a, $b)=@_;
  my $res = $a->[0] <=> $b->[0];
  if ($res==0) {
    $res=($a->[0]==0 ? $a->[1] <=> $b->[1] : $a->[1] cmp $b->[1]); 
       # zero flag is timestamp comparison 
  }
  $res
}

sub fixRootFolders {
  foreach my $dir (keys %directories) {
    print "Processing directory '$dir'\n";
    my %ignoredFiles;
    if (-r "$dir/albumfiles.txt") {
      open (F, "$dir/albumfiles.txt");
      my $line;
      while (defined($line=readline F)) {
        if ($line =~ /^-/) {
          $line = substr ($line, 1);
          chomp $line;
          #print ">> '$line'\n";
          $ignoredFiles{$line}=1;
        }
      }
      close(F);
    }
    my @files;
    opendir(DIR, $dir) || die "can't opendir '$dir/'";
    @files = map { if (exists $ignoredFiles{$_}) { "-$_"} else { $_} } 
        map {$_->[0]} sort {compareFiles $a->[1], $b->[1]} 
        map {[$_, ($dir eq "." ? [0, - mtime "$dir/$_"] : 
            (-d "$dir/$_" ? [0, mtime "$dir/$_"] : [1, $_ ]))]} 
        grep { $_ !~ /^\./ and (/\.zip$/ or isImage($_) or -d "$dir/$_" ) }
        readdir(DIR);
    closedir DIR;
    #print join "\n", @files;
    
    open (F, ">$dir/albumfiles.txt");
    print F join "\n", @files;
    close(F);
    
  }
}