Alexey Solofnenko blog

Long time ago I believed that all dependencies should be checked in into CVS. Since that in every company I worked we did that. Unfortunately when my projects became too big it was clear - version control systems do not work well with many big blobs. The problem is not just with blobs, but with the overhead that version control adds to store them. It is also more difficult to clean them up.

Maven is supposed to solve third party dependencies problem, but I never liked its idea that all projects should be all alike. It is possible to use Maven from ANT, but then it like using computers to hammer nails in. This task can be solved by a simple macro. My solution was the following - create a web server where all external libraries are placed (in a simpler case it could be just a shared directory). Because we need a guarantee that the files are available at any time, we have to have a copy of all our external dependencies. The directory structure is ${root}/${product}/${version}/${project}.zip. I preferred zip, because JBuilder can access sources and documentation directly from it. There are two major types of dependencies - libraries and packages. With libraries, only jar files are needed to be extracted. From packages all files are needed. An additional problem was the way how most libraries are packaged - for some reason people like to put all the files into directories with version number and archive them. Version number had to be removed from directory paths and jar file names, when necessary. So I created this script:

<macrodef name="download">
  <attribute name="name"/>  <!-- library name -->
  <attribute name="jars-only" default="true"/>  <!-- only jars are needed -->
  <attribute name="mod" default=""/>  <!-- mod version, to force update -->
  <element name="commands" implicit="true" optional="true"/>
        <!-- extra commands, if changed, mod should be updated too -->
  <sequential>
    <if>
      <not>
        <available file="lib/@{name}/${@{name}-version}@{mod}"/>
      </not>
      <then>
        <delete failonerror="false" includeemptydirs="true">
          <fileset dir="lib/@{name}" defaultexcludes="no"/>
        </delete>
        <mkdir dir="lib/@{name}"/>
        <get src="${root}/@{name}/${@{name}-version}/@{name}.zip"
                dest="lib/@{name}/@{name}.zip" usetimestamp="true"/>
        <if><istrue value="@{jars-only}"/>
                <then>
                  <unzip src="lib/@{name}/@{name}.zip" dest="lib/@{name}">
                    <patternset><include name="**/*.jar"/></patternset>
                  </unzip>
                  <move todir="lib/@{name}">
              <fileset dir="lib/@{name}" includes="*/**/*.jar"/>
              <mapper type="flatten"/>
            </move>
          </then>
          <else>
            <unzip src="lib/@{name}/@{name}.zip" dest="lib/@{name}">
              <patternset><exclude name="**/CVS/"/></patternset>
            </unzip>
            <move todir="lib/@{name}">
              <fileset dir="lib/@{name}" includes="*/"/>
              <regexpmapper from="^[^/\\]+[/\\](.*)$" to="\1"/>
            </move>
          </else>
        </if>
        <commands/>
        <touch file="lib/@{name}/${@{name}-version}@{mod}"/>
      </then>
    </if>
  </sequential>
</macrodef>

Usage is quite simple:

  <property name="xerces-version" value="2.8.1"/>
  <property name="productA-version" value="3.4"/>

  <target name="fetch">
    <download name="xerces"/>
    <download name="productA" jars-only="false" mod=".0">
      <copy file="conf/default.xml" 
           todir="lib/productA/conf" overwrite="true"/>
    </download>
  </target>

I put version numbers outside to be able easily change them. In fact it is better to put them into a property file and load it in your build script.

ANT 1.7 is getting closer, so I decided to rebuild ANT used in our build system. Last time I did that I noticed an abysmal file deletion performance (actually it was resource collection creation performance). Our formal build process reported about 1200 files changed. Well, it was a long time, since I rebuilt ANT for official builds, but I think the report included generated JavaDoc files too. After ANT rebuild, I launched normal build with clean target first and ... it paused... It took almost three minutes to delete generated files. There were a lot of files to be deleted, but three minutes seemed a lot. After that I ran two builds with new and old ANT and new ANT was actually faster. There was something else affecting the builds, but the file deletion still seems too long. I will run build in profiler to see what is going on.

I have been running parallel ANT for several months now, but the results are not very interesting. The build time reduced about 15% on P4 HT. I will need to run it on Solaris to see the difference. Another problem is logging - my existing scripts go crazy trying to analyze what is happening with the build. I guess each printed line should be prefixed with thread id.