Optimization · Tips & Tricks · Uncategorized

Decrease Your Build Time

Delphi is known for its fast build speeds, at least for Win32 apps. But when you compile your same FireMonkey app for other platforms, the build time increases dramatically and can be up to 15 times slower. Granted, it is still much faster than most C++ compilers, but when you are used to blazing fast builds, then compiling for other platforms can test your patience. We show some techniques to increase your build speeds by up to a factor of 3.

Why so slow?

But why are the other compilers so much slower than the Win32 compiler to begin with? I don’t work for Embarcadero, so I cannot be certain, but a good guess is that this has to do with the LLVM backend that is used for all non-Windows compilers. LLVM is a great compiler backend that offers many features and code optimizations. The biggest advantage of LLVM for compiler vendors like Embarcadero is that you “only” need to provide a scanner, parser and abstract syntax tree (AST) for your language (I am over-simplifying here). LLVM will take care of most optimizations, and more importantly, of converting the result to target specific instruction sets such as x86, x64, Arm and Arm64. This enabled Embarcadero to add support for other platforms relatively quickly. The downside is that this backend is (much) slower than the hand-written Win32 compiler. I don’t know if this is intrinsic to LLVM or to the way it is used by the Delphi compilers.

I know that Delphi 10.4.2 has seen considerable compiler (speed) improvements, but only for the platform (Win32) that least needs it IMHO. Sure, a lot of Delphi developers still write mainly for Windows, and they will no doubt appreciate these improvements. But it would be nice to see similar improvements for the other platforms as well.

Pre-compile your source!

In this article I will focus mostly on one technique to increase your build speed: pre-compile most of your source code to DCUs. Then build your main app using those pre-built DCUs. This eliminates the scanning, parsing and AST generation stages for those source files, and even decreases linking time.

However, this technique may not be suitable for everyone, and it takes a bit of setup time. The idea is that you pre-compile those parts of your source code that don’t change often. Much like Embarcadero provides pre-compiled DCUs for its RTL, VCL and FMX source code.

I know what you are thinking: Isn’t that what the Compile option is for (as opposed to Build)? Yes, the Compile option is supposed to use already generated DCU files instead of rebuilding them. But in my experience, as projects get more complex, this doesn’t always work, especially for the LLVM backends. Performing a Compile instead of Build will often either recompile the source anyway, or result in (internal) compiler errors.

At Grijjy, we organize our source code in a similar way Embarcadero does: we have an RTL directory with common (framework-independent) source code, a FMX directory with our own FireMonkey controls, a ThirdParty directory with 3rd party source code, and various project directories with the actual frontend and backend apps we produce.

However, the code in the RTL, FMX and ThirdParty directories doesn’t change very often. This makes them great targets for pre-compilation. The remainder of this article assumes the following directory structure:

  • C:\Source: main source code directory
    • \RTL: common Run Time Library code
    • \FMX: common FireMonkey code and controls
    • \ThirdParty: base directory for 3rd party source code
      • \LibX: sample 3rd party library
      • \LibY: another sample 3rd party library
    • \MyApp: your main application
    • \DCU: base directory for the pre-compiled DCU files
      • \Win32\Debug: output directory for Win32 Debug DCUs
      • \Win32\Release: output directory for Win32 Release DCUs
      • \iOSDevice64\Debug: output directory for iOS Debug DCUs
      • etc…

Your directory structure may (and probably will) vary, but the same concepts still apply.

Step 1: Build your DCUs

The first step is to create a new Delphi project that builds these DCUs. You will never need to run this project; you only build it to generate the DCUs. Start by creating a new empty Multi-Device (FireMonkey) Application. You can remove the main form, as well as the uses-clause and initialization section of the generated DPR file. You can also remove any target platforms you don’t care about (such as “iOS Simulator”). Next, you need to set the “Unit output directory” for this project:

  • Open the Project Options.
  • Select the “Building | Delphi Compiler” node in the option tree.
  • Select the “All configurations – All platforms” target in the Target combo box.
  • Set the “Unit output directory” to “C:\Source\DCU\$(Platform)\$(Config)” (or the corresponding directory on your system).

The $(Platform) and $(Config) variables will be automatically expanded, so this works for all platforms (Win32, OSX64 etc.) and all configurations (Debug and Release).

Next, add a new unit that will list all units you want to pre-compile. For example:

unit IncludedUnits;

interface

uses
  { Custom RTL Units }
  MyRTL.Classes,
  MyRTL.Utils,
  { Custom FMX Units }
  MyFMX.Media,
  MyFMX.Controls,
  { 3rd Party Units }
  LibX.Foo,
  LibX.Bar,
  LibY.Tools;

implementation

end.

If some of these units are only applicable to certain platforms, then you can use the regular {$IF Defined(...)} directives to configure this. Also, if this list becomes very large, it may be more manageable to separate this unit into multiple units (for example, RtlUnits, FmxUnits and ThirdPartyUnits).

Now build this project for all platform/configuration combinations you care about. Afterwards, when you take a look at your C:\Source\DCU directory, you will find the DCU files for all platform/configuration combinations here. For some platforms, there are also object files (.o files) here. These are needed as well.

Step 2: Use the pre-built DCUs

Now that we have the DCUs we can use them to build your main app (instead of compiling the source code every time).

Update Library and Debug DCU Paths

To do this, we must first update our library paths (using the “Tools | Options” menu option). Say that for example, your current Android32 library path looks like this:

Then you need to add the DCU\Android\Release directory to this list, above the source directories:

This way, the DCU directory is used first and the Source directories are only used if the unit cannot be found in the DCU directory.

We will look at another way to handle this later in this article.

If your generated DCUs contain patches to original Delphi RTL/FMX files, then you must make sure that you move this directory to the very top of the list (above $(BDSLIB)\$(Platform)\Release).

When using DCUs, these directories are used for Release builds. For Debug builds, you need to add the DCU\Android\Debug directory to the “Debug DCU path” (on the same page as “Library path”):

Again, you need to push this to the top of the list if you have any Delphi RTL/FMX patches.

You need to repeat these steps for all other platforms you care about as well.

Build using DCUs

Now, when you build your main app, you should see that the compiler doesn’t compile the original RTL/FMX/ThirdParty source code anymore. And, depending on how much source is in these directories, building should be much faster now, especially for macOS, iOS and Android.

You should be able to run and debug your app in the usual way. Since the source directories are still part of your library path, you should be able to debug the original source files as well, even though only the pre-built DCUs are used for building.

Speed gains

I am using this technique to build our Lumicademy client apps. We have quite a large collection of common RTL code (similar to our open source GrijjyFoundation library), as well as some custom FMX controls and 3rd party libraries. As a result, I am seeing considerable gains in build speed when using pre-built DCUs. Of course, your mileage will vary, but to give you an idea, these are the speed gains we are seeing:

This chart shows the build times (in seconds) of the Debug version of our Lumicademy app on all supported platforms. Each bar is separated into 2 parts: the left part shows the compilation time and the right part shows the link time. We show 2 bars for each platform: the top bar (blue/orange) shows the “normal” build time (using just source code) and the bottom bar (gray/yellow) shows the build time using pre-compiled DCUs for common source code.

As you can see, using pre-compiled DCUs saves a lot of time. Building time is typically halved, but can be up to almost 3 times faster for some platforms. The speed gains for the Release build (not shown here) are even higher.

Some observations:

  • Even for the Win32 platform, using pre-compiled DCUs is still a bit faster. But since the Win32 compiler is already very fast, the gains are modest (about a factor of 1.4).
  • The Win64 compiler is also pretty fast compared to the other platforms, but can still be about twice as fast using pre-compiled DCUs.
  • I expected that using pre-compiled DCUs would have a big effect on compilation time, but it looks like it also dramatically decreases link time (which I didn’t expect). Don’t know exactly why that is, but it is a nice bonus!

How to handle source code changes

This is all very nice, but what if you need to make changes to your common source code? Do you need to restore your library paths, then build the DCUs again, and finally change your library paths back again? Of course, that is one way to do it, but it gets tedious quickly. A slightly less tedious way is to add a single character to your DCU path to make it invalid:

The path is now grayed out, meaning it is invalid and will (can) not be used. Of course, you need to repeat this for all platforms and for the Debug DCU Paths as well. Fortunately, there is a better way.

Multiple Delphi configurations

Although it is not a secret, it is also not well known that you can have multiple Delphi configurations. So you can have one configuration for your day-to-day use, and another configuration just for rebuilding your DCUs. The regular configuration would only have the DCU directories in the Library Paths and Debug DCU Paths, while the alternative configuration would only have the source code directories. You can also use the alternative configuration to build your app using source code instead of DCUs. This may be easier while you are in the process of making changes to your common source code.

So how do you create an alternative Delphi configuration? You cannot do this inside the IDE. Instead, you need to use the registry editor and an additional Delphi command line option. Fortunately, this isn’t very complicated:

  • Open the registry editor (RegEdit).
  • Navigate to the branch “HKEY_CURRENT_USER\SOFTWARE\Embarcadero\BDS\21.0”, where the last element of the path is the Delphi version you are using (21.0 for Delphi 10.4.x).
  • Choose “File | Export” to export this branch to a text file.
  • Open the text file in a text editor. It should start like this:
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Embarcadero\BDS\21.0]
"ProductVersion"="27"
"RegCompany"=""
"RootDir"="C:\\Program Files (x86)\\Embarcadero\\Studio\\21.0\\"
...

Now we need to replace the “\BDS\” part in each section with an alternative configuration name. Let’s say we want to name this configuration “BDSSrc” instead:

  • In the text editor, replace all occurrences of “\BDS\” with “\BDSSrc\” (without the double quotes). Be sure to use two backslashes in both the search and replacement terms to avoid renaming any other text that may contain the string “BDS”.
  • Save the text file.
  • Back in the registry editor, choose “File | Import” and select the file you just saved.

You should now see an additional “BDSSrc” node below the “BDS” node. Now, to use this configuration, you need to start the Delphi IDE with an additional command line parameter. This is easiest done by creating a shortcut on your desktop:

  • From the Windows Start menu, drag the Delphi icon to your desktop to create a link.
  • Rename the link to something like “Delphi Source Config”.
  • Right-click the link and select “Properties”.
  • On the Shortcut tab, add a -rBDSSrc command line parameter to the Target:

I also changed the icon to make it visually distinct from the default configuration.

Now, when you start this configuration, all changes you make to the settings in the IDE will only apply to this configuration. Your original Delphi configuration will remain unchanged. So you can setup your 2 configurations like this:

  • For the default configuration, change the Library Paths and Debug DCU Paths to only reference the DCU directories.
  • For the “Delphi Source Config” configuration, change the Library Paths to only reference the original source code directories. The Debug DCU Paths should not contain any reference to source directories or custom DCU directories.

If you need to rebuild the DCUs now, you only have to start the alternative “Delphi Source Config” configuration. You can then switch back to your default configuration to build your app with the newly build DCUs.

Debugging with a DCU-only configuration

There are some caveats with using multiple configurations though. For example, if you use your regular (DCU-only) configuration to debug your app, and you step into one of the pre-built units, then you will get an error message similar to this:

This is because the IDE no longer knows where the original source code is located. There are two ways to fix this:

First, you could click the “Browse” button to locate the original source file. When you leave the checkbox “Add directory to Debug Source Path” checked, then the directory will be added to your project options so it survives IDE restarts. You will find this directory in the “Project | Options” dialog box, under “Debugger | Source Path” (which is platform and configuration specific).

However, this adds the source path only for your project. A better way to fix this is by manually adding the source directories to your global IDE options, by selecting the “Tools | Options” menu option. There, you navigate to “Debugger | Embarcadero Debuggers” and add the source directories to the “Debug source path” section. An additional advantage is that the paths here apply to all platforms and configurations, so you only need to add them once.

Whichever option you choose, you will be able to debug your common source code the regular way now, even though the app is build using its DCUs only.

iOS/macOS caveat

The final minor caveat I noticed when using an alternative configuration, is that you may receive the following error message when deploying to an iOS or macOS device:

[Error Error] Specified profile not found:
'C:\Users\<UserName>\AppData\Roaming\Embarcadero\BDSSrc\21.0\MacBook Pro.profile'

This is because the -rBDSSrc command line parameter not only forces the IDE to use a different registry key, but it also uses a different configuration directory under your user roaming AppData folder. You can fix this error by copying the missing file from the original directory (C:\…\BDS\21.0\) to the alternative directory (C:\…\BDSSrc\21.0\). Or in this case, it is easier to just open the “Tools | Options” menu option, navigate to “Deployment | Connection Profile Manager” and click the “Save” button. This will recreate this file for you.

Other ways to decrease build times

There are other ways to decrease your build times. Arguably, fast build times are most important for Debug build, since you will perform those much more often than Release builds. You might expect that Debug builds are faster than Release builds because the compiler and linker don’t have to go through a bunch of optimization phases. But the opposite is true. For the LLVM based compilers, Debug builds can be more than 50% slower.

The reason for this is the generation and storage of debug information. There can be a lot of it. However, for most debugging purposes, you don’t need full debug information. This is especially true for the LLVM based compilers, since debugging on those platforms is limited anyway. You can get considerable speed gains my switching to Limited Debug information:

  • Choose “Project | Options”.
  • Navigate to the “Building | Delphi Compiler | Compiling” page.
  • Select the Debug configuration(s) you want to change this setting for.
  • Set the “Debug information” option to “Limited Debug information” (instead of “Debug information).

This increases build speed by about 25% on my system. Combining this with using pre-built DCUs can make the gains even higher.

Of course, there are other more obvious ways to increase your build speed, such as upgrading your computer and/or switching to SSD drives to store your source code and temporary build files. Single-core CPU speed is an important factor here, since the Delphi compiler is still mostly single-threaded.

Is it worth it?

Sure looks like there is quite a bit of setup required to make these techniques work. So is it worth going through all this trouble? Of course the answer depends on your own situation. If you have a lot of common source code that doesn’t change often, and you regularly need to build apps for non-Windows platforms, then I think this effort is definitely worth it.

It has some other advantages as well. For example, by using pre-built DCUs, the compiler needs to create fewer temporary files. This reduces the load on your hard disk, which can be especially important if you use an SSD (where its lifetime depends on the number of write operations).

Another advantage is that this technique reduces the chance for internal compiler errors, or other errors that sometimes happen during the build process (such as not being able to write or find certain files). It also takes less memory, which may be beneficial when building very large projects.

Just give it a try. No harm, no foul, right?

And if you have any other tips for decreasing build times, let me know in the comments!

3 thoughts on “Decrease Your Build Time

  1. Why don’t you use packages?
    I use them for all.my stuff, ej
    MyRTL and such, with the explicit rebuild option
    So they never get recompiled without asking

    Like

    1. I don’t think you can use packages on iOS. Are you using packages for Android and macos? If that works reliably, I may check it out. I usually stay away from packages to simplify deployment and avoid versioning issues, but I can change my mind…

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s