This page contains various articles related to the generator JSON configuration.

A lot of entries in a config file take path names as their values (top-level “include”, “manifest” keys of a library entry, output path of compile keys, asf.). Quite a few of them, like the top-level include paths, are interpreted relative to the config file in which they appear, and this relation is retained no matter from where you reference the config file.

This might not hold true in each and every case, though. For some keys you might have to take care of relative paths yourself. The authoritative reference is always the corresponding documentation of the individual config keys. If a key takes a path value it will state if and how these values are interpreted. Please check there.

A good help when dealing with paths is also to use macros, if you need to abstract away from a value appearing multiple times. E.g.

"let" : {"MyRoot": ".", "BUILD_PATH" : "build"}
"myjob" : { ... "build_dir" : "${MyRoot}/${BUILD_PATH}" ... }

This should make it more intuitive to maintain a config file.

The main payload of the cache key is to point to the directory for the compile cache. It is very recommendable to have a system-wide compile cache directory so cache contents can be shared among different projects and libraries. Otherwise, the cache has to be rebuilt in each enviornment anew, costing extra time and space.

The compile cache directory can become very large in terms of contained files, and a count of a couple of thousand files is not unusual. You should take care that your file system is equipped to comply with these demands. Additionally, disk I/O is regularly high on this directory so a fast, local disk is recommendable. Don’t use a network drive .

A word of advice is also in place for the time being: Cache management is not optimal currently. If you experience strange results or error messages during development, deleting the cache directory is sometimes the necessary and sufficient cure. We hope to improve this in the future.

Config files let you define simple macros with the let key. The value of a macro can be a string or another JSON-permissible value (map, array, ...). You refer to a macro value in a job definition by using ${<macro_name>}.

  "let": {"MyApp" : "demobrowser"}
  ...
  "myjob" : { "settings" : {"qx.application" : "${MyApp}.Application"}}

If the value of the macro is a string you can use a reference to it in other strings, and the macro reference will be replaced by its value. You can have multiple macro references in one string. Usually, these macro references will show up in map values or array elements, but can also be used in map keys.

  "myjob" : {"${MyApp}.resourceUri" : "resource"}

If the value of the macro is something other than a string, things are a bit more restrictive. References to those macros can not be used in map keys (for obvious reasons). The reference has still to be in a string, but the macro reference has to be the only contents of that string. The entire string will then be replaced by the value of the macro. That means, you can do something like this:

  "let" : {"MYLIST" : [1,2,3], ...},
  "myjob" : { "joblist" : "${MYLIST}", ...}

and the “joblist” key will get the value [1,2,3].

A special situation arises if you are using a top-level let, i.e. a let section on the highest level in the config file, and not in any job definition. This let map will be automatically applied to every job run, without any explicit reference (so be aware of undesired side effects of bindings herein).

When assembling a job to run, the precedence of all the various let maps is

With imported jobs top-level definitions will take precedence over any definitions from the external config file (as if they were the ‘first’ let section in the chain).

Logging is an important part of any reasonably complex application. The Generator does a fair bit of logging to the console by default, listing the jobs it performs, adding details of important processing steps and reporting on errors and potential inconsistencies. The log key lets you specify further options and tailor the Generator console output to your needs. You can e.g. add logging of unused classes in a particular library/name space.

extend and run keywords are currently the only keywords that reference other jobs (This section might be renamed to “Scope of names”). One thing to note here is that job names are evaluated in the context of the current job map. As you will see (see section on top-level "include"s), a single configuration might eventually contain jobs from multiple config files, the local job definitions, and zero to many imported job maps (from other config files), which again might contain imported configs. From within any map, only those jobs are referenceable that are contained somewhere in this map. Unqualified names (like “myjob”) are taken to refer to jobs on the same level as the current job, path-like names (containing “/”) are taken to signify a job in some nested name space down from the current level. Particularly, this means you can never reference a job in a map which is “parallel” to the current job map. It’s only jobs on the same level or deeper.

This is particularly important for imported configs (imported with a top-level “include” keyword, see further down). Those configs get attached to the local “jobs” map under a dedicated key (their “name space” if you will). If in this imported map there is a “run” job (see the next section) using unqualified job names, these job names will be resolved using the imported map, not the top-level map. If the nested “run” job uses path-like job names, these jobs will be searched for relative to the nested map. You get it?!

Now, how exactly is a job (let’s call this the primary job) treated that says to “extend” another job (let’s call this the secondary job). Here is what happens:

• The primary job provides sort of the master definition for the resulting job. All its definitions take precedence.
• The secondary job is searched in the context of the current “jobs” map (see above).
• Keys of the secondary job that are not available in the primary job are just added to the job definition.
• Keys of the secondary job that are already present in the primary job are discarded.
• Important exception to this latter rule are the let, “library”, “variants” and settings keys: Here, it is not a complete either-or, but rather the same principles are applied one level deeper, on the value level of these keys. That is, existing keys in the “let” map of the primary job take precedence but new keys from the secondary “let” are added, and so for the other keys. Secondary “library” entries are prepended. That means, those keys accumulate all their inner keys over all jobs in the transitive “extend” hull of the primary job.
• Obviously, each secondary job is extended itself before being processed in this way, so it brings in its own full definition. As stated before it is important to note that this extending is done in the secondary job’s own context, which is not necessarily the context of the primary job.
• If there are more than one job in the “extend” list, the process is re-applied iteratively with all the remaining jobs in the list.

Important to note here: Macro evaluation takes place only after all extending has been done. That is, macros are applied to the fully extended job, making all macro definitions available that have accumulated along the way, with a ‘bottom-to-top-left-to-right’ precedence (macro definitions in the primary job take precedence over definitions in secondary jobs, and within the list of secondary jobs, the job more to the left wins over the job more to the right). That is, order in the “extend” list matters. But that also means that macros are not evaluated in the original context of the job. This makes it possible to tweak a job definition for a new environment, but can also be a source of surprises if you wanted to have some substitution taking place in the original config file, and realize it doesn’t.

Additionally to the above described features, with the configuration system you can

• create jobs in your local configuration with same names as those imported from another configuration file. The local job will take precedence and “shadow” the imported job; the imported job gets automatically added to the local job’s extend list.
• extend one job by another by only partially specifying job features. The extending job can specify only the specific parts it wants to re-define. The jobs will then be merged intelligently, giving precedence to local definitions of simple data types (strings, numbers, boolean, null) and combining complex values (maps and arrays); in the case of maps this is a deep merging process. Sample:

  "build-script" : {
    "compile-dist" : {
      "format" : "on"
    }
  }

• selectively block merging of features by using the = tag in front of the key name, like:

  ...
    {
      "=open-curly" : ...,
      ...
    }
  ...

• override an imported job entirely by guarding the local job with an override tag, as above, like:

  "jobs" : {
    "=build-script" : {...},
    ...
  }

“run” jobs are jobs that bear the run keyword. Since these are kind of meta jobs and ment to invoke a sequence of other jobs, they have special semantics. When a run keyword is encountered in a job, for each sub-job in the “run” list a new job is generated (so called synthetic jobs, since they are not from the textual config files). For each of those new jobs, a job name is auto-generated using the initial job’s name as a prefix. As for the contents, the initial job’s definition is used as a template for the new job. The extend key is set to the name of the current sub-job (it is assumed that the initial job has been expanded before), so the settings of the sub-job will eventually be included, and the “run” key is removed. All other settings from the initial job remain unaffected. This means that all sub-jobs “inherit” the settings of the initial job (This is significant when sub-jobs evaluate the same key, and maybe do so in a different manner).

In the overall queue of jobs to be performed, the initial job is replaced by the list of new jobs just generated. This process is repeated until there are no more “run” jobs in the job queue, and none with unresolved “extend”s.

(since 0.8.1)

The asset-let key is basically a macro definition for #asset compiler hints, but with a special semantics. Keys defined in the “asset-let” map will be looked for in #asset hints in source files. Like with macros, references have to be in curly braces and prefixed with $. So a “asset-let” entry in the config might look like this:

  "asset-let" :
  {
    "qx.icontheme" : ["Tango", "Oxygen"],
    "mySizes" : ["16", "32"]
  }

and a corresponding #asset hint might use it as:

#asset(qx/icon/${qx.icontheme}/${mySizes}/*)

The values of these macros are lists, and each reference will be expanded into all possible values with all possible combinations. So the above asset declaration would essentially be expanded into:

#asset(qx/icon/Tango/16/*)
#asset(qx/icon/Tango/32/*)
#asset(qx/icon/Oxygen/16/*)
#asset(qx/icon/Oxygen/32/*)

(until 0.8)

The themes key is basically a macro definition for #asset compiler hints, but with a special semantics. Keys defined in the “themes” map will be looked for in #asset hints in source files. Like with macros, references have to be in curly braces and prefixed with $. So a “themes” entry in the config might look like this:

  "themes" :
  {
    "qx.icontheme" : ["Tango", "Oxygen"],
    "mySizes" : ["16", "32"]
  }

and a corresponding #asset hint might use it as:

#asset(qx/icon/${qx.icontheme}/${mySizes}/*)

The values of these macros are lists, and each reference will be expanded into all possible values with all possible combinations. So the above asset declaration would essentially be expanded into:

#asset(qx/icon/Tango/16/*)
#asset(qx/icon/Tango/32/*)
#asset(qx/icon/Oxygen/16/*)
#asset(qx/icon/Oxygen/32/*)

The library key of a configuration holds information about source locations that will be considered in a job (much like the CLASSPATH in Java). Each element specifies one such library. The term “library” is meant here in the broadest sense; everything that has a qooxdoo class structure with source code can be considered a library in this context. This includes applications like the Showcase or the Feedreader, add-ins like the Testrunner or the Apiviewer, contribs from the qooxdoo-contrib repository like the Inspector, or of course the qooxdoo framework library itself. The main purpose of any such library entry is to provide the path to its “Manifest” file.

Manifest files serve to provide meta information for a library in a structured way. Their syntax is again JSON, and part of them is read by the generator, particularly the provides section. See here for more information about manifest files.

Contributions can be included in a configuration like any other libraries: You add an appropriate entry in the library array of your configuration. Like other libraries, the contribution must provide a Manifest.json file with appropriate contents.

If the contribution resides on your local file system, there is actually no difference to any other library. Specify the relative path to its Manifest file and you’re basically set. The really new part comes when the contribution resides online, in the qooxdoo-contrib repository. Then you use a special syntax to specify the location of the Manifest file. It is URL-like with a contrib scheme and will usually look like this:

contrib://<ContributionName>/<Version>/<ManifestFile>

The contribution source tree will then be downloaded from the repository, the generator will adjust to the local path, and the contribution is then used just like a local library. A consideration that comes into play here is the question where to put the files locally:

• The default location is a subdirectory from your project folder named cache-downloads. This has to be reflected in the uri parameter of the library entry.

So, for example an entry for the “trunk” version of the “HtmlArea” contribution with default download location would look like this:

{
  "manifest" : "contrib://HtmlArea/trunk/Manifest.json",
  "uri"          : "../cache-downloads/HtmlArea/trunk"
}

• Mind that the uri parameter reflects the path from your application’s index.html to the local root directory of the contribution (wherever that is in your particular case).

• You can configure a different download directory using the cache/downloads key. If this key is specified the given path will be used as root directory for the downloaded contribs. Again, remember to reflect this path in the uri key of your contrib library entry.

URIs are used in a qooxdoo application to refer from one part to other parts like resources. There are places within the generator configuration where you can specify uri parameters. What they mean and how this all connects is explained in this section.

The first important thing to note is:

Within qooxdoo the library is a fundamental concept, and libraries in this sense contain all the things you are able to include in the final Web application, such as class files (.js), graphics (.gif, .png, ...), static HTML pages (.htm, .html), style sheets (.css), and translation files (.po).

But not all of the above resource types are actually referenced through URIs in the application. Among those that are you find in the source version:

• references to class files
• references to graphics
• references to static HTML
• references to style sheet files

The build version uses a different approach, since it strives to be a self-contained Web application that has no outgoing references. Therefore, all necessary resources are copied over to the build directory tree. Having said that, URIs are still used in the build version, yet these are only references confined to the build directory tree:

• JS class code is put into the (probably various) output files of the generator run (what you typically find under build/script/*.js). The bootstrap file references the others with relative URIs.
• Graphics and other resources are referenced with relative URIs from the compiled scripts. Those resources are typically found under the build/resource path.
• Translation strings and CLDR information are directly included in the bootstrap file and thus need not be referenced through URIs.

So, in summary, in the build version some references are resolved by directly including the specific information, while the remaining references are usually confined to the build directory tree. This way, you can just pack it up and copy it to your web server for deployment. The source version is normally only used during development and the application can usually be tested in the browser directly off of the file system. Only in cases where you e.g. need to include interaction with a backend you will want to run the source version from a web server environment. For those cases the following details will be especially interesting. Others might want to skip the remainder of this section for now.

Although the scope and relevance of URIs vary between source and build versions, the underlying mechanisms are the same in both cases, with the special twist that when creating the build version there is only a single “library” considered, the build tree itself, which suffices to get all the URIs out fine. These mechanisms are described next.

So how does the generator create all of those URIs in the final application code? All those URIs are constructed through the following three components:

to_libraryroot  + library_internal_path + resource_path
       [1]                [2]                 [3]

So for example a graphics file in the qooxdoo framework might get referenced using the following components “../../qooxdoo-0.8-sdk/framework/” [1] + “source/resource/qx/” [2] + “icon/Oxygen/16/actions/go-up.png” [3], to produce the entailing URI “../../qooxdoo-0.8-sdk/framework/source/resource/qx/icon/Oxygen/16/actions/go-up.png”. These general parts have the following meaning:

• [1] : URI path to the library root (as will be valid when running the app in the browser); this is directly the value given in the uri parameter of the library’s entry.
• [2] : path segment within the specific library; most of this is governed by convention, but can be tailored in the library’s Manifest.json. The consumer of the library has no influence on it.
• [3] : path segment leading to the specific resource; this is gleaned from scanning the library’s resource directory tree.

Part [1] is exactly what you specify with the uri subkey of an entry in the library key list. This should be fine for most applications. But be aware of the following consequence:

This is usually fine, as long as you don’t have different autonomous parts in your application using the same library from different directories. (See also further down).

Alternatively to specifying the uri key explicitly, a sensible value for [1] can be calculated by the generator using the following information:

applicationroot_to_configdir + configdir_to_libraryroot
            [1.1]                      [1.2]

The parts have the following meaning:

• [1.1] : path from the Web application’s root to the configuration file’s directory; this information is derived from the root key of the compile-source config key.
• [1.2] : path from the configuration file’s directory to the root directory of the library (the one containing the Manifest.json file); this information is derived from the library’s manifest key.

For the compile-dist config key, dedicated script and resource keys are available. The values of these keys cover the scope of [1] + [2] in the first figure.

Since [1.2] is always known (otherwise the whole library would not be found), only [1.1] has to be given in the config. The features of this approach, in contrast to specifying [1], are:

• The application root can be specified individually for each compile job. This means you could have more than one application root in your project, e.g. when your main application offers an iframe, into which another application from the same project is loaded; qooxdoo’s Demobrowser application takes advantage of exactly that.
• Relative file system paths have to match with relative URIs in the running application. So this approach won’t work if e.g the relative path from your config directory to the library makes no sense when the app is run from a web server.

From the above discussion, there is one important point to take away, in order to create working URIs in your application:

While either are optional in their respective contexts, it is mandatory to at least specify one of them for the URI generation to work.

Libraries you specify in your own config are in your hand, and you can provide uri parameters as you see fit. If you want to tweak the “uri” setting of a library entry that is added by including another config file, you simply re-define the library entry of that particular library locally. The generator will realize that both entries refer to the same library, and your local settings will take precedence.

You can specify library keys in your own config in these ways:

• You either define a local job which either shaddows or “extends” an imported job, and provide this local job with a library key. Or,
• You define a local “libraries” job and provide it with a “library” key. This job will be used automatically by most of the standard jobs (source, build, etc.), and thus your listed libraries will be used in multiple jobs (not just one as above).

Packages are a concept that allows you to partition your application physically. The idea is to spread the entire application over multiple JavaScript files, in order to optimize download and startup behaviour. On page load only the essential first part of the application is loaded, while others remain on the server and will only be loaded on demand. As a consequence, the initial code part is smaller, so it’s faster to download, consumes less bandwidth and starts up faster in the browser. Other parts are then loaded on demand during the user session. This incures a bit of latency when the user enters a certain application path for the first time and the correpsonding part has to be loaded. On the other side, parts that pertain to a certain application path (e.g. an options dialogue) never have to be downloaded, if this application path is not entered during the running session.

In order to realize this concept, you have the option to specify parts of your application, while the build process takes care of mapping these (logical) parts to physical packages that are eventually written to disk. At run time of your application, the inital package will contain loader intelligence that knows when to download further parts. There are two different but related terms here: You as a user define parts of your application. These are logical or visual related elements, like all elemens that make up a complex dialogue, or the contents of an interactive tab pane. The build process then figures out all the dependencies of these parts and collects them into packages. Since some parts might have overlapping dependencies, these are optimized so that they are not included twice in different packages. Also, you might want to specify which classes should remain together in a common package, how small packages might minimally get, and so forth. So you define the logical partitioning of your application and specify some further constraints, and the build process will take care of the rest, producing the best physical split of the entire app under the given constraints.

Consequently, the config currently allows you to specify those logical parts of your application by giving a suitable name to each and listing the top-level classes or class patterns for each. You can then list parts you want to collaps into a single physical package. With the size key you require that the smallest package be no less than that many kilobytes of size; if packages become smaller they will be suitably merged with others. This way, you can limit fragmentation of your application and decrease the effect of network latency when loading the application. See the packages key reference section for the nitty-gritty.

Within qooxdoo there are a couple of features that are not so much applications although they share a lot of the classical application structure. The APIViewer and TestRunner are good examples for those. (In the recent repository re-org, they have been filed under component correspondingly). They are applications but receive their actual meaning from other applications: An APIViewer in the form of class documentation it presents, the TestRunner in the form of providing an environment to other application’s test classes. On their own, both applications are “empty”, and the goal is it to use them in the context of another, self-contained application. The old build system supported make targets like ‘api’ and ‘test’ to that end.

While you can always include other applications’ classes in your project (by adding an entry for them to the library key of your config), you wouldn’t want to repeat all the necessary job entries to actually build this external app in your environment. So the issue here is not to re-use classes, but jobs.

So, the general issue we want to solve is to import entire job definitions in our local configuration. The next step is then to make them work in the local environment (e.g. classes have to be compiled and resources be copied to local folders). This concepts is fairly general and scales from small jobs (where you just keep their definition centrally, in order to use them in multiple places) to really big jobs (like e.g. creating a customized build version of the Apiviewer in your local project).

Practically, there are two steps involved in using external jobs:

1. You have to include the external configuration file that contains the relevant job definitions. Do so will result in the external jobs being added to the list of jobs of your local configuration. E.g. you can use

   generator.py ?

   to get a list of all available jobs; the external jobs will be among this list.

2. There are now two way to utilize these jobs:
   • You can either invoke them directly from the command line, passing them as arguments to the generator.
   • Or you define local jobs that extend them.

In the former case the only way to influence the behaviour of the external job is through macros: The external job has to parameterize its workings with macro references, you have to know them and provide values for them that are suitable for your environment (A typical example would be output paths that you need to customize). Your values will take precendence over any values that might be defined in the external config. But this also means you will have to know the job, know the macros it uses, provide values for them (e.g. in the global let of your config), resolve conflicts if other jobs happen to use the same macros, and so forth.

In the latter case, you have more control over the settings of the external job that you are actually using. Here as well, you can provide macro definitions that parameterize the behaviour of the job you are extending. But you can also supply more job keys that will either shaddow the keys of the same name in the external job, or will be extended by them. In any case you will have more control over the effects of the external job.

Add-ins use exactly these mechanisms to provide their functionality to other applications (in the sense as ‘make test’ or ‘make api’ did it in the old system). Consequently, to support this in the new system, the add-in applications (or more precisely: their job configuration) have to expose certain keys and use certain macros that can both be overridden by the using application. The next sections describe these build interfaces for the various add-in apps. But first more practical detail about the outlined ...

In order to include an add-in feature in an existing app, you first have to include its job config. On the top-level of the config map, e.g. specify to include the Apiviewer config:

"include" : [{"path": "../apiviewer/config.json"}]

The include key on this level takes an array of maps. Each map specifies one configuration file to include. The only mandator key therein is the file path to the external config file (see here for all the gory details). A config can only include what the external config is willing to export. Among those jobs the importing config can select (through the import key) or reject (through the block key) certain jobs. The resulting list of external job definitions will be added to the local jobs map.

If you want to fine-tune the behaviour of such an imported job, you define a local job that extends it. Imported jobs are referenced like any job in the current config, either by their plain name (the default), or, if you specify the as key in the include, by a composite name <as_value>::<original_name>. Suppose you used an “as” : “apiconf” in your include, and you wanted to extend the Apiviewer’s build-script job, this could look like this:

"myapi-script" :
{
  "extend" : ["apiconf::build-script"]
  ...
}

As a third step, the local job will usually have to provide additional information for the external job to succeed. Which exactly these are depends on the add-in (and should eventually be documented there). See the section specific to the APIViewer for a concrete example.

For brevity, let’s jump right in into a config fragment that has all necessary ingredients. These are explained in more detail afterwards.

{
  "include" : [{"as" : "apiconf", "path" : "../apiviewer/config.json"}],
  "jobs" : {
    "myapi" : {
        "extend" : ["apiconf::build"],
        "let" : {
            "ROOT"  :  "../apiviewer",
            "BUILD_PATH" : "./api",
            "API_INCLUDE" : ["qx.*", "myapp.*"],
            "API_EXCLUDE" : ["myapp.tests.*"]
            },
        "library" : { ... },
        "settings" : {
            "myapp.resourceUri" : "./resource"
            }
        }
    }
}

The myapi job extends the build job of APIViewer’s job config. This “build” job is itself a run job, i.e. it will be expanded in so many individual jobs as its run key lists. All those jobs will get the “myapi” job as a context into which they are expanded, so all other settings in “myapi” will be effective in those jobs.

In the let key, the ROOT, BUILD_PATH, API_INCLUDE and API_EXCLUDE macros of the APIViewer config are overridden. This ensures the APIViewer classes are found, can be processed, and the resulting script is put into a local directory. Furthermore, the right classes are included in the documentation data.

The library key has to at least add the entry for the current application, since this is relevant for the generation of the api documentation for the local classes.

So in short, the ROOT, BUILD_PATH, API_INCLUDE and API_EXCLUDE macros define the interface between the apiviewer’s “run” job and the local config.

The optimize key is a subkey of the compile-dist key. It allows you to tailor the forms of code optimization that is applied to the Javascript code when the build version is created. Currently, there are four categories which can be optimized.

With string optimization, strings are extracted from the class definition and put into lexical variables. The occurrences of the strings in the class definition is then replaced by the variable name. This mainly benefits IE6 and repetitive references to the same string literal.

Long variable names are made short. Lexical variables (those declared with a var statement) are replaced by generated names that are much shorter (1-2 characters on average). Dependending on the original code, this can result in significant space savings.

This is less an optimization in space or time, but rather a way to enforce privates. Private members of a class (those beginning wit "__") are replaced with generated names, and are substituted throughout the class. If some other class is accessing those privates, these references are not updated and will eventually fail when the access happens. This will lead to a runtime error.

Calls to this.base(), which invoke the corresponding superclass method, are inlined, i.e. the superclass code is inserted in place of the this.base() call.