Benchmarking Using the Frame Rate Test

About the X-Plane Framerate Test

X-Plane contains a built-in framerate test mode. In this mode the sim ignores preferences and loads a fully known and controllable configuration, runs some framerate tests (with user interaction disabled, e.g. no dialog boxes will block operation) and quits. Information is output to the Log.txt file.

The framerate test is aimed at two audiences:

• It allows Laminar Research to gather performance information from beta users’ machines without the risk of configuration error.
• It allows driver writers to run X-Plane as part of an automated regression system.

This document is written for programmers who want to use X-Plane in a production environment, like an automated regression harness.

FPS Test Options And Commands In Detail

The framerate test is controlled by command-line options. Please note that options and preferences discussed here are accurate for RC3 but may be subject to change in the future.

Running a FPS Test

To run a framerate test, you use the command-line option:

‐‐fps_test=N

Where N is a numeric code indicating the FPS test to be run. The FPS tests are pre-built settings.

The basic framerate test has two modes, depending on the file type you pass it:

• For X-Plane “movies” (.smo files), the replay will run for 90 seconds, in 3 stages (each of which will be output its results individually to your Log.txt file):
  • 30 seconds of a forward view with the panel.
  • 30 seconds of a forward view with no panel.
  • 30 seconds with a forward view, no panel, simulation paused.
• For flight data recorder (.fdr) files, the replay will run to completion in whichever view you specify (see “FPS Test Codes” below). At the end, your Log.txt will contain a single summary of the replay’s performance. You can add the ‐‐verbose flag to get output for every frame, if you’d like to run a more sophisticated statistical analysis on the data than just getting the mean frame rate.

FPS Test Codes

Currently the framerate test numbers are built from three digits (leading zeros can be omitted):

• Hundreds digits: Viewpoint. Supported values are:
  • 0: default (cockpit view)
  • 1: view from above
  • 2: nighttime cockpit view
• Tens digits: Weather preset. Supported values are:
  

  Value	Clouds	Visibility (miles)	Rain
  0	Clear	25
  1	Light cumulus @ 2 km; cirrus @ 5 km	25
  0	Clear	15
  3	Broken cumulus @ 2 km; cirrus @ 5 km	10
  4	Stratus @ 1 km; cirrus @ 5 km	3	Moderate
  5	Overcast @ 500 m; cirrus @ 5 km	1	Heavy
  6	Overcast @ 500 m; cirrus @ 5 km	0.8	Heavy
  7	Overcast @ 500 m; stratus @ 5 km	0.8	Light

• Ones digits: Rendering preset.
  • Supported values are:
    • 1: low
    • 2: medium
    • 3: high
    • 4: very high
    • 5: extreme
  • This controls a range of rendering settings; use individual preferences for more control (see below). Note that 1 gives you “high” tex res (4x reduction in all dimensions, 2 gives you “very high” (2x reduction) and 3 gives you “extreme” (no reduction). This is useful because the _tex_res_ pref seems to not be command-line controllable.

Note that the fly-over interferes with some views—see preferences below.

Minimum FPS Mode

X-Plane can also be run in minimum-fps mode. In this mode, the sim will run one 30 second test (with flight model and panel on) and then return 1 on success or 0 on failure. Framerate is not logged. Typically this would be used to fail an automated regression test, e.g.

./X-Plane-i686 ‐‐fps_test=1 ‐‐require_fps=30 || echo "X-Plane is running slower than 30 fps."

Note that you must use ‐‐require_fps with ‐‐fps_test.

Playing Replay Movies

Normally the fps test will simply leave the plane on the runway; however you can also program X-Plane to load and play a “replay movie” (.smo file) – this is a binary file containing a replay of an X-Plane flight. Example:

./X-Plane-i686 ‐‐fps_test=1 ‐‐load_smo=Output/movies/test.smo

The .smo or .fdr file is a relative path from the root of the X-Plane folder. The timedemo test contains one movie, test.smo.

The fps test will run for 90 seconds (or 30 seconds for ‐‐require_fps) regardless of movie length; you should set your movie to about 90 seconds. Note that the .smo file format is fixed-size; you will not save disk space with shorter movies. (But shorter movies will zip more efficiently.)

One use of replay movies is to vary the viewpoint (by flying the plane) to get a more representative rendering load.

Controlling Individual Settings

You can override individual settings using the ‐‐pref command; the syntax is:

‐‐pref:=

For example:

./X-Plane-i686 ‐‐fps_test=1 ‐‐pref:_x_res_req_ALL=1440 ‐‐pref:_y_res_req_ALL=900

will run the sim with fps test 1 but at 1440×900 res.

Preference names are strings; you can find them by viewing the contents of the Resources/preferences folder (after running the sim normally to init preferences).

• Res X-System.prf contains a few resolution settings, global to all apps.
• Set X-Plane.prf contains most non-binary X-Plane prefs that can be set.

Here are some settings that are useful for testing X-Plane performance. Note that I have not tested all of these; this comes from dumping a prefs file to the terminal.

WARNING: if you use a fps test number, it will override settings set by individual preferences. When the FPS test is used, you can only control resolution-related preferences.

Note: in X-Plane patches before 941, _alias_req will change FSAA but will not correctly show the results in the UI when ‐‐fps_test is in use.

Changes in Settings For 945

Window Management Changes

To use 945 in Windowed mode. (Note: window must be no larger than the size of the primary screen)

‐‐pref:_x_res_wind_ALL= ‐‐pref:_y_res_wind_ALL=

To use 940 in full screen mode at current monitor settings:

‐‐pref:_is_full_ALL=1 ‐‐pref:_x_res_full_ALL=0 ‐‐pref:_y_res_full_ALL=0

To use 940 in full screen mode at a specific res:

‐‐pref:_is_full_ALL=1 ‐‐pref:_x_res_full_ALL=1024 ‐‐pref:_y_res_full_ALL=768 ‐‐pref:_bpp_full_ALL=32

Note: X-Plane will not change the monitor res unless it can determine that the selected res matches a published resolution from the OS/driver. Usually this means supplying a bits per pixel (bpp) of 32.

However, X-Plane does _not_ actually put the device into full screen mode, so there is no advantage to running “full screen” vs. running in a window with the window size set to the monitor size.

Replay Movies

In X-Plane 945, replays are called .rep files and live in Output/replays/. Unlike the 900 time demo, the replay will play from start to end and the time demo will end with one framerate count output.

Custom Log Paths

The command-line option ‐‐log_path= will redirect the output file Log.txt to another directory. The very beginning of the file will end up in the x-plane dir (if writable) but most of the file, including all of the interesting parts) will be redirected. The path must be an absolute path including file name, e.g.

./X-Plane-i686 ‐‐log_path=/my_dir/log.txt

Changes for Version 10

Version 10 keeps the version 945 screen res format. Besides all v9 features, v10 allows the settings from “Resources/settings.txt” to be overridden using the ‐‐ren:XXXX=YYY syntax. An example:

X-Plane_NODEV_OPT.app/Contents/MacOS/X-Plane_NODEV_OPT ‐‐fps_test=2 ‐‐load_smo=Output/replays/test_flight.rep \
‐‐pref:_is_full_ALL=1 ‐‐pref:_x_res_full_ALL=1024 ‐‐pref:_y_res_full_ALL=768 ‐‐pref:_bpp_full_ALL=32 \
‐‐ren:draw_HDR=1

X-Plane 10 contains a config file in the Resources folder called “settings.txt”. The line

SETTING category type name max

defines the setting parameter,s e.g

SETTING EFFECTS SLIDER shadow_quality 9

means that the shadow_quality setting will show as a slider, with a min of 0 and a max of 9. You can change these settings from the cmd line, e.g.

‐‐ren:shadow_quality=5

overriding the defaults from fps tests. You can use the command line to rapidly find all settings like this:

grep “^SETTING ” resources.settings.txt

The time demo is also data driven – you can view the default settings for each of the rendering settings as follows:

grep DEMO Resources/settings.txt

The TIMEDEMO lines define each time demo (1-X) and the DEMO_SETTING shows the deafults used.)

Note that the time demos affect the rendering settings in settings.txt, which are mostly engine related, but _do not_ set raw windowing factors like resolution and full screen anti-aliasing. Thus you should expect to use ‐‐pref to customize FSAA and use monitor controls to get the res you want.

As a general rule:

• EFFECTS settings tend to push the GPU harder, e.g. consume more shading power.
• DRAWING settings tend to push bus bandwidth and the CPU harder, e.g. they result in more geometry.
• EXPERT settings should be left at their default, under all conditions.

QA Scripts and Visual Output

X-Plane 945 features a simple script file engine to produce conformance tests. To run a test file, use the command-line ‐‐qa_script=. The X-Plane 945 time demo comes with a sample script (conformance.txt) that can be used for baseline driver testing.

The script commands consist of a series of single-line commands, with white space and lines starting with # as comments.

QA script commands

WAIT

causes the script to pause, letting the sim run, for a number of seconds.

NEXT_FRAME

causes the sim to render the next frame. When settings are changed, you will need to either wait or go to the next frame to see the changes.

CMND

executes one of X-Plane’s built in commands – any command that can be tied to a joystick or key binding can be executed by script. To find the commands, look in Resources/plugins/commands.txt

DREF

sets a sim dataref to a given value. Look in Resources/plugins/datarefs.txt for a list of datarefs.

LOOK

causes the sim to change the camera to be placed at x,y,z and look in a given heading. You can use the data outut screen to determine the current camera position for writing scripts.

RELOAD

causes scenery to reload – use this after changing settings to change scenery.

TIMEER_START
TIMER_STOP

measures an interval at a given framerate. Use this to measure fps for a given camera angle and framerate. The tag cannot have whitespace.

PUSH_APP_PATH POP_APP_PATH

Changes the current directory for the script to a new path (path should be a global path starting with /) for the purpose of taking pictures. Popping the path sets it back to the previous path.

Back to Knowledge Base