FEX

NAME
FEX FEXBash - Fast x86-64 and x86 emulation.

SYNOPSIS
FEX <args> ...

FEXBash <args> ...

DESCRIPTION
FEX allows you to run x86 and x86-64 binaries on an AArch64 host, similar to qemu-user and box86. It has native support for a rootfs overlay, so you don't need to chroot, as well as some thunklibs so it can forward things like GL to the host. FEX presents a Linux 5.0 interface to the guest, and supports both AArch64 and x86-64 as hosts. FEX is very much work in progress, so expect things to change.

ENVIRONMENT
FEX_MULTIBLOCK

Controls multiblock code compilation

Can cause long JIT compilation times and stutter

JSON key: 'Multiblock'

default: true

FEX_MAXINST

Maximum number of instruction to store in a block

JSON key: 'MaxInst'

default: 5000

FEX_ENABLECODECACHINGWIP

Enable the code caching subsystem

JSON key: 'EnableCodeCachingWIP'

default: false

FEX_ENABLECODECACHEVALIDATION

Enable expensive validation when loading code caches

JSON key: 'EnableCodeCacheValidation'

default: false

FEX_HOSTFEATURES

Allows controlling of the CPU features in the JIT.

off: Default CPU features queried from CPU features

{enable,disable}sve: Will force enable or disable sve even if the host doesn't support it

{enable,disable}avx: Will force enable or disable avx even if the host doesn't support it

{enable,disable}afp: Will force enable or disable afp even if the host doesn't support it

{enable,disable}lrcpc: Will force enable or disable lrcpc even if the host doesn't support it

{enable,disable}lrcpc2: Will force enable or disable lrcpc2 even if the host doesn't support it

{enable,disable}cssc: Will force enable or disable cssc even if the host doesn't support it

{enable,disable}pmull128: Will force enable or disable pmull128 even if the host doesn't support it

{enable,disable}rng: Will force enable or disable rng even if the host doesn't support it

{enable,disable}clzero: Will force enable or disable clzero even if the host doesn't support it

{enable,disable}atomics: Will force enable or disable ARMv8.1 LSE atomics even if the host doesn't support it

{enable,disable}fcma: Will force enable or disable fcma even if the host doesn't support it

{enable,disable}flagm: Will force enable or disable flagm even if the host doesn't support it

{enable,disable}flagm2: Will force enable or disable flagm2 even if the host doesn't support it

{enable,disable}crypto: Will force enable or disable crypto extensions even if the host doesn't support it

{enable,disable}rpres: Will force enable or disable rpres even if the host doesn't support it

{enable,disable}svebitperm: Will force enable or disable svebitperm even if the host doesn't support it

{enable,disable}preserveallabi: Will force enable or disable preserve_all abi even if the host doesn't support it

{enable,disable}wfxt: Will force enable or disable wfxt even if the host doesn't support it

{enable,disable}3dnow: Will force enable or disable 3DNow! even if the host doesn't support it

{enable,disable}sse4a: Will force enable or disable SSE4a even if the host doesn't support it

{enable,disable}mops: Will force enable or disable FEAT_MOPS even if the host doesn't support it

JSON key: 'HostFeatures'

default: 'FEXCore::Config::HostFeatures::OFF'

Available Options: enablesve, disablesve, enableavx, disableavx, enableafp, disableafp, enablelrcpc, disablelrcpc, enablelrcpc2, disablelrcpc2, enablecssc, disablecssc, enablepmull128, disablepmull128, enablerng, disablerng, enableclzero, disableclzero, enableatomics, disableatomics, enablefcma, disablefcma, enableflagm, disableflagm, enableflagm2, disableflagm2, enablefrintts, disablefrintts, enablecrypto, disablecrypto, enablerpres, disablerpres, enablesvebitperm, disablesvebitperm, enablepreserveallabi, disablepreserveallabi, enablewfxt, disablewfxt, enable3dnow, disable3dnow, enablesse4a, disablesse4a, enablemops, disablemops

FEX_SMALLTSCSCALE

Scales the cycle counter on systems that have low frequencies.

JSON key: 'SmallTSCScale'

default: true

FEX_HIDEHYBRID

Hides hybrid CPU core arrangement.

JSON key: 'HideHybrid'

default: true

FEX_CPUFEATUREREGISTERS

Allows overriding cpu feature flags for manual testing

JSON key: 'CPUFeatureRegisters'

default: ''

FEX_ROOTFS

Which Root filesystem prefix to use

This can be a filesystem path

eg: ~/RootFS/Debian_x86_64

Or this can be a name of a rootfs

If the named rootfs exists in the FEX data folder then it will use that one

eg: $XDG_DATA_HOME/fex-emu/RootFS/<RootFS name>/

If XDG_DATA_HOME is unset, ~/.local/share will be used in its place.

eg: $HOME/.local/share/fex-emu/RootFS/<RootFS name>/

JSON key: 'RootFS'

default: ''

FEX_THUNKHOSTLIBS

Folder to find the host-side thunking libraries.

JSON key: 'ThunkHostLibs'

default: '/usr/lib/fex-emu/HostThunks'

FEX_THUNKGUESTLIBS

Folder to find the guest-side thunking libraries.

JSON key: 'ThunkGuestLibs'

default: '/usr/share/fex-emu/GuestThunks'

FEX_THUNKCONFIG

A json file specifying where to overlay the thunks.

This can be a filesystem path

eg: ~/MyThunkConfig.json

Or this can be a named of a Thunk config file

If the named config file exists in the FEX data folder folder the it will use that one

eg: $XDG_DATA_HOME/fex-emu/ThunkConfigs/<ThunkConfig name>

If XDG_DATA_HOME is unset, ~/.local/share will be used in its place.

eg: $HOME/.local/share/fex-emu/ThunkConfigs/<ThunkConfig name>

JSON key: 'ThunkConfig'

default: ''

FEX_ENV

Adds an environment variable to the emulated environment.

JSON key: 'Env'

default: ''

FEX_HOSTENV

Adds an environment variable to the host environment.

This can be useful for setting environment variables that thunks can pick up.

Typically isn't necessary since the guest libc isn't thunked. But is possible.

JSON key: 'HostEnv'

default: ''

FEX_ADDITIONALARGUMENTS

Allows the user to pass additional arguments to the application

JSON key: 'AdditionalArguments'

default: ''

FEX_DISABLEL2CACHE

Disables FEXCore's JIT L2 cache lookup. Saving memory.

Can potentially introduce more stutters.

JSON key: 'DisableL2Cache'

default: true

FEX_DYNAMICL1CACHE

Switches FEXCore's JIT L1 cache to be dynamically sized. Saving memory.

Can potentially introduce more stutters.

JSON key: 'DynamicL1Cache'

default: true

FEX_DYNAMICL1CACHEINCREASECOUNTHEURISTIC

Threshold of lookups per second that the L1 dynamic cache should increase its size.

Lower numbers means more aggressive scaling upward to the maximum size.

Higher numbers means more conservative scaling, using less memory.

Can potentially introduce stutters, more likely the higher the number.

Don't have this number smaller than the decrease count!

JSON key: 'DynamicL1CacheIncreaseCountHeuristic'

default: 250

FEX_DYNAMICL1CACHEDECREASECOUNTHEURISTIC

Threshold of lookups per second that the L1 dynamic cache should decrease its size.

The higher the number, the more aggressively it reduces the L1 cache size.

Lower numbers means more conservative memory savings.

Can potentially introduce more stutters, more likely the higher the number.

Don't have this number larger than the increase count!

JSON key: 'DynamicL1CacheDecreaseCountHeuristic'

default: 50

FEX_SINGLESTEP

Single stepping configuration.

JSON key: 'SingleStep'

default: false

FEX_GDBSERVER

Enables the GDB server.

JSON key: 'GdbServer'

default: false

FEX_DUMPIR

Folder to dump the IR in to.

[no, stdout, stderr, server, <Folder>]

JSON key: 'DumpIR'

default: 'no'

FEX_PASSMANAGERDUMPIR

Allows controlling when FEX dumps its IR.

off: IR dumping will be disabled

beforeopt: Dump IR before any optimizations

afteropt: Dump IR after all optimizations

beforepass: Dump IR before every optimization pass

afterpass: Dump IR after every optimization pass

JSON key: 'PassManagerDumpIR'

default: 'FEXCore::Config::PassManagerDumpIR::OFF'

Available Options: beforeopt, afteropt, beforepass, afterpass

FEX_DUMPGPRS

When the test harness ends, print the GPR state.

JSON key: 'DumpGPRs'

default: false

FEX_O0

Disables optimizations passes for debugging.

JSON key: 'O0'

default: false

FEX_GLOBALJITNAMING

Uses JITSymbols to name all JIT state as one symbol

Useful for querying how much time is spent inside of the JIT

Profiling tools will show JIT time as FEXJIT

JSON key: 'GlobalJITNaming'

default: false

FEX_LIBRARYJITNAMING

Uses JITSymbols to name JIT symbols grouped by library

Useful for querying how much time is spent in each guest library

Can be used to help guide thunk generation

JSON key: 'LibraryJITNaming'

default: false

FEX_BLOCKJITNAMING

Uses JITSymbols to name JIT symbols

Useful for determining hot blocks of code

Has some file writing overhead per JIT block

JSON key: 'BlockJITNaming'

default: false

FEX_GDBSYMBOLS

Integrates with GDB using the JIT interface.

Needs the fex jit loader in GDB, which can be loaded via `jit-reader-load libFEXGDBReader.so.`

Also needs x86_64-linux-gnu-objdump in PATH.

Can be very slow.

JSON key: 'GDBSymbols'

default: false

FEX_INJECTLIBSEGFAULT

Sets the environment variable LD_PRELOAD=libSegFault.so

This allows the user to very easily enable libSegFault without dealing with environment variables

Very useful for applications that have launch scripts that set the variable to nothing at launch

Set this in an application configuration for injecting in to only specific applications.

Note: If x86/x86_64 libSegFault.so isn't installed then this option won't work.

JSON key: 'InjectLibSegFault'

default: false

FEX_DISASSEMBLE

Allows controlling of the vixl disassembler for generated ARM code.

off: No disassembly will be output

dispatcher: Will enable disassembly of the JIT dispatcher loop

blocks: Will enable disassembly of the translated instruction code blocks

stats: Will print stats when disassembling the code

JSON key: 'Disassemble'

default: 'FEXCore::Config::Disassemble::OFF'

Available Options: dispatcher, blocks, stats

FEX_X86DISASSEMBLE

Enables x86/x86-64 guest disassembly output for compiled blocks.

Requires FEX to be built with -DENABLE_ZYDIS=TRUE

JSON key: 'X86Disassemble'

default: false

FEX_FORCESVEWIDTH

Allows overriding the SVE width in the vixl simulator.

Useful as a debugging feature.

JSON key: 'ForceSVEWidth'

default: 0

FEX_DISABLETELEMETRY

Disables telemetry at runtime.

Useful for CI instcountCI mostly

JSON key: 'DisableTelemetry'

default: false

FEX_SILENTLOG

Disables logging

JSON key: 'SilentLog'

default: true

FEX_OUTPUTLOG

File to write FEX output to.

[stderr, server, <Filename>]

JSON key: 'OutputLog'

default: 'server'

FEX_TELEMETRYDIRECTORY

Redirects the telemetry folder that FEX usually writes to.

By default telemetry data is stored in {$FEX_APP_DATA_LOCATION,{$XDG_DATA_HOME,$HOME}/fex-emu/Telemetry/}

JSON key: 'TelemetryDirectory'

default: ''

FEX_PROFILESTATS

Enables FEX's low-overhead sampling profile statistics.

Requires a supported version of Mangohud to see the results

JSON key: 'ProfileStats'

default: false

FEX_ENABLEGPUVISPROFILING

Enables profiling when FEX was built with the gpuvis profiler backend.

JSON key: 'EnableGpuvisProfiling'

default: false

FEX_SMCCHECKS

Checks code for modification before execution.

none: No checks

mtrack: Page tracking based invalidation (default)

full: Validate code before every run (slow)

JSON key: 'SMCChecks'

default: mtrack

FEX_TSOENABLED

Controls TSO IR ops.

Highly likely to break any multithreaded application if disabled.

JSON key: 'TSOEnabled'

default: true

FEX_VECTORTSOENABLED

When TSO emulation is enabled, controls if vector loadstores should also be atomic.

JSON key: 'VectorTSOEnabled'

default: false

FEX_MEMCPYSETTSOENABLED

When TSO emulation is enabled, controls if memcpy and memset should also be atomic.

Only affects REP MOVS and REP STOS instructions

JSON key: 'MemcpySetTSOEnabled'

default: false

FEX_HALFBARRIERTSOENABLED

When TSO emulation is enabled, controls if unaligned loads and stores should be backpatched to half-barrier atomics.

Can be dangerous due to aligned loadstores through the same code now become non-atomic.

JSON key: 'HalfBarrierTSOEnabled'

default: true

FEX_STRICTINPROCESSSPLITLOCKS

Strict global lock when handling an unaligned atomic that crosses a 16-byte or cacheline granularity

This is required to ensure a split-lock doesn't tear inside the process

JSON key: 'StrictInProcessSplitLocks'

default: false

FEX_KERNELUNALIGNEDATOMICBACKPATCHING

When the kernel unaligned atomic handler is enabled, use backpatching to reduce kernel context switches.

JSON key: 'KernelUnalignedAtomicBackpatching'

default: true

FEX_VOLATILEMETADATA

Use volatile metadata in PE files to inform TSO instructions when available.

When metadata is unavailable falls back to the currently enabled TSO options.

JSON key: 'VolatileMetadata'

default: true

FEX_X87REDUCEDPRECISION

Emulates X87 floating point using 64-bit precision. This reduces emulation accuracy and may result in rendering bugs.

JSON key: 'X87ReducedPrecision'

default: false

FEX_STALLPROCESS

Forces a process to stall out on initialization

Useful for a process that keeps restarting and doesn't work

JSON key: 'StallProcess'

default: false

FEX_HIDEHYPERVISORBIT

Hides the hypervisor CPUID bit when set.

Should only be used for applications that have issues with this set.

JSON key: 'HideHypervisorBit'

default: false

FEX_STARTUPSLEEP

Sleeps the process at startup for a duration of seconds.

Useful if an application crashes too quickly to attach a debugger.

JSON key: 'StartupSleep'

default: 0

FEX_STARTUPSLEEPPROCNAME

Contrains the startup sleep to only apply to processes that match this name.

JSON key: 'StartupSleepProcName'

default: ''

FEX_MONOHACKS

Permits a hook-based SMC approach and smaller JIT blocks when mono is detected.

JSON key: 'MonoHacks'

default: true

FEX_SERVERSOCKETPATH

Override for a FEXServer socket path. Only useful for chroots.

JSON key: 'ServerSocketPath'

default: ''

FEX_NEEDSSECCOMP

Disables inline syscalls in order to support seccomp handling

JSON key: 'NeedsSeccomp'

default: false

FEX_EXTENDEDVOLATILEMETADATA

Configuration provided volatile metadata. Only implemented for WoW64/arm64ec.

Limited in its use but can be handy.

Extends on top of what Microsoft has for volatile metadata, but also supported for WoW64.

Colon delimited modules, then semi-colon delimited instructions, then comma delimited ranges

Default disables TSO in the module, unless instructions overlap the range

<module>;<offset begin>-<offset-end>,...;<instruction offset to force TSO>,...:<another>

examples:

* Disable TSO for a full module: Just provide the module name:

`hl2_linux`

* Disable TSO for a part of the module:

`hl2_linux;<offset begin>-<offset-end>`

* Disable TSO for a part of the module, but enable TSO for some instructions within the module

`hl2_linux;<offset begin>-<offset-end>;<instruction offset>,<instruction offset>`

* Disable TSO for multiple modules

`hl2_linux:libsdl2.so`

JSON key: 'ExtendedVolatileMetadata'

default: ''

FEX_APP_CONFIG_LOCATION

Allows the user to override where FEX looks for configuration files

By default FEX will look in ${XDG_CONFIG_HOME, $HOME/.config}/fex-emu/

This will override the full path

If FEX_PORTABLE is declared then relative paths are also supported

For FEX: Relative to the FEX binary

For WINE: Relative to %LOCALAPPDATA%

default: ''

FEX_APP_CONFIG

Allows the user to override where FEX looks for only the application config file

By default FEX will look in ${XDG_CONFIG_HOME, $HOME/.config}/fex-emu/Config.json

This will override this file location

One must be careful with this option as it will override any applications that load with execve as wellIf you need to support applications that execve then use FEX_APP_CONFIG_LOCATION insteadIf FEX_PORTABLE is declared then relative paths are also supported

For FEX: Relative to the FEX binary

For WINE: Relative to %LOCALAPPDATA%

default: ''

FEX_APP_DATA_LOCATION

Allows the user to override where FEX looks for data files

By default FEX will look in {$XDG_DATA_HOME, $HOME/.local/share}/fex-emu/

This will override the full path

This is the folder where FEX stores generated files like IR cache

default: ''

FEX_PORTABLE

Allows FEX to run without installation. Global locations for configuration and binfmt_misc are ignored.

For FEX on Linux:

These files are instead read from <FEXPath>/fex-emu/ by default.

For Arm64ec/Wow64 WINE builds:

These files are instead read from $LOCALAPPDATA/fex-emu/ by default.

For further customization, see FEX_APP_CONFIG_LOCATION and FEX_APP_DATA_LOCATION.

default: ''

FEX_APP_CACHE_LOCATION

Allows the user to override where FEX stores and loads cache files

By default FEX will look in ${XDG_CACHE_HOME, $HOME/.cache}/fex-emu/

This will override the full path, trailing forward-slash is expected to exist

default: ''

FILES
$XDG_CONFIG_DIR/fex-emu
    Default FEX user configuration directory
$prefix/share/fex-emu/AppConfig
    System level application configuration files
$prefix/share/fex-emu/GuestThunks
    guest-side thunk data libraries
$prefix/lib/fex-emu/HostThunks
    host-side thunks for guest communication