Digitaalisten ihmistieteiden osasto
PL 24 (Unioninkatu 40)
00014 Helsingin yliopisto

fin-clarin ( ATT ) helsinki.fi

Kerro meille omasta kieliaineistostasi!

FIN-CLARIN    CSC - Tieteen tietotekniikan keskus
Kansainvälinen CLARIN-projekti


This is semi-automatically converted version of the README of HFST distribution.

Helsinki Finite-State Technology (library and application suite)

This package contains a bridging library for multiple FST libraries and toolkits and set of tools for processing of finite-state automate especially for linguistic systems. HFST library and tools are licenced under GNU GPL licence version 3, you may read the full licence in the file named COPYING. The authors specified in AUTHORS file may be contacted about licencing issues.

There's an online wiki-based information source about HFST in Kitwiki HFST pages https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstHome


Installation method depends on operating system and the version you want to install. For stable versions there exists packages for some of the better operating system and package manager combinations. For bleeding edge newest versions, development and non-supported operating systems and versions you will have to perform installation from the sources.

Installation packages for Debian and Ubuntu

Debian packages are now available as part of Apertium, see HfstDownloads. Old installation packages are available on HFST Sourceforge archive.

HFST and related software can be downloaded from HFST download directory. This folder contains sources and debian packages for HFST API library and tool packages. The debian packages are experimental; the requirements of debian or ubuntu installations are same as main packages. SFST is excluded from the packages as it has portability issues with hash_maps and hash_sets. The python packages contain binaries for swig-generated Python bindings for HFST.

Installation packages for Windows

HFST download directory contains an experimental installer for Windows, HfstInstaller.exe. It has ready-compiled HFST library, tools and Python bindings. Performing installation from the sources is also possible on Windows with MinGW and Cygwin.

Installation for MacPorts

The port tree for HFST and related software can be downloaded from HFST download directory. To add this port tree to your macports installation, unpack it to wherever and insert that directory to your sources.conf, e.g. in the default installation to /opt/local/etc/macports/sources.conf:

    # If an rsync URL points to a .tar file, a signed .rmd160 must exist next to
    # it on the server and will be used to verify its integrity.
    rsync://rsync.macports.org/release/tarballs/ports.tar [default]

Now you can install HFST and related software using port commands:

  sudo port install hfst

Installation for Gentoo Linux

HFST software is available in science overlay. For portage, use:

  layman -a science

To add the repository. Then:

  emerge -av hfst

to install. Parts of the spell-checking tools are in Finnish overlay. If you use Paludis, just try to cave resolve sci-misc/hfst to get the current instructions.

Installation for other systems

For rest of the systems, HFST needs to be installed from the source, the traditional GNU/linux way. To begin your installation, you need to start by gathering the dependencies, as advised in the following chapter.


Please note that the dependencies for the library can be set during the compile time. Settings that determine dependencies relate to features the resulting library will have; e.g. if you disable openfst you cannot use weighted finite-state automata. If you enable foma, you will be able to read foma format automata as HFST files and use foma's algorithms to process automata. The command ./configure --help lists all features that can be controlled with configure switches and whether they are enabled or disabled by default.

Compilation requirements

  • To use the OpenFST backend (default):
    • source code of OpenFST version 1.2.10 is bundled with HFST and included by default when building HFST
    • compiling against OpenFST library and linking may require recent GCC version and pthread and m libraries
    • to disable OpenFST support, configure switch --without-openfst may be used
  • To use the SFST backend (optional):
    • the SFST library version 1.4.6g is bundled with HFST but not included by default when building HFST
    • SFST requires readline and ncurses
    • The SFST frontend a.k.a. the SFST-PL parser a.k.a. hfst-sfstpl2fst does *not* require the SFST library to be installed; the library is only used for library-stuff like reading SFST format automata.
    • to enable SFST backend, configure switch --with-sfst must be used
  • To use the foma backend (optional):
    • the foma library version 0.9.16alpha is bundled with HFST but not included be default when building HFST
    • foma requires editline, termcap (or ncurses) and zlib libraries to compile
    • to enable foma backend, configure switch --with-foma may be used
    • the hfst-xfst frontend currently requires foma binary; foma library is not needed to be linked for this
  • For proper Unicode support in corpus tools:
    • Glib version 2.16 or newer; configure
    • Pkg-config 0.14 or newer
    • Without unicode libraries the corpus tools may be unable to handle operations such as case-folding properly (default)
  • To compile corpus processing tool hfst-proc, you need to use the configure switch --enable-proc ??

Note that if you did install dependent libraries, such as libxml or glib to your home directory instead of using your system's package manager (or supported default location):

  • If you only have a local version of a library, you can use it with appropriate LDFLAGS and CXXFLAGS, eg.
    ./configure LDFLAGS=-L/path/to/local/lib --prefix=/path/to/local/installation
    make CXXFLAGS=-I/path/to/local/headers

If you are building a development version you loaded from the version control system, you must have new brand of GNU development tools installed:

  • autoconf >=2.62
  • automake >=1.11.1
  • libtool >=2.2
  • >=gettext-0.17

  • GNU tool-chain is also needed with distributed packages if the user wishes to modify Makefile.am or configure.ac files.

  • Mac OS X users are advised to use MacPorts; Mac OS X 10.6 with XCode 2.3.2 at least is not sufficient

  • A package loaded from hfst web site does not have these requirements

The source codes loaded from the version controls system will also require parse generator system:

  • GNU flex 2.5.35 or compatible and
  • GNU bison 1.31 (2.4 suggested) or compatible
  • flex 2.5.4-2.5.33 will choke on perfectly valid rules used in hfst
  • bison older than 1.31 do not support name-prefix needed for having multiple parsers in one library
  • A package loaded from hfst web site does not have these requirements
  • source code loaded from version control system requires them only to bootstrap; if you use systems with archaic versions of flex or bison and cannot install updates, you might be able to get the needed files from somewhere

When running HFST software or using HFST libraries from HFST-enabled software:

  • If the executable is dynamically linked (almost always), the operating system must be able to find hfst libraries
  • If you install the libraries in non-standard paths, you need to ensure that operating system is aware of this; In linux this may happen by setting LD_LIBRARY_PATH, on Mac DYLD_LIBRARY_PATH
  • the hfst-xfst frontend currently *requires* foma binary (--with-foma to enable)
  • the hfst-xfst frontend needs GNU compatible getopt, or basic getopts without GNU-incompatible getopt installed
  • the hfst-sfstpl2fst frontend does *not* require SFST libraries or binaries, you do *not* need to enable SFST libraries (via the switch --with-sfst) in order to compile SFST-PL scripts to HFST automata.

Installation from the sources

INSTALL describes the GNU build system in detail, but for most users the usual::

    (as root) make install

should result in a local installation and::

    (as root) make uninstall

in its uninstallation. If you aren't going to be linking to the library after building it and don't need to be able to send debugging information, you can save a considerable amount of space and memory by doing:

    make install-strip

instead of make install. This strips all the symbols from the binaries, reducing sizes by a factor of 5-10.

If you would rather install in e.g. your home directory (or aren't the system administrator), you can tell ./configure:

        ./configure --prefix=$HOME

The HFST library may link to numerous FST handling backends with varying licences. If you are going to redistribute the HFST library you compiled, make sure there are no clashes in the licences of the linked libraries; all of them are free and open source, but under strict interpretation you may not be able to redistribute combination of strict GNU GPLv2 and Apache APLv2 in the same binary (e.g. foma and OpenFST).

If you are checking out the development versions from SVN you must first create and install the necessary autotools files from the host system:

  autoreconf -i

It is common practice to keep generated files out of version control: http://www.gnu.org/software/automake/manual/automake.html#CVS.

For further installation instruction refer to file INSTALL, which contains the standard installation instructions for GNU autoconf based software.

If you are compiling HFST from source on Windows with MinGW or Cygwin, use the switch --enable-windows when running ./configure.


In this section we list the errors that pop up commonly on `our support channel irc://FreeNode/#hfst, in our bug tracker https://sourceforge.net/tracker/?group_id=224521&atid=1061990 or on our development mailing list hfst-development@lists.sf.net.

::malloc has not been declared

*During the compilation* errors of form::

  /usr/include/c++/4.3/cstdlib:124: error: '::malloc' has not been declared


  your configure failed to find malloc, check README for further instructions

are mosts often caused by broken library installation. The simplest solution in Linux-based platforms is ldconfig:

  ldconfig -v

This is actually told and performed by the autotools libtool library installation, but it's easy to miss. It looks like this:

  Libraries have been installed in:

  If you ever happen to want to link against installed libraries
  in a given directory, LIBDIR, you must either use libtool, and
  specify the full pathname of the library, or use the '-LLIBDIR'
  flag during linking and do at least one of the following:
     - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
       during execution
     - add LIBDIR to the 'LD_RUN_PATH' environment variable
       during linking
     - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
     - have your system administrator add LIBDIR to '/etc/ld.so.conf'

  See any operating system documentation about shared libraries for
  more information, such as the ld(1) and ld.so(8) manual pages.

If you installed a library on non-standard path, or installed it to the default /usr/local/lib, but your variant of Linux doesn't support libraries there, you may need to set it up and/or ldconfig the directory explicitly:

  export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
  ldconfig -v -n /usr/local/lib

If all else fails, try installing the library to wherever your blend of Linux installs all its libraries, such as /usr/lib.

See also: http://nerdland.net/unstumping-the-internet/malloc-has-not-been-declared/ for the gory details.

Error while loading shared libraries: libhfst.so.0: cannot open shared object file: No such file or directory

*After installing HFST and running programs*, the installed programs should on most systems be able to find and use the shared libraries that just got installed alongside the programs that were installed on the same go, but this is not always the case. Typically on first installation of the HFST library or after a major version update of HFST library, the system may not know about it. To fix this, you must run ldconfig on GNU systems. To ensure proper linking, use ldconfig -v to get a print out of known libraries, the listing should include libhfst.so indicating the current version.

If this is the first time you install a library on your system by hand, it may happen on some systems that the library configuration does not include that directory at all. Notably on ubuntu it seems that /usr/local/lib is not one of library directories, and that is the default target for manually installed libraries. Either fix this by doing

./configure --prefix=/usr
or check your distributions manuals on how to set up new library directories. Same applies for libraries installed to e.g. home directory.

See also the previous error description.

chmod: cannot access 'scripts/hfst-foma-wrapper.sh': No such file or directory

*During the configuration phase.* This is a case of autoconf being silly; the commands for creating a script and making it executable from autoconf are ran in parallel but must be ran once per file and autoconf just doesn't keep track of the files it creates. The script for doing this is copied to configure.ac from autoconf manual so it supposably will always work correctly...

....libs/lt-hfst-strings2fst: Incorrect utf-8 coding

*During the make check phase*. This indicates that a test that tests for expected failure fails expectedly, which is also indicated by a green word PASS or XFAIL on the next line. This is expected behaviour and not an indication of a bug. If there is a bug effecting that or other tests in the test suite, it will be indicated by a line starting with word FAIL or XPASS in red colour.

Same applies for other messages printed during make check phase. The cases where something actually fails will be clearly stated in the end of the test suite by a message such as::

  2 of 36 tests failed
  Please report to hfst-bugs@helsinki.fi

These errors can be reported either to the stated mail address or the HFST bug tracker in sf.net infrastructure https://sourceforge.net/tracker/?atid=1061990&group_id=224521&func=browse.

cat: hvVqf:o:l:u:: No such file or directory

*During use of bash-based scripts*, an error message including things like::

  cat: -l: No such file or directory
  cat: version,quiet,format:,output:,latin1::,utf8::: No such file or directory
  cat: -n: No such file or directory
  cat: hfst-lexc: No such file or directory
  cat: --: No such file or directory

indicate that script is trying to use Mac OS X's getopt as if it was GNU getopt. However default getopt in Mac OS X does not work at all like GNU getopt. Easiest solution is to install working getopt, e.g. by using MacPorts:

  sudo port install getopt

The newer versions of bash scripts detect Mac OS X's getopt and fallback to using getopts. Note that getopts does not support long options and filenames must be last parameters on commandline with it, so its use is strongly discouraged.

libc++-abi.dylib: terminate called throwing an exception

*During program execution* (Mac OS X only), errors of form::

  terminate called throwing an exception
  Abort trap: 6

Can be caused by, just about any exceptional situation that does not have specific handler. On Linux it will read::

  terminate called after throwing an instance of 'ImplementationTypeNotAvailableException'

And then you'll know that this specific exception is about backend that was disabled during `configure` phase. Or it might read::

 terminate called after throwing an instance of 'UndefinedSymbolPairsFound'

And you'd know it's something with the alphabet. But OS X won't tell us this. So it is an unexpected error situation. Usual suspects are still: missing library in configure, empty file, reading error, writing error...

Further information

The HFST wiki site contains further details of the HFST system.

Reporting bugs

Bugs can be reported via email to HFST team bug mail address <hfst-bugs@helsinki.fi>, or preferably to HFST's bug tracking system at sourceforge https://sourceforge.net/tracker/?atid=1061990&group_id=224521&func=browse When reporting, please include at least following:

  • version of software used, if command-line tool (hfst-toolname --version)
  • version of hfst-library, if possible
  • steps to reproduce, attach or all related files if possible
  • information about platform used (e.g. uname -a)

libxml2: http://www.xmlsoft.org/ libreadline: http://www.gnu.org/software/readline/