HFST: Making a new release of HFST

At Github Releases, offer

• source package: hfst-x.y.z.tar.gz
• minimal lgpl-licensed source package: libhfst-x.y.z.tar.gz

At Python Packaging index, offer

• source package
• 32-bit bindings for python versions 2.7, 3.5 and 3.6 for Windows
• 32/64-bit bindings (universal binaries) for python versions 2.7, 3.4, 3.5 and 3.6 for Mac OS X

At Github download directory, offer

• a package for checking command line tools, hfst-ospell and morphologies on taito: check-hfst-x.y.z.tar.gz

Debian packages for HFST, including python bindings (for python 2 and 3), are released via Apertium. Binaries for Windows and Mac, not including python bindings, are also released via Apertium.

HFST Git repository

The files to change when releasing are configure.ac and lgpl-release/configure.ac (in 12 places, notice the LONGVERSION) and libhfst/src/Makefile.am where you have to change the value of libhfst_la_LDFLAGS.

Also change the version number in python/setup.py, python/hfst/__init__.py and python/pypi/setup.py (python/pypi/hfst/__init__.py is copied from python/hfst/__init__.py) if you are releasing the python bindings. At least in the beginning, we use four-digit numbering scheme for PyPi python bindings, so HFST release x.y.z has a corresponding python bindings release x.y.x.0. We can then offer further python releases x.y.z.1, x.y.z.2 and so on without the need to make a new HFST release every time.

Firstly ensuring that everything is in place and seemingly workable, execute on HFST machine:

make && make check
make dist

Copy the tarball to taito-shell and execute there

rm -fR hfst-installation
mkdir hfst-installation
rm -fR hfst-x.y.z
tar -xzvf hfst-x.y.x.tar.gz
cd hfst-x.y.z

Then compile HFST as said on page HfstCSCInstallation. If there are no errors, you can make a release.

Assuming everything is in place, update NEWS and ChangeLog with info about the purpose of this release. The NEWS file shall contain human-readable summary of the major changes since last release (if there are no major changes, do not bother yourself with releasing ;-). The ChangeLog is nearly machine-readable file of all changes in all of the files in the source tree. In git you can create it with git log --oneline --decorate [--color] Commit these to trunk.

(C++11 and gcc 5 issue: https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html)

Git release and source tarball

Tag the release version

git tag -a v3.12.0 -m "Release 3.12.0"
git push origin v.3.12.0

to make a release in github.

Also copy the distribution tarball to github as well as the LGPL version containing just openfst back-end and python bindings (described at the bottom of this page).

Copy the release information in NEWS to github:

Debian packages

Debian packages for hfst and python bindings are released as part of Apertium.

Windows release

Command line tools are released as part of Apertium.

Mac OSX release

Command line tools are released as part of Apertium.

Python bindings

See instructions here. Python bindings can be released more often than there are new releases of HFST, just update the fourth digit of the release (version in file python/pypi/setup.py and __version__ in python/hfst/__init__.py). When there is again a new release of HFST available, update also the other numbers and set the fourth number to zero.

Source distribution

On HFST machine, go to python/pypi directory and copy HFST C++ source files (make sure that you have a fresh version of flex/yacc-generated cc files)

./copy-files.sh

Edit the version number in setup.py and hfst/__init__.py, if needed.

Make sure that libhfst.py is up-to-date:

python3 setup.py build_ext --inplace

Then create a source distribution

python3 setup.py sdist

Test that it works

cd dist
tar -xzvf hfst-x.y.z.wb0.tar.gz
cd hfst-x.y.z.wb0
python3 setup.py build_ext --inplace

(Create the following file named .pypirc under $HOME or %USERPROFILE% on windows. This needs to be done only once.)

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi

[pypitest]
repository=https://testpypi.python.org/pypi

(Register package to PyPi. This needs to be done only once. If you have done this already, you can skip this step.)

twine register -r [ pypi | pypitest ] dist/hfst-x.y.z.wb0.tar.gz

Then release a source distribution

twine upload -r [ pypi | pypitest ] dist/hfst-x.y.z.wb0.tar.gz

Check that it works

[sudo [-H]] python3.4 -m pip install [-i https://testpypi.python.org/pypi] [--upgrade] hfst

Windows

Prerequisites

Git Bash shell (https://git-scm.com/download/win).

Flex and Bison for windows (https://sourceforge.net/projects/winflexbison/).

Swig (http://swig.org/download.html). Swig defines ssize_t differently from msvc compiler, so you have to change it in python files PythonNN-32\include\pyconfig.h:

//typedef _W64 int ssize_t;
typedef long ssize_t; // CHANGED: HFST

Windows x86-64 executable installers for different python versions can be fetched from python official pages. You have to upgrade pip, setuptools and wheel:

C:\PythonNN\python.exe -m pip install [pip|setuptools|wheel] --upgrade

C++ compilers and toolkits can also be fetched from python official pages. For python 3.5: install 'Microsoft Visual C++ Build Tools 2015' (update: 2017), check 'Win 8.1 SDK' and 'Win 10 SDK'. (VC15++: header files OAIdl.h:473 and propidlbase.h:319 contain variables named bool, they must be changed to bool_.)

Python 2.7: First install Microsoft Visual C++ 2008 SP1 Redistributable Package (x64) and then Microsoft Visual C++ Compiler for Python 2.7. NOTE: the compiler does not support C++11 standard! (Basically it is possible to compile python 2.7 interpreter from scratch with a newer VC compiler...)

Twine (via /c/PythonNN/python.exe -m pip install twine, installed by default to /c/PythonNN/Scripts/twine.exe). See instructions about creating a .pypirc file above.

Generating the distribution

Run Git Bash shell and pull the latest version. Make sure you have frech cc and hh files generated from flex/yacc files by running ./compile-parsers-win.sh in folder libhfst/src/parsers. Add directory where you have win_flex.exe and win_bison.exe to PATH before running the script.

Go to directory python/pypi and copy everything needed for a pypi distribution with copy-files-win.sh.

Run command prompt:

• for python3.4 wheel, open Windows SDK 7.1 Command Prompt
• for python3.5 and higher, use VS2015 x64 Native Tools Command Prompt
• for python2.7, use Visual C++ 2008 64-bit Command Prompt

Add swig to path

PATH=%PATH%;C:\PATH\TO\swigwin-3.0.5\

and run pypi setup file

C:\PythonNN\python.exe setup.py build_ext --inplace [--generate-wrapper]

Then perform the tests in Git Bash shell (todo: spaces in pathnames e.g. "Program Files"):

cd test
./test.sh --python PYTHON --pythonpath PATH

If they pass, create the distribution in command prompt:

C:\PythonNN\python.exe setup.py bdist_wheel

Upload it to pypi (on command prompt, git bash shell doesn't work properly):

/c/PythonNN/Scripts/twine.exe upload [-r testpypi] hfst-3.12.0b0-cp[34|35|36]-cp[34|35|36]m-[win_amd64|win32].whl

Test that everything works by installing the newly uploaded wheel on Windows:

C:\Python[34|35|36]\python.exe -m pip install [-i https://testpypi.python.org/pypi] [--upgrade] hfst

Include MSVCP140.DLL and VCRUNTIME140.DLL from C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Redist\MSVC\14.10.25017\onecore\x86\Microsoft.VC150.CRT.

Bugfixes

If you get an error like

error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

run

SET DISTUTILS_USE_SDK=1
SET MSSdk=1

The following warning should be fixed:

C:\PythonNN\lib\site-packages\wheel\pep425tags.py:77: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):

Mac OSX

On Mac machine, go to python/pypi directory and copy HFST C++ source files (make sure that you have a fresh version of flex/yacc-generated cc files)

./copy-files.sh

Manually compile foma back-end files:

for file in back-ends/foma/*.c; do clang -fPIC -std=c99 -arch i386 -arch x86_64 -mmacosx-version-min=10.7 -DHAVE_FOMA -c $file ; done

Then create the wheel (make sure you have installed fresh versions of pip, setuptools and wheel for the version of python you are using) and upload it

python3.4 setup.py bdist_wheel
twine upload -r pypitest hfst-x.y.z.wb0-cp34-cp34m-macosx_10_7_x86_64.whl

ans test that it works

[sudo [-H]] python3.4 -m pip install -i https://testpypi.python.org/pypi hfst

Update documentation

Create a .zip file and upload it to http://pythonhosted.org/hfst.

Python API documentation

In hfst repository:

cd path/to/hfst/python/doc
doxygen Doxyfile
cd html
cp -R * /path/to/hfst.github.io/python/
# add, commit and push to github

Check changes in the ABI

The relevant policy is https://www.debian.org/doc/debian-policy/ch-sharedlibs.html section 8.1 paragraph 4.

Dump the symbols from earlier and newer revision for comparison:

nm -DgC ./libhfst/src/.libs/libhfst.so.NN.0.0 | grep -vi vtable | perl -wpne 's/^\w+\s+\w+\s+//;' | sort > /tmp/hfst-REV_NN.nm

LGPL release with libhfst, python and openfst back-end

Copy a fresh dist package in directory scripts and then run in the scripts directory

./make-libhfst-package hfst-x.y.z.tar.gz ../lgpl-package

Morphologies

#!/bin/sh

# This script downloads and installs HFST morphologies and acompanying scripts
# Requires HFST installation, wget, tar, make, install
# Author: Martin Matthiesen, CSC based on a script by Erik Axelson


# Base URL for download
DOWNLOAD_URL=http://sourceforge.net/projects/hfst/files/morphological-transducers

# Languages to install
LANGUAGES="english finnish french german italian omorfi swedish turkish"

# Determine HFST_ROOT
if ( which hfst-lookup >& /dev/null ) then
    HFST_LOOKUP=`which hfst-lookup`
    HFST_ROOT=`dirname $HFST_LOOKUP|sed 's%/bin$%%'`
else
    echo hfst-lookup not found, is hfst installed? Aborting.
    exit 1
fi


# Set download directory
TEMP_DIR=/tmp/hfst
mkdir -p $TEMP_DIR
cd $TEMP_DIR

echo Morphology installer
echo This script installs the hfst morphologies for the languages: $LANGUAGES
echo
echo Installing using prefix $HFST_ROOT
echo Downloading to and installing from  $TEMP_DIR
echo
echo Hit enter to continue, CTRL-C to stop.
read


# Get, extract and install morphologies
for LANG in $LANGUAGES; do 
    PACKAGE=hfst-$LANG-installable.tar.gz;
    PACKAGE_DIR=`basename $PACKAGE .tar.gz`;

    # Get package if it does not exist
    echo "### Getting $PACKAGE (if needed)..."
    if !([ -f $PACKAGE ]) then
   wget $DOWNLOAD_URL/$PACKAGE;
    fi

    # Unpack if not done earlier
    echo "### Unpacking $PACKAGE (if needed)..."
    if !([ -f $PACKAGE_DIR/Makefile ]) then 
   tar -xzvf $PACKAGE;
    fi

    # Install to HFST_ROOT
    echo "### Installing $PACKAGE ..."
    if ! (make -C $PACKAGE_DIR install prefix=$HFST_ROOT); then
   echo "### ERROR ###: Error in installing "$LANG to $HFST_ROOT;
    fi
done