HFST: Making a new release of HFST
At Github Releases, offer
- source package: hfst-x.y.z.tar.gz
- minimal lgpl-licensed source package: libhfst-x.y.z.tar.gz
At Python Packaging index, offer
- source package
- 32-bit bindings for python versions 2.7, 3.5 and 3.6 for Windows
- 32/64-bit bindings (universal binaries) for python versions 2.7, 3.4, 3.5 and 3.6 for Mac OS X
At Github download directory, offer
- a package for checking command line tools, hfst-ospell and morphologies on taito: check-hfst-x.y.z.tar.gz
Debian packages for HFST, including python bindings (for python 2 and 3), are released via Apertium.
Binaries for Windows and Mac, not including python bindings, are also released via Apertium.
HFST Git repository
The files to change when releasing are
configure.ac
and
lgpl-release/configure.ac
(in 12 places, notice the
LONGVERSION
)
and
libhfst/src/Makefile.am
where you have to change the value of
libhfst_la_LDFLAGS
.
Also change the version number in
python/setup.py
,
python/hfst/__init__.py
and
python/pypi/setup.py
(
python/pypi/hfst/__init__.py
is copied from
python/hfst/__init__.py
) if you are releasing the python bindings.
At least in the beginning, we use four-digit numbering scheme for PyPi python bindings, so HFST release x.y.z
has a corresponding python bindings release x.y.x.0. We can then offer further python releases x.y.z.1, x.y.z.2
and so on without the need to make a new HFST release every time.
Firstly ensuring that everything is in place and seemingly workable, execute on HFST machine:
make && make check
make dist
Copy the tarball to
taito-shell
and execute there
rm -fR hfst-installation
mkdir hfst-installation
rm -fR hfst-x.y.z
tar -xzvf hfst-x.y.x.tar.gz
cd hfst-x.y.z
Then compile HFST as said on page
HfstCSCInstallation. If there are no errors, you can make a release.
Assuming everything is in place, update
NEWS
and
ChangeLog
with info about the purpose of this release.
The
NEWS
file shall contain human-readable summary of the major changes since last release (if there are no major changes, do not bother yourself with releasing ;-).
The
ChangeLog
is nearly machine-readable file of all changes in all of the files in the source tree. In git you can create it with
git log --oneline --decorate [--color]
Commit these to trunk.
(C++11 and gcc 5 issue:
https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html)
Git release and source tarball
Tag the release version
git tag -a v3.12.0 -m "Release 3.12.0"
git push origin v.3.12.0
to
make a release in github.
Also copy the distribution tarball to github as well as the LGPL version containing just openfst back-end and python bindings
(described at the bottom of this page).
Copy the release information in
NEWS
to github:
Debian packages
Debian packages for hfst and python bindings are released as part of Apertium.
Windows release
Command line tools are released as part of Apertium.
Mac OSX release
Command line tools are released as part of Apertium.
Python bindings
See instructions
here.
Python bindings can be released more often than there are new releases of HFST, just update the fourth digit of the release (
version
in file
python/pypi/setup.py
and
__version__
in
python/hfst/__init__.py
).
When there is again a new release of HFST available, update also the other numbers and set the fourth number to zero.
Source distribution
On HFST machine, go to
python/pypi
directory and copy HFST C++ source files (make sure that you have a fresh version of flex/yacc-generated cc files)
./copy-files.sh
Edit the version number in
setup.py
and
hfst/__init__.py
, if needed.
Make sure that
libhfst.py
is up-to-date:
python3 setup.py build_ext --inplace
Then create a source distribution
python3 setup.py sdist
Test that it works
cd dist
tar -xzvf hfst-x.y.z.wb0.tar.gz
cd hfst-x.y.z.wb0
python3 setup.py build_ext --inplace
(Create the following file named
.pypirc
under
$HOME
or %USERPROFILE% on windows. This needs to be done only once.)
[distutils]
index-servers =
pypi
pypitest
[pypi]
repository=https://pypi.python.org/pypi
[pypitest]
repository=https://testpypi.python.org/pypi
(Register package to PyPi. This needs to be done only once. If you have done this already, you can skip this step.)
twine register -r [ pypi | pypitest ] dist/hfst-x.y.z.wb0.tar.gz
Then release a source distribution
twine upload -r [ pypi | pypitest ] dist/hfst-x.y.z.wb0.tar.gz
Check that it works
[sudo [-H]] python3.4 -m pip install [-i https://testpypi.python.org/pypi] [--upgrade] hfst
Windows
Prerequisites
Git
Bash
shell (
https://git-scm.com/download/win).
Flex and Bison for windows (
https://sourceforge.net/projects/winflexbison/).
Swig (
http://swig.org/download.html). Swig defines
ssize_t
differently from msvc compiler, so you have to change it in python files
PythonNN-32\include\pyconfig.h
:
//typedef _W64 int ssize_t;
typedef long ssize_t; // CHANGED: HFST
Windows x86-64 executable installers for different python versions can be fetched from
python official pages.
You have to upgrade pip, setuptools and wheel:
C:\PythonNN\python.exe -m pip install [pip|setuptools|wheel] --upgrade
C++ compilers and toolkits can also be fetched from
python official pages.
For python 3.5: install 'Microsoft Visual C++ Build Tools 2015' (update: 2017), check 'Win 8.1 SDK' and 'Win 10 SDK'.
(VC15++: header files
OAIdl.h:473
and
propidlbase.h:319
contain variables named
bool
, they must be changed to
bool_
.)
Python 2.7: First install
Microsoft Visual C++ 2008 SP1 Redistributable Package (x64) and then
Microsoft Visual C++ Compiler for Python 2.7. NOTE: the compiler does not support C++11 standard! (Basically it is possible to
compile python 2.7 interpreter from scratch with a newer VC compiler...)
Twine (via /c/PythonNN/python.exe -m pip install twine, installed by default to /c/PythonNN/Scripts/twine.exe). See instructions about creating a
.pypirc
file above.
Generating the distribution
Run
Git
Bash
shell and pull the latest version. Make sure you have frech cc and hh files generated from flex/yacc files by running
./compile-parsers-win.sh
in folder
libhfst/src/parsers
. Add directory where you have
win_flex.exe
and
win_bison.exe
to
PATH
before running the script.
Go to directory
python/pypi
and copy everything needed for a pypi distribution with
copy-files-win.sh
.
Run command prompt:
- for python3.4 wheel, open
Windows SDK 7.1 Command Prompt
- for python3.5 and higher, use
VS2015 x64 Native Tools Command Prompt
- for python2.7, use
Visual C++ 2008 64-bit Command Prompt
Add swig to path
PATH=%PATH%;C:\PATH\TO\swigwin-3.0.5\
and run pypi setup file
C:\PythonNN\python.exe setup.py build_ext --inplace [--generate-wrapper]
Then perform the tests in
Git
Bash
shell (todo: spaces in pathnames e.g. "Program Files"):
cd test
./test.sh --python PYTHON --pythonpath PATH
If they pass, create the distribution in command prompt:
C:\PythonNN\python.exe setup.py bdist_wheel
Upload it to pypi (on command prompt, git bash shell doesn't work properly):
/c/PythonNN/Scripts/twine.exe upload [-r testpypi] hfst-3.12.0b0-cp[34|35|36]-cp[34|35|36]m-[win_amd64|win32].whl
Test that everything works by installing the newly uploaded wheel on Windows:
C:\Python[34|35|36]\python.exe -m pip install [-i https://testpypi.python.org/pypi] [--upgrade] hfst
Include
MSVCP140.DLL
and
VCRUNTIME140.DLL
from
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Redist\MSVC\14.10.25017\onecore\x86\Microsoft.VC150.CRT
.
Bugfixes
If you get an error like
error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
run
SET DISTUTILS_USE_SDK=1
SET MSSdk=1
The following warning should be fixed:
C:\PythonNN\lib\site-packages\wheel\pep425tags.py:77: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
warn=(impl == 'cp')):
Mac OSX
On Mac machine, go to
python/pypi
directory and copy HFST C++ source files (make sure that you have a fresh version of flex/yacc-generated cc files)
./copy-files.sh
Manually compile foma back-end files:
for file in back-ends/foma/*.c; do clang -fPIC -std=c99 -arch i386 -arch x86_64 -mmacosx-version-min=10.7 -DHAVE_FOMA -c $file ; done
Then create the wheel (make sure you have installed fresh versions of
pip
,
setuptools
and
wheel
for the version of python you are using) and upload it
python3.4 setup.py bdist_wheel
twine upload -r pypitest hfst-x.y.z.wb0-cp34-cp34m-macosx_10_7_x86_64.whl
ans test that it works
[sudo [-H]] python3.4 -m pip install -i https://testpypi.python.org/pypi hfst
Update documentation
Create a
.zip
file and upload it to
http://pythonhosted.org/hfst.
Python API documentation
In hfst repository:
cd path/to/hfst/python/doc
doxygen Doxyfile
cd html
cp -R * /path/to/hfst.github.io/python/
# add, commit and push to github
Check changes in the ABI
The relevant policy is
https://www.debian.org/doc/debian-policy/ch-sharedlibs.html section 8.1 paragraph 4.
Dump the symbols from earlier and newer revision for comparison:
nm -DgC ./libhfst/src/.libs/libhfst.so.NN.0.0 | grep -vi vtable | perl -wpne 's/^\w+\s+\w+\s+//;' | sort > /tmp/hfst-REV_NN.nm
Repeat for earlier revision (=/usr/local/lib/libhfst.so.MM.0.0), diff, find relevant removals and replacements.
LGPL release with libhfst, python and openfst back-end
Copy a fresh dist package in directory
scripts
and then run in the
scripts
directory
./make-libhfst-package hfst-x.y.z.tar.gz ../lgpl-package
Morphologies
#!/bin/sh
# This script downloads and installs HFST morphologies and acompanying scripts
# Requires HFST installation, wget, tar, make, install
# Author: Martin Matthiesen, CSC based on a script by Erik Axelson
# Base URL for download
DOWNLOAD_URL=http://sourceforge.net/projects/hfst/files/morphological-transducers
# Languages to install
LANGUAGES="english finnish french german italian omorfi swedish turkish"
# Determine HFST_ROOT
if ( which hfst-lookup >& /dev/null ) then
HFST_LOOKUP=`which hfst-lookup`
HFST_ROOT=`dirname $HFST_LOOKUP|sed 's%/bin$%%'`
else
echo hfst-lookup not found, is hfst installed? Aborting.
exit 1
fi
# Set download directory
TEMP_DIR=/tmp/hfst
mkdir -p $TEMP_DIR
cd $TEMP_DIR
echo Morphology installer
echo This script installs the hfst morphologies for the languages: $LANGUAGES
echo
echo Installing using prefix $HFST_ROOT
echo Downloading to and installing from $TEMP_DIR
echo
echo Hit enter to continue, CTRL-C to stop.
read
# Get, extract and install morphologies
for LANG in $LANGUAGES; do
PACKAGE=hfst-$LANG-installable.tar.gz;
PACKAGE_DIR=`basename $PACKAGE .tar.gz`;
# Get package if it does not exist
echo "### Getting $PACKAGE (if needed)..."
if !([ -f $PACKAGE ]) then
wget $DOWNLOAD_URL/$PACKAGE;
fi
# Unpack if not done earlier
echo "### Unpacking $PACKAGE (if needed)..."
if !([ -f $PACKAGE_DIR/Makefile ]) then
tar -xzvf $PACKAGE;
fi
# Install to HFST_ROOT
echo "### Installing $PACKAGE ..."
if ! (make -C $PACKAGE_DIR install prefix=$HFST_ROOT); then
echo "### ERROR ###: Error in installing "$LANG to $HFST_ROOT;
fi
done