Resolving Install Errors for the lightweight_mmm Python Package

To experiment with different specifications of a marketing mix model, I tried installing the lightweight_mmm Python package from Google. When using the standard pip3 install lightweight_mmm command in terminal, I was hitting this error:

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [66 lines of output]
      The Meson build system
      Version: 0.62.2
      Source dir: /private/var/folders/ly/3d_s7td56qb7rn4qc2c6tmvm0000gn/T/pip-install-yc6ribm7/scipy_ae66ccfb8522422ab91e65b44570f273
      Build dir: /private/var/folders/ly/3d_s7td56qb7rn4qc2c6tmvm0000gn/T/pip-install-yc6ribm7/scipy_ae66ccfb8522422ab91e65b44570f273/.mesonpy-ur2kyhcf/build
      Build type: native build
      Project name: SciPy
      Project version: 1.9.1
      C compiler for the host machine: cc (clang 14.0.0 "Apple clang version 14.0.0 (clang-1400.0.29.202)")
      C linker for the host machine: cc ld64 820.1
      C++ compiler for the host machine: c++ (clang 14.0.0 "Apple clang version 14.0.0 (clang-1400.0.29.202)")
      C++ linker for the host machine: c++ ld64 820.1
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 8.2.0 "GNU Fortran (GCC) 8.2.0")
      Fortran linker for the host machine: gfortran ld64 820.1
      Program cython found: YES (/private/var/folders/ly/3d_s7td56qb7rn4qc2c6tmvm0000gn/T/pip-build-env-gddbg9u0/overlay/bin/cython)
      Program pythran found: YES (/private/var/folders/ly/3d_s7td56qb7rn4qc2c6tmvm0000gn/T/pip-build-env-gddbg9u0/overlay/bin/pythran)
      Program cp found: YES (/bin/cp)
      Program python found: YES (/Users/<username>/.pyenv/versions/3.11.2/bin/python3.11)
      Found pkg-config: /usr/local/bin/pkg-config (0.29.2)
      Library npymath found: YES
      Library npyrandom found: YES
      Found CMake: /usr/local/bin/cmake (3.26.0)
      Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
      Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)

      ../../scipy/meson.build:130:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig, framework and cmake

I tried to install openblas using homebrew, but it was already installed:

$ brew install openblas
==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################## 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json

Warning: openblas 0.3.21 is already installed and up-to-date.
To reinstall 0.3.21, run:
  brew reinstall openblas

So maybe the build process was failing because the compiler couldn’t find it on the system. On some searching, I found this github issue that discussed the same problem. Essentially, we get more information on openblas, using brew info openblas in the terminal which should show us this output:

==> openblas: stable 0.3.21 (bottled), HEAD [keg-only]
Optimized BLAS library
https://www.openblas.net/
/usr/local/Cellar/openblas/0.3.21 (23 files, 130MB)
  Poured from bottle on 2023-02-11 at 08:52:47
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/openblas.rb
License: BSD-3-Clause
==> Dependencies
Required: gcc ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
openblas is keg-only, which means it was not symlinked into /usr/local,
because macOS provides BLAS in Accelerate.framework.

For compilers to find openblas you may need to set:
  export LDFLAGS="-L/usr/local/opt/openblas/lib"
  export CPPFLAGS="-I/usr/local/opt/openblas/include"

For pkg-config to find openblas you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/openblas/lib/pkgconfig"
==> Analytics
install: 1,074 (30 days), 63,095 (90 days), 515,454 (365 days)
install-on-request: 171 (30 days), 10,278 (90 days), 85,564 (365 days)
build-error: 4 (30 days)

In the output above, we find the options to be set for compilers to find openblas. I run these commands in the terminal:

$ export LDFLAGS="-L/usr/local/opt/openblas/lib"
$ export CPPFLAGS="-I/usr/local/opt/openblas/include"
$ export PKG_CONFIG_PATH="/usr/local/opt/openblas/lib/pkgconfig"

I now try installing lightweight_mmm using pip3 install lightweight_mmm in the terminal, and this time the installation gets past the step it was failing, but runs for a long time and fails again:

  Using cached scipy-1.6.1.tar.gz (27.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [119 lines of output]
      setup.py:461: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /private/var/folders/ly/3d_s7td56qb7rn4qc2c6tmvm0000gn/T/pip-modern-metadata-_yaq83ex'), proceeding with generating Cython sources and expanding templates
        warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
      setup.py:563: DeprecationWarning:

        `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
        of the deprecation of `distutils` itself. It will be removed for
        Python >= 3.12. For older Python versions it will remain present.
        It is recommended to use `setuptools < 60.0` for those Python versions.
        For more details, see:
          https://numpy.org/devdocs/reference/distutils_status_migration.html


        from numpy.distutils.core import setup
      Running from SciPy source directory.
      INFO: lapack_opt_info:
      INFO: lapack_armpl_info:
      INFO: customize UnixCCompiler
      INFO:   libraries armpl_lp64_mp not found in ['/Users/<username>/.pyenv/versions/3.11.2/lib', '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      ...

On further searching, it appears that the problem could be the version of python I’m currently using (3.11.2).

I tried using a version of Python < 3.9 and checked if this was working. First, using the instructions here, I used pyenv to install version 3.8.16, with the --enable-shared option:

   $ env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.8.16

Next, I set this to be the global Python version

   $ pyenv global 3.8.16

I now try installing the package again using pip3 install lightweight_mmm, and this time it works:

Installing collected packages: typing-extensions, tensorboard-plugin-wit, pytz, pyasn1, libclang, flatbuffers, zipp, wrapt, wheel, urllib3, tqdm, threadpoolctl, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, rsa, pyparsing, pyasn1-modules, protobuf, pillow, packaging, oauthlib, numpy, MarkupSafe, kiwisolver, keras, joblib, immutabledict, idna, grpcio, gast, fonttools, cycler, charset-normalizer, certifi, cachetools, absl-py, werkzeug, scipy, requests, python-dateutil, patsy, opt_einsum, multipledispatch, importlib-metadata, h5py, google-pasta, google-auth, contourpy, cftime, astunparse, scikit-learn, requests-oauthlib, pandas, netcdf4, matplotlib, markdown, jaxlib, jax, xarray, statsmodels, seaborn, numpyro, google-auth-oauthlib, tensorboard, arviz, tensorflow, lightweight_mmm
  Running setup.py install for jax ... done
Successfully installed MarkupSafe-2.1.2 absl-py-1.4.0 arviz-0.11.2 astunparse-1.6.3 cachetools-5.3.0 certifi-2022.12.7 cftime-1.6.2 charset-normalizer-3.1.0 contourpy-1.0.7 cycler-0.11.0 flatbuffers-23.3.3 fonttools-4.39.2 gast-0.4.0 google-auth-2.16.3 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.51.3 h5py-3.8.0 idna-3.4 immutabledict-2.2.3 importlib-metadata-6.1.0 jax-0.4.6 jaxlib-0.4.6 joblib-1.2.0 keras-2.11.0 kiwisolver-1.4.4 libclang-16.0.0 lightweight_mmm-0.1.7 markdown-3.4.3 matplotlib-3.6.1 multipledispatch-0.6.0 netcdf4-1.6.3 numpy-1.24.2 numpyro-0.11.0 oauthlib-3.2.2 opt_einsum-3.3.0 packaging-23.0 pandas-1.5.3 patsy-0.5.3 pillow-9.4.0 protobuf-3.19.6 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2023.2 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 scikit-learn-1.2.2 scipy-1.10.1 seaborn-0.11.1 six-1.16.0 statsmodels-0.13.5 tensorboard-2.11.2 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.11.1 tensorflow-estimator-2.11.0 tensorflow-io-gcs-filesystem-0.31.0 termcolor-2.2.0 threadpoolctl-3.1.0 tqdm-4.65.0 typing-extensions-3.10.0.2 urllib3-1.26.15 werkzeug-2.2.3 wheel-0.40.0 wrapt-1.15.0 xarray-2023.1.0 zipp-3.15.0

While searching for a fix, I found this website that looks at Python 3.9 support for the 360 most downloaded packages on PyPi. This is a great reference to understand which packages (of the top 360) still do not support Python 3.9 (or later versions).