Upgrade to Eigen 5.0.0

[jolars]

Eigen 5.0.0 was released recently. They changed versioning scheme, so the version bump is not as dramatic as it first seems.

Here's a list of the breaking changes. I suspect the requirement on C++14 may be the most problematic among these.

[eddelbuettel]

Thanks for opening the issue; I also saw the note about 5.0.0 being really 3.5.0 in their old scheme. All makes sense.

I have my hands a little full with an ongoing large RcppArmadillo transition plus something new for Rcpp so if someone wants to lead this I would welcome it. I am sure we can coordinate it. Going to a minimum of C++14 should be viable: CRAN is also asking maintainers to remove C++11 requirements where possible. But there are always some old(er) packages...

RcppEigen has a few homegrown patches since the very early days that need to be carried forward but that is hopefully not too onerous.

[jolars]

I noticed that there was also a 3.4.1 version released simultaneously with various bug fixes (https://libeigen.gitlab.io/news/eigen_3.4.1_released/). I'm not sure if it's worth to upgrade to it first, which I guess should be quite straight-forward in comparison.

I just upgraded one of my packages to Eigen 5.0.0 and it required a couple of compatibility work-arounds to get to work. For instance the move from Eigen::all to Eigen::placeholders::all, which I am guessing will be a common issue for many of the packages.

[eddelbuettel]

I am more or less a placeholder maintainer here as Doug moved on to all things Julia a long time ago and I do not use Eigen in my own projects so this is always a bit of a re-learning curve ... but part of the "fun" is that we have a few local patches that need to be reapplied. See this directory for them and the howToDiff file.

I agree with you that going from 3.4.0 to 3.4.1 is probably a good idea before we tackle 3.5.0 aka 5.0.0. If you have some spare cycles, do you want to tackle trying to create a 3.4.1 branch? I am sure we can count on @yixuan (who has been instrumental in the past for version updates) and @jaganmn for additional sets of eyes and tips.

Getting 5.0.0 into CRAN should be possible but transitions (as Debian calls it when a change in package or library requires a coordinated 'wave' of updates) are work. I have been on a long one for (Rcpp)Armadillo and its move to C++14 for a while now (which should conclude soon as CRAN imposed a Deadline on a number of affected packages...). It requires a bit of planning, reverse dependency checking, PR / patch creation and care. Entirely doable. But new package work comes first, of course.

[yixuan]

Hi all, I'm a bit tied up this month, but may take a look some time in December. There are also users interested in the impact of Eigen 5.0.0 on my own Spectra package, e.g., yixuan/spectra#183.

[jolars]

I would prefer not to take the reins on this, at least not at the moment. But I'd be happy to help out with all the various patches that I suspect will need to be submitted to the various dependents once we start the upgrade process.

[eddelbuettel]

I had a little bit of idle time and gave it a go -- new branch is here. It passes tests here, and passed the GitHub Actions (as can be expected, we only run Linux which I use locally too). Next up would be some tire-kicking and maybe seeing what r-universe gets (as it also run macOS and windows).

PS I added the branch to my r-universe setup to source and binaries for linux, macos, windows, ... are here. It passes all tests at r-universe which is a good first step.

PPS But it looks like I butchered something in the CholMod support as eg lme4 falls over :-/

PPPS I got that sorted out, at least temporarily, but using the previous version of CholmodSupport.h plus a mini-fix. Using the current one is a TODO. We do however have a repurcussion with stan headers so we will need help from the stan team.

[eddelbuettel]

A very large number of reverse dependencies fail, and many/most/all(?) fall over StanHeaders. So paging @bgoodri : can you (or someone on the Stan team) take a look? What we have is in the branch listed in the previous comment, i.e. here as well as in this r-universe build off the branch which I dubbed 0.4.9.9-0. It's not super urgent -- we were slow reacting to the Eigen 5.0.0 release late summer -- but it would be nice to remain current.

One of the errors we see it e.g.

In file included from /usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/ArrayBase.h:98,
                 from /usr/local/lib/R/site-library/RcppEigen/include/Eigen/Core:333:
/home/dirk/tmp/lib/StanHeaders/include/stan/math/prim/eigen_plugins.h:51:3: error: ISO C++ forbids declaration of ‘EIGEN_EMPTY_STRUCT_CTOR’ with no type [-Wtemplate-body]
   51 |   EIGEN_EMPTY_STRUCT_CTOR(val_Op);
      |   ^~~~~~~~~~~~~~~~~~~~~~~
/home/dirk/tmp/lib/StanHeaders/include/stan/math/prim/eigen_plugins.h:117:3: error: ISO C++ forbids declaration of ‘EIGEN_EMPTY_STRUCT_CTOR’ with no type [-Wtemplate-body]
  117 |   EIGEN_EMPTY_STRUCT_CTOR(d_Op);
      |   ^~~~~~~~~~~~~~~~~~~~~~~
/home/dirk/tmp/lib/StanHeaders/include/stan/math/prim/eigen_plugins.h:148:3: error: ISO C++ forbids declaration of ‘EIGEN_EMPTY_STRUCT_CTOR’ with no type [-Wtemplate-body]
  148 |   EIGEN_EMPTY_STRUCT_CTOR(adj_Op);
      |   ^~~~~~~~~~~~~~~~~~~~~~~
/home/dirk/tmp/lib/StanHeaders/include/stan/math/prim/eigen_plugins.h:193:3: error: ISO C++ forbids declaration of ‘EIGEN_EMPTY_STRUCT_CTOR’ with no type [-Wtemplate-body]
  193 |   EIGEN_EMPTY_STRUCT_CTOR(vi_Op);
      |   ^~~~~~~~~~~~~~~~~~~~~~~

[bgoodri]

I am on it, but I seem to recall @brianward or @SteveBronder saying that we have to patch the Eigen that Stan uses (outside of R) to fix some their bugs but Eigen hasn't upstreamed those yet in a release. If so, then maybe RcppEigen would want to take those or else I can place the patched Eigen files higher up the include path when rstan compiles them like we used to occasionally do to supersede bugs in Boost that BH inherited.

[eddelbuettel]

Hi Ben et al, and lovely to hear from you! We can surely try to coordinate for stan use with / without R / CRAN. So far I only tried to get RcppEigen to the new release (and that wasn't all that hard, I could probably also try their current master, or a fork with fixes if that last approach is best for us). Let me know how I can help. What I made so far is in the branch, and the branch corresponds to what 'my' (i.e. eddelbuettel, as opposed to rcppcore which has the release) r-universe has.

[eddelbuettel]

FYI, I had pivoted the r-universe build briefly to another branch to ensure it built correctly under all settings. That being the case, I now merged that branch as #145 (giving us much better OpenMP support out of the box) and rebased so we should be back in business with r-universe under my handle being a test release of this Eigen 5.0.0 branch.

PS And I marked this as 0.4.9.9-1 to make it distinct from what we had before as 0.4.9.9-0.

[eddelbuettel]

Hi @bgoodri and @SteveBronder: anything new? Did you get a chance to take a good look? Feasible? Big task / small task?

[bgoodri]

I suspect it is not a big task. It seems Eigen 5.0 removed the EIGEN_EMPTY_STRUCT_CTOR macro that Stan has been using in stan/math/prim/eigen_plugins.h . There is a good chance StanHeaders can just put those lines into an #ifdef depending on the Eigen version, but it would be good if @SteveBronder or @brianward could confirm that is unlikely to break Stan in ways I cannot anticipate.

[bgoodri]

I meant @WardBrian (sorry @brianward) .

[WardBrian]

Based on my reading of the Eigen removal, simply removing the EIGEN_EMPTY_STRUCT_CTOR lines from stan-math is probably reasonable, since I don't think we support gcc 4.9 any longer.

No clue if there would be other issues, I haven't had the chance to try the full 5.0 release.

[bgoodri]

Your guess is that we can just delete those EIGEN_EMPTY_STRUCT_CTOR lines and that won't cause problems under the old Eigen either, as long as a sane compiler is used?