Dynamic linking during package precompilation fails in RStudio but not in R or Julia

[edwardlavender]

I am on MacOS Sonoma 14.6.1, R version 4.4.1, RStudio version 2024.12.0+467 (2024.12.0+467), Julia version 1.10.5.

I have a lot of trouble with the ArchGDAL package in RStudio, but not Julia.

Here is a reproducible example.

I am using JuliaCall ‘0.17.6’. My base Julia environment looks like this:

Status `~/.julia/environments/v1.10/Project.toml`
  [31a5f54b] Debugger v0.7.10
  [35a29f4d] DocumenterTools v0.1.20
  [aa1ae85d] JuliaInterpreter v0.9.41
  [6f49c342] RCall v0.14.6
  [295af30f] Revise v3.7.2
  [fd094767] Suppressor v0.2.8

In RStudio, I set up a new, local project and load ArchGDAL:

library(JuliaCall)
julia_setup()
julia_library("Pkg")
julia_command('Pkg.activate(".")')
julia_install_package("ArchGDAL")
julia_library("ArchGDAL")

In RStudio, this fails with:

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/libproj.25.dylib"
dlopen(/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/libproj.25.dylib, 0x0001): Library not loaded: @rpath/libsqlite3.0.dylib
Referenced from: <4C4C4476-5555-3144-A137-D6738C18B715> /Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/libproj.25.9.5.1.dylib
Reason: tried: '/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/lib/julia/libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/lib/julia/../libsqlite3.0.dylib' (no such file), '/Library/Frameworks/R.framework/Re

I tried some more fiddling:

julia_command("Pkg.gc()")
julia_command("Pkg.resolve()")
julia_command("Pkg.build()")
julia_command("Pkg.precompile()")
julia_command("using ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/libgeotiff.5.3.2.dylib"
dlopen(/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/libgeotiff.5.3.2.dylib, 0x0001): Library not loaded: @rpath/libproj.25.dylib
  Referenced from: <4C4C4488-5555-3144-A15D-2913CDF72C04> /Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/libgeotiff.5.3.2.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/./libproj.25.dylib' (no such file), '/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/./libproj.25.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/lib/julia/libproj.25.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/lib/julia/../libproj.25.dylib' (no such file), '/Library/Frameworks/R.framework/
  

In pure R, surprisingly, this works:

library(JuliaCall)
julia_setup()
julia_command('Pkg.activate(".")')
julia_library("ArchGDAL")

In Julia, this also works:

julia> using Pkg
julia> Pkg.activate(".")
Activating project at `~/Desktop/ArchGDAL-hell`
julia> using ArchGDAL
julia>

In RStudio, I noticed PATH is different and RSTUDIO_FALLBACK_LIBRARY_PATH is set:

> system("env")

PATH=/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/usr/local/MacGPG2/bin:/Applications/quarto/bin:/Library/TeX/texbin:/usr/texbin:/Applications/RStudio.app/Contents/Resources/app/quarto/bin:/Applications/RStudio.app/Contents/Resources/app/bin/postback

RSTUDIO_FALLBACK_LIBRARY_PATH=/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/rstudio-fallback-library-path-1332363136

Whereas in pure R:

> system("env")

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
__CFBundleIdentifier=org.R-project.R

(LD_LIBRARY_PATH does not appear in system("env") in RStudio or R. )

In RStudio, I tried unsetting RSTUDIO_FALLBACK_LIBRARY_PATH without success:

library(JuliaCall)
Sys.unsetenv("RSTUDIO_FALLBACK_LIBRARY_PATH")
Sys.unsetenv("DYLD_LIBRARY_PATH")  # Just in case
Sys.unsetenv("LD_LIBRARY_PATH")    # Just in case

julia_setup()
julia_library("Pkg")
julia_command('Pkg.activate(".")')
julia_install_package("ArchGDAL")
julia_library("ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/libgdal.36.dylib"
dlopen(/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/libgdal.36.dylib, 0x0001): Library not loaded: @rpath/libgeotiff.5.dylib
  Referenced from: <4C4C440B-5555-3144-A182-91BFB6306FAD> /Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/libgdal.36.3.10.1.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/./libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/./libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/lib/julia/libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/lib/julia/../libgeotiff.5.dylib' (no such file), '/Library/Frameworks/R.framework/R

ChatGPT also suggested starting RStudio with:

env -i PATH="$PATH" open -a RStudio

But this is not solve the issue.

Does anyone have any experience with this? Any advice would be much appreciated!

[edwardlavender]

Update: I updated R and Julia to the latest release version but this did not solve the problem. In RStudio I still get an error:

julia_library("ArchGDAL")
Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/53a1f1913ffb77ca87b5aaa9af754b865706047f/lib/libproj.25.dylib"
dlopen(/Users/lavended/.julia/artifacts/53a1f1913ffb77ca87b5aaa9af754b865706047f/lib/libproj.25.dylib, 0x0001): Library not loaded: @rpath/libsqlite3.0.dylib
  Referenced from: <4C4C44E0-5555-3144-A1C4-44156A1331B9> /Users/lavended/.julia/artifacts/53a1f1913ffb77ca87b5aaa9af754b865706047f/lib/libproj.25.9.3.0.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/53a1f1913ffb77ca87b5aaa9af754b865706047f/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/artifacts/53a1f1913ffb77ca87b5aaa9af754b865706047f/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/../libsqlite3.0.dylib' (no such file), '/Library/Frameworks/R.framework/Re

[edwardlavender]

Update: I discovered that if you run julia_library("ArchGDAL") in RStudio five times it cycles through different errors until eventually no error message is returned. However, reading an example raster still fails.

# [1]
julia_library("ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/libproj.25.dylib"
dlopen(/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/libproj.25.dylib, 0x0001): Library not loaded: @rpath/libsqlite3.0.dylib
Referenced from: <4C4C4476-5555-3144-A137-D6738C18B715> /Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/libproj.25.9.5.1.dylib
Reason: tried: '/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/artifacts/a799f4ba8b350a440ab68a5c5c0358b5637930e8/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/../libsqlite3.0.dylib' (no such file), '/Library/Frameworks/R.framework/Re

# [2]
julia_library("ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/libgeotiff.5.3.2.dylib"
dlopen(/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/libgeotiff.5.3.2.dylib, 0x0001): Library not loaded: @rpath/libproj.25.dylib
Referenced from: <4C4C4488-5555-3144-A15D-2913CDF72C04> /Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/libgeotiff.5.3.2.dylib
Reason: tried: '/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/./libproj.25.dylib' (no such file), '/Users/lavended/.julia/artifacts/d8b8ab4432262c09a95b48d10e4398e049530171/lib/./libproj.25.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/libproj.25.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/../libproj.25.dylib' (no such file), '/Library/Frameworks/R.framework/

# [3]
julia_library("ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/libgdal.36.dylib"
dlopen(/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/libgdal.36.dylib, 0x0001): Library not loaded: @rpath/libgeotiff.5.dylib
Referenced from: <4C4C440B-5555-3144-A182-91BFB6306FAD> /Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/libgdal.36.3.10.1.dylib
Reason: tried: '/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/./libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/artifacts/cabda474bc44103fddd19d28b132ad4758052739/lib/./libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/../libgeotiff.5.dylib' (no such file), '/Library/Frameworks/R.framework/R

# [4]
julia_library("ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library ""
dlopen(.dylib, 0x0001): tried: '.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/../.dylib' (no such file), '/usr/lib/.dylib' (no such file, not in dyld cache), '.dylib' (no such file), '/Library/Frameworks/R.framework/Resources/lib/.dylib' (no such file), '/Library/Java/JavaVirtualMachines/jdk-11.0.18+10/Contents/Home/lib/server/.dylib' (no such file), '/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/rstudio-fallback-library-path-731816702/.dylib' (no such file)
Stacktrace:
  [1] cplseterrorhandler
@ ~/.julia/packages/GDAL/8oAvY/src/libgdal.jl:2145 [inlined]
[2] __init__()
@ GDAL ~/.julia/packages/GDAL/8oAvY/src/GDAL.jl:27
[3] run_module_init(mod::Module, i::Int64)
@ Base ./loading.jl:

This now works:

# [5]:
julia_library("ArchGDAL")

But ArchGDAL.readraster() fails:

julia_command('dataset = ArchGDAL.readraster("https://raw.githubusercontent.com/edwardlavender/patter/main/inst/extdata/dat_gebco.tif")')

Error: Error happens in Julia.
could not load library ""
dlopen(.dylib, 0x0001): tried: '.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.11.3+0.aarch64.apple.darwin14/lib/julia/../.dylib' (no such file), '/usr/lib/.dylib' (no such file, not in dyld cache), '.dylib' (no such file), '/Library/Frameworks/R.framework/Resources/lib/.dylib' (no such file), '/Library/Java/JavaVirtualMachines/jdk-11.0.18+10/Contents/Home/lib/server/.dylib' (no such file), '/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/rstudio-fallback-library-path-281754957/.dylib' (no such file)
Stacktrace:
  [1] gdalopenex(pszFilename::String, nOpenFlags::Int64, papszAllowedDrivers::Ptr{Cstring}, papszOpenOptions::Ptr{Cstring}, papszSiblingFiles::Ptr{Cstring})
@ GDAL ~/.julia/packages/GDAL/8oAvY/src/libgdal.jl:7211
[2] #read#79
@ ~/.juli

Surprisingly julia_library("ArchGDAL") and ArchGDAL.readraster() appear to work in pure R.

sessionInfo() looks identical in R and RStudio:

R version 4.4.2 (2024-10-31)
Platform: aarch64-apple-darwin20
Running under: macOS Sonoma 14.6.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/Zurich
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] JuliaCall_0.17.6

loaded via a namespace (and not attached):
[1] compiler_4.4.2 tools_4.4.2    Rcpp_1.0.14    knitr_1.49     xfun_0.50      evaluate_1.0.3

[ViralBShah]

The fundamental issue here is that R and Julia are both trying to use their own versions of this library, and unless they are using the same ones, there will be clashes one way or another.