© G. Dalle, J. Smit, A. Hill.
© G. Dalle, J. Smit, A. Hill.
In this post, you will learn about tools to initialize, structure and distribute Julia packages.
A vast majority of Julia packages are hosted on GitHub (although less common, other options like GitLab are also possible). GitHub is a platform for collaborative software development, based on the version control system Git. If you are unfamiliar with these technologies, check out the GitHub documentation.
The first step is therefore creating an empty GitHub repository.
You should try to follow package naming guidelines and add a ".jl" extension at the end, like so: "MyAwesomePackage.jl".
Do not insert any files like README.md, .gitignore or LICENSE.md, this will be done for you in the next step.
Indeed, we can leverage PkgTemplates.jl to automate package creation (like ]generate from Pkg.jl but on steroids).
The following code gives you a basic file structure to start with:
julia> using PkgTemplates
julia> dir = Utils.path(:site) # replace with the folder of your choice
/home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site
julia> t = Template(dir=dir, user="myuser", interactive=false);
julia> t("MyAwesomePackage")
[ Info: Running prehooks
[ Info: Running hooks
Activating project at `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage`
Updating registry at `~/.julia/registries/General.toml`
No Changes to `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage/Project.toml`
No Changes to `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage/Manifest.toml`
Precompiling project...
✓ MyAwesomePackage
1 dependency successfully precompiled in 0 seconds
Info We haven't cleaned this depot up for a bit, running Pkg.gc()...
Active manifest files: 4 found
Active artifact files: 3 found
Active scratchspaces: 0 found
Deleted no artifacts, repos, packages or scratchspaces
Activating project at `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/sharing`
[ Info: Running posthooks
[ Info: New package is at /home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage
/home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage
Then, you simply need to push this new folder to the remote repository https://github.com/myuser/MyAwesomePackage.jl, and you're ready to go. The rest of this post will explain to you what each part of this folder does, and how to bend them to your will.
To work on the package further, we develop it into the current environment and import it:
julia> using Pkg
julia> Pkg.develop(path=sitepath("MyAwesomePackage")) # ignore sitepath
Resolving package versions...
Updating `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/sharing/Project.toml`
[18d36611] + MyAwesomePackage v1.0.0-DEV `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage`
Updating `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/sharing/Manifest.toml`
[18d36611] + MyAwesomePackage v1.0.0-DEV `~/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage`
julia> using MyAwesomePackage
The most useful aspect of PkgTemplates.jl is that it automatically generates workflows for GitHub Actions.
These are stored as YAML files in .github/workflows, with a slightly convoluted syntax that you don't need to fully understand.
For instance, the file CI.yml contains instructions that execute the tests of your package (see below) for each pull request, tag or push to the main branch.
This is done on a GitHub server and should theoretically cost you money, but your GitHub repository is public, you get an unlimited workflow budget for free.
A variety of workflows and functionalities are available through optional plugins.
The interactive setting Template(..., interactive=true) allows you to select the ones you want for a given package.
Otherwise, you will get the default selection, which you are encouraged to look at.
The purpose of the test subfolder in your package is unit testing: automatically checking that your code behaves the way you want it to.
For instance, if you write your own square root function, you may want to test that it gives the correct results for positive numbers, and errors for negative numbers.
julia> using Test
julia> @test sqrt(4) ≈ 2
Test Passed
julia> @testset "Invalid inputs" begin
@test_throws DomainError sqrt(-1)
@test_throws MethodError sqrt("abc")
end;
Test Summary: | Pass Total Time
Invalid inputs | 2 2 0.0s
Such tests belong in test/runtests.jl, and they are executed with the ]test command (in the REPL's Pkg mode).
Unit testing may seem rather naive, or even superfluous, but as your code grows more complex, it becomes easier to break something without noticing.
Testing each part separately will increase the reliability of the software you write.
Advanced: To test the arguments provided to the functions within your code (for instance their sign or value), avoid @assert (which can be deactivated) and use ArgCheck.jl instead.
At some point, your package may require test-specific dependencies.
This often happens when you need to test compatibility with another package, on which you do not depend for the source code itself.
Or it may simply be due to testing-specific packages like the ones we will encounter below.
For interactive testing work, use TestEnv.jl to activate the full test environment (faster than running ]test repeatedly).
VSCode: The Julia extension also has its own testing framework, which relies on sprinkling "test items" throughout the code. See TestItemRunner.jl and ReTestItems.jl for indications on how to use them optimally.
Advanced: If you want to have more control over your tests, you can try
Code coverage refers to the fraction of lines in your source code that are covered by tests. It is a good indicator of the exhaustiveness of your test suite, albeit not sufficient. Codecov is a website that provides easy visualization of this coverage, and many Julia packages use it. It is available as a PkgTemplates.jl plugin, but you have to perform an additional configuration step on the repo for Codecov to communicate with it.
To make your code easy to read, it is essential to follow a consistent set of guidelines. The official style guide is very short, so most people use third party style guides like BlueStyle or SciMLStyle.
JuliaFormatter.jl is an automated formatter for Julia files which can help you enforce the style guide of your choice.
Just add a file .JuliaFormatter.toml at the root of your repository, containing a single line like
style = "blue"Then, the package directory will be formatted in the BlueStyle whenever you call
julia> using JuliaFormatter
julia> JuliaFormatter.format(MyAwesomePackage)
true
VSCode: The default formatter falls back on JuliaFormatter.jl.
Advanced: You can format code automatically in GitHub pull requests with the julia-format action, or add the formatting check directly to your test suite.
Of course, there is more to code quality than just formatting. Aqua.jl provides a set of routines that examine other aspects of your package, from unused dependencies to ambiguous methods. It is usually a good idea to include the following in your tests:
julia> using Aqua, MyAwesomePackage
julia> Aqua.test_all(MyAwesomePackage)
LoadError: Some tests did not pass: 3 passed, 1 failed, 0 errored, 0 broken.
in expression starting at string:1
Meanwhile, JET.jl is a complementary tool, similar to a static linter. Here we focus on its error analysis, which can detect errors or typos without even running the code by leveraging type inference. You can either use it in report mode (with a nice VSCode display) or in test mode as follows:
julia> using JET, MyAwesomePackage
julia> JET.report_package(MyAwesomePackage)
[toplevel-info] virtualized the context of Main (took 0.012 sec)
[toplevel-info] entered into /home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage/src/MyAwesomePackage.jl
[toplevel-info] exited from /home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage/src/MyAwesomePackage.jl (took 0.009 sec)
[toplevel-info] analyzed 0 top-level definitions (took 0.006 sec)
JET.JETToplevelResult{JET.JETAnalyzer{JET.BasicPass}, Base.Pairs{Symbol, Any, NTuple{4, Symbol}, @NamedTuple{concretization_patterns::Vector{Symbol}, analyze_from_definitions::Bool, toplevel_logger::IOContext{Base.PipeEndpoint}, ignore_missing_comparison::Bool}}}(JET.JETAnalyzer{JET.BasicPass}(JET.AnalyzerState(0x0000000000007b7e, Core.Compiler.InferenceResult[], Core.Compiler.InferenceParams(3, 4, 8, 32, 3, true, true, false, true, false), Core.Compiler.OptimizationParams(true, 100, 1000, 250, 20, 32, true, false, false), IdDict{Core.Compiler.InferenceResult, Union{JET.AnalysisResult, JET.CachedAnalysisResult}}(), JET.InferenceErrorReport[], nothing, Bool[], JET.__toplevelmod__, Dict{Int64, Symbol}(), nothing, 0), JET.AnalysisCache(IdDict{Core.MethodInstance, Core.CodeInstance}()), JET.BasicPass(), Core.Compiler.CachedMethodTable{Core.Compiler.OverlayMethodTable}(Core.Compiler.IdDict{Core.Compiler.MethodMatchKey, Union{Nothing, Core.Compiler.MethodMatchResult}}(Any[#undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef, #undef], 0, 0), Core.Compiler.OverlayMethodTable(0x0000000000007b7e, # 1 method for callable object:
[1] iterate(::Tuple{}, ::Int64)
@ ~/.julia/packages/JET/1ZQeA/src/analyzers/jetanalyzer.jl:184)), JET.JETAnalyzerConfig(true)), JET.VirtualProcessResult(Set(["/home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage/src/MyAwesomePackage.jl"]), String[], Set(Module[Main.var"##JETVirtualModule#225", Main.var"##JETVirtualModule#225".MyAwesomePackage]), JET.ToplevelErrorReport[], JET.InferenceErrorReport[], Type[], Main => Main.var"##JETVirtualModule#225"), "JETAnalyzer: \"/home/runner/work/modernjuliaworkflows.github.io/modernjuliaworkflows.github.io/__site/MyAwesomePackage/src/MyAwesomePackage.jl\"", Base.Pairs{Symbol, Any, NTuple{4, Symbol}, @NamedTuple{concretization_patterns::Vector{Symbol}, analyze_from_definitions::Bool, toplevel_logger::IOContext{Base.PipeEndpoint}, ignore_missing_comparison::Bool}}(:concretization_patterns => [:x_], :analyze_from_definitions => true, :toplevel_logger => IOContext(Base.PipeEndpoint(RawFD(4294967295) closed, 0 bytes waiting)), :ignore_missing_comparison => true))
julia> JET.test_package(MyAwesomePackage)
Test Passed
Note that both Aqua.jl and JET.jl might pick up false positives: refer to their respective documentations for ways to make them less sensitive.
Finally, ExplicitImports.jl can help you get rid of generic imports to specify where each of the names in your package comes from. This is a good practice and makes your code more robust to name conflicts between dependencies.
Even if your code does everything it is supposed to, it will be useless to others (and pretty soon to yourself) without proper documentation. Adding docstrings everywhere needs to become a second nature. This way, readers and users of your code can query them through the REPL help mode. DocStringExtensions.jl provides a few shortcuts that can speed up docstring creation by taking care of the obvious parts.
"""
myfunc(a, b; kwargs...)
One-line sentence describing the purpose of the function,
just below the (indented) signature.
More details if needed.
"""
function myfunc end;However, package documentation is not limited to docstrings.
It can also contain high-level overviews, technical explanations, examples, tutorials, etc.
Documenter.jl allows you to design a website for all of this, based on Markdown files contained in the docs subfolder of your package.
Unsurprisingly, its own documentation is excellent and will teach you a lot.
To build the documentation locally, just run
julia> using Pkg
julia> Pkg.activate("docs")
julia> include("docs/make.jl")Then, use LiveServer.jl from your package folder to visualize and automatically update the website as the code changes (similar to Revise.jl):
julia> using LiveServer
julia> servedocs()To host the documentation online easily, just select the Documenter plugin from PkgTemplates.jl before creation.
Not only will this fill the docs subfolder with the right contents: it will also initialize a GitHub Actions workflow to build and deploy your website on GitHub pages.
The only thing left to do is to select the gh-pages branch as source.
Advanced: You may find the following Documenter plugins useful:
Assuming you are looking for an alternative to Documenter.jl, you can try out Pollen.jl. In another category, Replay.jl allows you to replay instructions entered into your terminal as an ASCII video, which is nice for tutorials.
Scientific software is often hard to grasp, and the code alone may not be very enlightening. Whether it is for package documentation or to write papers and books, you might want to interleave code with texts, formulas, images and so on. In addition to the Pluto.jl and Jupyter notebooks, take a look at Literate.jl to enrich your code with comments and translate it to various formats. Quarto is another cross-language notebook system that supports Python, R and Julia, while Books.jl is more relevant to draft long documents.
The Julia community has adopted semantic versioning, which means every package must have a version, and the version numbering follows strict rules.
The main consequence is that you need to specify compatibility bounds for your dependencies: this happens in the [compat] section of your Project.toml.
To initialize these bounds, use the ]compat command in the Pkg mode of the REPL, or the package PackageCompatUI.jl.
As your package lives on, new versions of your dependencies will be released.
The CompatHelper.jl GitHub Action will help you monitor Julia dependencies and update your Project.toml accordingly.
In addition, Dependabot can monitor the dependencies... of your GitHub actions themselves.
But don't worry: both are default plugins in the PkgTemplates.jl setup.
Advanced: It may also happen that you incorrectly promise compatibility with an old version of a package. To prevent that, the julia-downgrade-compat GitHub action tests your package with the oldest possible version of every dependency, and verifies that everything still works.
If your package is useful to others in the community, it may be a good idea to register it, that is, make it part of the pool of packages that can be installed with
pkg> add MyAwesomePackage # made possible by registrationNote that unregistered packages can also be installed by anyone from the GitHub URL, but this a less reproducible solution:
pkg> add https://github.com/myuser/MyAwesomePackage # not idealTo register your package, check out the general registry guidelines. The Registrator.jl bot can help you automate the process. Another handy bot, provided by default with PkgTemplates.jl, is TagBot: it automatically tags new versions of your package following each registry release. If you have performed the necessary SSH configuration, TagBot will also trigger documentation website builds following each release.
Advanced: If your package is only interesting to you and a small group of collaborators, or if you don't want to make it public, you can still register it by setting up a local registry: see LocalRegistry.jl.
Obtaining consistent and reproducible results is an essential part of experimental science. DrWatson.jl is a general toolbox for running and re-running experiments in an orderly fashion. We now explore a few specific issues that often arise.
A first hurdle is random number generation, which is not guaranteed to remain stable across Julia versions. To ensure that the random streams remain exactly the same, you need to use StableRNGs.jl. Another aspect is dataset download and management. The packages DataDeps.jl, DataToolkit.jl and ArtifactUtils.jl can help you bundle non-code elements with your package. A third thing to consider is proper citation and versioning. Giving your package a DOI with Zenodo ensures that everyone can properly cite it in scientific publications. Similarly, your papers should cite the packages you use as dependencies: PkgCite.jl will help with that.
To ensure compatibility with earlier Julia versions, Compat.jl is your best ally.
Making packages play nice with one another is a key goal of the Julia ecosystem. Since Julia 1.9, this can be done with package extensions, which override specific behaviors based on the presence of a given package in the environment. PackageExtensionTools.jl eases the pain of setting up extensions.
Furthermore, the Julia ecosystem as a whole plays nice with other programming languages too. C and Fortran are natively supported. Python can be easily interfaced with the combination of CondaPkg.jl and PythonCall.jl. Other language compatibility packages can be found in the JuliaInterop organization, like RCall.jl.
Part of interoperability is also flexibility and customization: the Preferences.jl package gives a nice way to specify various options in TOML files.
Advanced: Some package developers may need to define what kind of behavior they expect from a certain type, or what a certain method should do. When writing it in the documentation is not enough, a formal testable specification becomes necessary. This problem of "interfaces" does not yet have a definitive solution in Julia, but several options have been proposed: Interfaces.jl, RequiredInterfaces.jl and PropCheck.jl are all worth checking out.
Once your package grows big enough, you might need to bring in some help. Working together on a software project has its own set of challenges, which are partially addressed by a good set of ground rules liks SciML ColPrac. Of course, collaboration goes both ways: if you find a Julia package you really like, you are more than welcome to contribute as well, for example by opening issues or submitting pull requests.