Iqtree not using all threads

Hello. I feel like I am perhaps missing something obvious. I’m running some custom ncov analyses using the snakemake in the shell version of nextstrain, and while I specified in the configuration file that I want the analysis to use all twelve cores, it’s only using one. iqtree has been running for two days straight, and if I could figure out what’s wrong, I could hopefully get this done quite a bit quicker.

System: Ubuntu 16.04
Processor: Intel® Xeon® CPU E5-2620 v2 @ 2.10GHz × 12
Memory: 62.9 GiB

Both those commands are passing an argument to use 12 threads, but the top command tells me that only one thread is in use, and of the 12 CPUs, one is at 100%, and the rest are at around 5%.

Hi @mbosmeny – what command are you calling snakemake with?

You may need to add --cores 12 as an argument to snakemake (and similarly for memory, via --resources mem_mb=32000 or similar).

In order to test whether this is indeed a snakemake thing or something else entirely, you could try running the augur tree ... command itself (i.e. outside of snakemake).

The snakemake command itself is pretty straightforward:

snakemake --profile ./my_profiles/test

However, I do have 12 cores specified in my config.yaml file:

# Set the maximum number of cores you want Snakemake to use for this pipeline. cores: 12

Upon further testing, the other steps in the pipeline use multiple cores. It seems like it’s exclusing to iqtree that I’m having this problem. I suspect it might just be a step in the tree building process that can’t be multiplexed very well?

Hi @mbosmeny! You are correct that not all internal steps of IQ-TREE’s tree inference can be parallelized, so there will be times when IQ-TREE only uses a single core.

One way to check what is happening behind the scenes is to inspect the IQ-TREE log. In our “ncov” workflow, this file is written out to your build’s results directory (e.g., results/test/) with an extension of .iqtree.log. This file contains a report of how many cores IQ-TREE thinks it has and how much time it spends on each step of the inference. You may also try running IQ-TREE manually with -nt AUTO and inspecting the beginning of the log output. In this mode, IQ-TREE will try to determine the optimal number of cores to use for tree building. If you see unexpected output from this command, that may help you track down the issue.