Augur Refine fails but does not generate an error log

Hi everyone,

I am currently trying to make a Louisiana focused build using data from Gisaid and their North-America focused build as multiple inputs. After some f trial and error with formatting the input data, I got to the tree refinement.
When running the command it will run for about an hour and then fails without an error log or error message.
The output is shown below

The log file is just an empty file.

I looked at the topics posted on here and double checked my metadata names matches the tree file , tried running it with a different environment using the -use-conda flag and updated the packages using conda activate nextstrain conda update --all

Do you have any suggestions I could try?

The error message as text
` augur refine --tree results/louisiana/tree_raw.nwk --alignment results/louisiana/aligned.fasta --metadata results/combined_metadata.tsv --output-tree results/louisiana/tree.nwk --output-node-data re sults/louisiana/branch_lengths.json --root hCoV-19/Wuhan/Hu-1/2019 hCoV-19/Wuhan/WH01/2019 --timetree --clock-rate 0.0008 --clock-std-dev 0.0004 --coalescent skyline --date-inferenc e marginal --divergence-unit mutations --date-confidence --no-covariance --clock-filt er-iqd 4 2>&1 | tee logs/refine_louisiana.txt

Activating conda environment: /home/evtlabuser/nextstrain/COVID/.snakemake/conda/4b3ea613
/bin/bash: line 1: 21239 Killed augur refine --tree results/louisiana/tree_raw.nwk --alignment results/louisiana/ali gned.fasta --metadata results/combined_metadata.tsv --output-tree results/louisiana/tree.nwk --output-node-data results/louisiana/bra nch_lengths.json --root hCoV-19/Wuhan/Hu-1/2019 hCoV-19/Wuhan/WH01/2019 --timetree --clock-rate 0.0008 --clock-std-dev 0.0004 --coale scent skyline --date-inference marginal --divergence-unit mutations --date-confidence --no-covariance --clock-filter-iqd 4 2>&1
21240 Done | tee logs/refine_louisiana.txt
[Thu Apr 8 18:17:42 2021]
Error in rule refine:
jobid: 4
output: results/louisiana/tree.nwk, results/louisiana/branch_lengths.json
log: logs/refine_louisiana.txt (check log file(s) for error message)
conda-env: /home/evtlabuser/nextstrain/COVID/.snakemake/conda/4b3ea613
shell:

    augur refine             --tree results/louisiana/tree_raw.nwk             --alignment results/louisiana/aligned.fasta                                                  --metadata results/combined_metadata.tsv             --output-tree results/louisiana/tree.nwk             --output-node-data re                                     sults/louisiana/branch_lengths.json             --root hCoV-19/Wuhan/Hu-1/2019 hCoV-19/Wuhan/WH01/2019             --timetree                                                               --clock-rate 0.0008             --clock-std-dev 0.0004             --coalescent skyline             --date-inferenc                                     e marginal             --divergence-unit mutations             --date-confidence             --no-covariance             --clock-filt                                     er-iqd 4 2>&1 | tee logs/refine_louisiana.txt

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Logfile logs/refine_louisiana.txt:

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message`

The log files
image

Hi @MaartenVanDiest, welcome! Sorry to hear you’re having trouble with your build.

The tell-tale clue in the output you shared is:

/bin/bash: line 1: 21239 Killed augur refine

This indicates that something, most likely your operating system, forcibly stopped the augur refine process (process id 21239), most likely because your system was experiencing an out-of-memory condition and started terminating processes to free up memory.

How much memory is available to your workflow run?

Our rough estimate for memory usage for the refine step of the workflow is about 15× the size on disk of the input MSA:

That’s a good to know. I have 2 GB available. I will have it extended to 16 GB to run properly and try again. Thanks for the help, Thomas!

Extending the memory fixed all the issues.

1 Like