Snakemake throwing IncompleteFilesException when using --forceall?

1

Hi,
In the nextstrain ncov build directory I normally use the --forceall flag to nextstrain build to rebuild from scratch, but now I get a message to use --rerun-incomplete and the build process exists. Is there not a way to force the build to start from the beginning, with downloading the reference dataset etc, or will this be done with the --rerun-incomplete flag?
Thanks,
Jon

2

Hmm. Those are options passed to Snakemake and should still be supported. Can you share the exact output you get when you use --forceall?

3

This is the output:

(base) jonr@jonr-HP-Z4-G4-Workstation:~/Prosjekter/Nextstrain_mamba/ncov$ nextstrain build . --configfile my_profiles/omicron/builds.yaml --cores 10 --forceall
Config file defaults/parameters.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with

    snakemake --cleanup-metadata <filenames>

To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
results/aligned_gisaid.fasta.xz
results/insertions_gisaid.tsv
results/translations/seqs_gisaid.gene.ORF1a.fasta.xz
results/translations/seqs_gisaid.gene.ORF1b.fasta.xz
results/translations/seqs_gisaid.gene.S.fasta.xz
results/translations/seqs_gisaid.gene.ORF3a.fasta.xz
results/translations/seqs_gisaid.gene.E.fasta.xz
results/translations/seqs_gisaid.gene.M.fasta.xz
results/translations/seqs_gisaid.gene.ORF6.fasta.xz
results/translations/seqs_gisaid.gene.ORF7a.fasta.xz
results/translations/seqs_gisaid.gene.ORF7b.fasta.xz
results/translations/seqs_gisaid.gene.ORF8.fasta.xz
results/translations/seqs_gisaid.gene.N.fasta.xz
results/translations/seqs_gisaid.gene.ORF9b.fasta.xz
4

Snakemake seems to be (maybe newly?) insisting on reconciling its metadata of the output files and the actual output files present before it’ll force re-run. That’s unexpected to me, but it’s not the first time Snakemake’s made behaviour choices that don’t make sense to me. You could do as it says and run

nextstrain build . \
    --configfile my_profiles/omicron/builds.yaml \
    --cleanup-metadata \
    <filenames…>

Or explicitly remove/clean up all existing output files instead

nextstrain build . clean --configfile my_profiles/omicron/builds.yaml

and then rebuild.

1 Like
5

I’m not sure whether this behaviour is new or not. To work around this, what I do is to simply always pass --ri (short for --rerun-incomplete), which does no harm if there are no incomplete metadata, in addition to other flags, like --forceall or (-F short).

I don’t think anything has changed within snakemake, I looked briefly into this but the exception has been there for 10 years.

1 Like