Augur frequencies error: RuntimeError: `nodes` value in results/_tip_frequencies.json is not a dictionary. Please check the formatting of this JSON!

Hello.

First of all, allow me to congratulate you all for the effort and love you’ve poured into the nextstrain sweet suite tool of tools. We appreciate it fully. It has deeply improved our understanding and analyses of our favorite organisms.

In that regard, I’m generating the tip_frequencies.json with no success. By the way, I don’t use the snakemake build format, I use augur with traditional and simple command line syntax with no problems.

My issue here, is that I’m stuck with the following error at the final step augur export v2:

$ augur export v2 --tree results/tree.nwk --metadata data/metadata-g.tsv --node-data results/branch_lenghts.json results/traits.json results/nt_muts.json results/aa_muts.json results/tip_frequencies.json --colors config/colors-g.tsv --lat-longs config/lat_longs-g.tsv --auspice-config config/auspice_config5-g.json --output auspice/21-hs-inv20-ago23-g.json

Validating schema of ‘results/aa_muts.json’…

Traceback (most recent call last):
File “/home/bioinfo1/miniconda3/envs/nextstrain/bin/augur”, line 10, in
sys.exit(main())
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/main.py”, line 10, in main
return augur.run( argv[1:] )
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/init.py”, line 75, in run
return args.command.run(args)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/export.py”, line 22, in run
return run_v2(args)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/export_v2.py”, line 903, in run_v2
node_data, node_attrs, node_data_names, metadata_names = parse_node_data_and_metadata(T, args.node_data, args.metadata)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/export_v2.py”, line 850, in parse_node_data_and_metadata
node_data = read_node_data(node_data_files) # node_data_files is an array of multiple files (or a single file)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/utils.py”, line 191, in read_node_data
return NodeDataReader(fnames, tree).read()
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_reader.py”, line 26, in read
node_data = self.build_node_data()
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_reader.py”, line 35, in build_node_data
for node_data_file in self.node_data_files:
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_reader.py”, line 51, in
return (NodeDataFile(fname) for fname in self.filenames)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_file.py”, line 18, in init
self.validate()
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_file.py”, line 68, in validate
raise RuntimeError(
RuntimeError: nodes value in results/_tip_frequencies.json is not a dictionary. Please check the formatting of this JSON!

As you can see, all my outputs are called from results/

On the other hand, if I run the same augur export v2 command, but without the argument results/tip_frequencies.json , it works flawlessly… of course, without integrating the frequencies to the final .json file. What I’m trying to say is that the aa_muts.json gets properly validated along everything else.

Thank you in advance for your kind answers, I’m really in the need for advice here.

Best regards!

Hi @vicss, welcome! Glad to hear Nextstrain’s tools are helpful for
you!

It sounds like your results/aa_muts.json node data file is fine, but results/tip_frequencies.json is malformed. In order to debug, could you share two things with us?

  1. The tip_frequencies.json file so we can inspect its contents.
  2. The exact command line you’re using to generate the tip_frequencies.json file.

These will help narrow down the issue. The version of Augur you’re running (augur --version) might also be helpful.

Thanks!

Hi @trs, thanks a lot for your reply!

Regarding the augur version, is augur 12.0.0.

About the command line, for the tip_frequencies.json:
$ augur frequencies --method kde --metadata data/metadata-g.tsv --tree results/tree.nwk --output results/tip_frequencies.json
for the file tip_frequencies.json As you can see, its the basic (but might be wrong as you said).

I calculated as well a slightly more complex _tip_frequencies.json with:
$ augur frequencies --method kde --metadata data/metadata-g.tsv --tree results/tree.nwk --min-date 2020-11-12 --max-date 2021-07-23 --pivot-interval-units weeks --include-internal-nodes --output results/_tip_frequencies.json into the file: _tip_frequencies.json

Both “versions” yield the exact same aforementioned error.

Thank you for your kind attention and help!

PD, did a correction (sept,10) at the tip_frequencies command, since its metadata-g.tsv the correct file and no metadata.tsv as stated initially. The respective tip_frequencies.json was corrected and uploaded as well.

Best regards

Ah! I’m sorry @vicss, I think I’ve accidentally mislead you here! I wasn’t thinking clearly about what the tip_frequencies.json file actually was when thinking about your issue. Once I saw what you uploaded and the command-line you used, it clicked and I realized the problem. Thank you for collecting the information!

Unlike Augur node data files, which get passed into augur export v2, the tip-frequencies file is an Auspice sidecar file, which doesn’t get incorporated into the final main Auspice dataset file. (See our new summary of data formats.)

Instead, the tip-frequencies file gets uploaded alongside the main dataset file (hence “sidecar”) with a common base filename, e.g.:

auspice/21-hs-inv20-ago23-g.json
auspice/21-hs-inv20-ago23-g_tip-frequencies.json

and Auspice knows to look for it when your Auspice config (config/auspice_config5-g.json) includes "frequencies" in the panels section (example) or --panels is passed to augur export v2 (doc).

Let me know if this helps, or if you need some more guidance on putting it all together. Unfortunately, I don’t think we yet have good docs on setting up frequencies.