Augur frequencies error: RuntimeError: `nodes` value in results/_tip_frequencies.json is not a dictionary. Please check the formatting of this JSON!

General
1

Hello.

First of all, allow me to congratulate you all for the effort and love you’ve poured into the nextstrain sweet suite tool of tools. We appreciate it fully. It has deeply improved our understanding and analyses of our favorite organisms.

In that regard, I’m generating the tip_frequencies.json with no success. By the way, I don’t use the snakemake build format, I use augur with traditional and simple command line syntax with no problems.

My issue here, is that I’m stuck with the following error at the final step augur export v2:

$ augur export v2 --tree results/tree.nwk --metadata data/metadata-g.tsv --node-data results/branch_lenghts.json results/traits.json results/nt_muts.json results/aa_muts.json results/tip_frequencies.json --colors config/colors-g.tsv --lat-longs config/lat_longs-g.tsv --auspice-config config/auspice_config5-g.json --output auspice/21-hs-inv20-ago23-g.json

Validating schema of ‘results/aa_muts.json’…

Traceback (most recent call last):
File “/home/bioinfo1/miniconda3/envs/nextstrain/bin/augur”, line 10, in
sys.exit(main())
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/main.py”, line 10, in main
return augur.run( argv[1:] )
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/init.py”, line 75, in run
return args.command.run(args)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/export.py”, line 22, in run
return run_v2(args)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/export_v2.py”, line 903, in run_v2
node_data, node_attrs, node_data_names, metadata_names = parse_node_data_and_metadata(T, args.node_data, args.metadata)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/export_v2.py”, line 850, in parse_node_data_and_metadata
node_data = read_node_data(node_data_files) # node_data_files is an array of multiple files (or a single file)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/utils.py”, line 191, in read_node_data
return NodeDataReader(fnames, tree).read()
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_reader.py”, line 26, in read
node_data = self.build_node_data()
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_reader.py”, line 35, in build_node_data
for node_data_file in self.node_data_files:
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_reader.py”, line 51, in
return (NodeDataFile(fname) for fname in self.filenames)
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_file.py”, line 18, in init
self.validate()
File “/home/bioinfo1/miniconda3/envs/nextstrain/lib/python3.8/site-packages/augur/util_support/node_data_file.py”, line 68, in validate
raise RuntimeError(
RuntimeError: nodes value in results/_tip_frequencies.json is not a dictionary. Please check the formatting of this JSON!

As you can see, all my outputs are called from results/

On the other hand, if I run the same augur export v2 command, but without the argument results/tip_frequencies.json , it works flawlessly… of course, without integrating the frequencies to the final .json file. What I’m trying to say is that the aa_muts.json gets properly validated along everything else.

Thank you in advance for your kind answers, I’m really in the need for advice here.

Best regards!

2

Hi @vicss, welcome! Glad to hear Nextstrain’s tools are helpful for
you!

It sounds like your results/aa_muts.json node data file is fine, but results/tip_frequencies.json is malformed. In order to debug, could you share two things with us?

  1. The tip_frequencies.json file so we can inspect its contents.
  2. The exact command line you’re using to generate the tip_frequencies.json file.

These will help narrow down the issue. The version of Augur you’re running (augur --version) might also be helpful.

Thanks!

3

Hi @trs, thanks a lot for your reply!

Regarding the augur version, is augur 12.0.0.

About the command line, for the tip_frequencies.json:
$ augur frequencies --method kde --metadata data/metadata-g.tsv --tree results/tree.nwk --output results/tip_frequencies.json
for the file tip_frequencies.json As you can see, its the basic (but might be wrong as you said).

I calculated as well a slightly more complex _tip_frequencies.json with:
$ augur frequencies --method kde --metadata data/metadata-g.tsv --tree results/tree.nwk --min-date 2020-11-12 --max-date 2021-07-23 --pivot-interval-units weeks --include-internal-nodes --output results/_tip_frequencies.json into the file: _tip_frequencies.json

Both “versions” yield the exact same aforementioned error.

Thank you for your kind attention and help!

PD, did a correction (sept,10) at the tip_frequencies command, since its metadata-g.tsv the correct file and no metadata.tsv as stated initially. The respective tip_frequencies.json was corrected and uploaded as well.

Best regards

4

Ah! I’m sorry @vicss, I think I’ve accidentally mislead you here! I wasn’t thinking clearly about what the tip_frequencies.json file actually was when thinking about your issue. Once I saw what you uploaded and the command-line you used, it clicked and I realized the problem. Thank you for collecting the information!

Unlike Augur node data files, which get passed into augur export v2, the tip-frequencies file is an Auspice sidecar file, which doesn’t get incorporated into the final main Auspice dataset file. (See our new summary of data formats.)

Instead, the tip-frequencies file gets uploaded alongside the main dataset file (hence “sidecar”) with a common base filename, e.g.:

auspice/21-hs-inv20-ago23-g.json
auspice/21-hs-inv20-ago23-g_tip-frequencies.json

and Auspice knows to look for it when your Auspice config (config/auspice_config5-g.json) includes "frequencies" in the panels section (example) or --panels is passed to augur export v2 (doc).

Let me know if this helps, or if you need some more guidance on putting it all together. Unfortunately, I don’t think we yet have good docs on setting up frequencies.

1 Like
5

Dear @trs,

Indeed, I misunderstood the instructions of your documentation, particularly about the sidecar file aspect. With your explanation I solved my issue.

Just one more question, if you allow me. Will the Nextstrain team offer a workshop (online or presential), or seminaries in the near future? I would like to train and improve my skills on this quintessential suite y’all developed.

I’m very grateful for your kind answers and your time.

Regards from Mexico!

2 Likes
6

Very glad to hear you got it working!

The Nextstrain team would like to run more workshops in general, but I’m not sure when/where the next one will be. It’s definitely our goal to produce more training material and help more folks use what we’ve developed! Let us know if there are specific topics you’re interesting in learning more about from a workshop or other training material.

7

Dear @trs.

Thank you again for your answer.

I’m very interested in the long-term forecasts of seasonal influenza A/H3N2 evolution. It’s been a little complicated for me implement it in my Avian/H5N2 vector.

I hope you can offer a workshop (presential or online).

Thanks again for your attentions

All best

8

Dear @vicss

I had exactly the same error as you (frequency.json is not a dictionary).
Can you share your experience of how you solved this error?

Thanks
Yang