view release on metacpan or  search on metacpan

lib/Bio/MUST/Apps/FortyTwo/Manual.pod  view on Meta::CPAN

# PODNAME: FortyTwo::Manual
# ABSTRACT: User Guide for Forty-Two
# CONTRIBUTOR: Mick VAN VLIERBERGHE <mvanvlierberghe@doct.uliege.be>

# perl -M'List::AllUtils qw(uniq)' -nle 'push @found, m/(C<.*?>)/g; END{ print join "\n", uniq @found }' Manual.pod

# Feedback Forty-two (Nicolas MAGAIN)
# 
# [DONE] 1. Lorsque je copie/colle les commandes depuis le manuel de Forty-two vers le terminal ou un éditeur de texte, je dois remplacer manuellement tous les tirets. Les tirets du manuel apparaissent plus courts et font bugger les commandes.
# 
# [DONE] 2. Le manuel s'arrête à la génération du fichier yaml mais il serait utile de faire un paragraphe sur le lancement du script forty-two.pl en lui même avec --config, --verbosity etc. On l'obtient en faisant forty-two.pl --help mais un mo...
# 
# [DONE] 3. Dans cette partie du manuel:
# query_orgs should be listed in a file (queries.txt) and spelled exactly as in your MSAs. This file will be processed by yaml‐generator‐42.pl to populate the config file. To easily draft a list of query_orgs, you can for example use the 10 to 20...
# $grep‐h\>*.fasta|cut‐f1‐d'@'|sort|uniq‐c|sort‐rn|head‐n10
# 22498 >Danio_rerio
# 21071 >Homo_sapiens
# 20722 >Mus_musculus
# 18933 >Monodelphis_domestica
# 18616 >Loxodonta_africana
# 17762 >Latimeria_chalumnae
# 17678 >Canis_familiaris
# 17114 >Xenopus_tropicalis
# 16665 >Anolis_carolinensis
# 16611 >Sarcophilus_harrisii
# 
# Dans cette partie, il n'est pas clair si le > doit faire partie du nom de l'organisme ou pas, puisqu'il apparait dans l'exemple ci-dessus. Une personne non-avertie serait tentée de recopier les noms tels quels dans son fichier queries./txt
# 
# Comme c'est sans le '>' vous pourriez préciser, et ensuite par exemple rajouter la commande ci-dessous pour directement créer queries.txt
# grep -h \>*.fasta | cut -f1 -d'@' | sort | uniq -c | sort -rn | head -n 10 | cut -f2 -d'>' > queries.txt
# 
# Si vous faites des lignes de code toutes faites comme ça, vous simplifierez la vie des débutants.
# 
# [DONE] 4a. Je trouve que la partie sur tax_filter n'est pas claire (sûrement destinée à des gens qui maîtrisent déjà ces concepts?). Je ne comprends pas bien à quoi cela sert.
# 
# [DONE] 4b. "but the generator only supports the plain tax_filter syntax shown in the first example." -> je trouve que cela n'est pas clair. Est-ce seulement la première des 4 lignes (+Poaceae) qui fonctionne par rapport aux trois autres examples, ...
# 
# [DONE] 4c. Un utilisateur qui n'a pas besoin de tax_filter ou qui ne comprend pas ce que c'est se demande, après lecture de ce paragraphe, ce qu'il doit faire concernant cette partie. Est-ce qu'il doit préparer quelque chose quand même, ou pourr...
# 
# [DONE] 5. Je n'ai pas trouvé d'explications dans le manuel (ou je n'ai pas compris que les explications portaient sur cela) au sujet de ces 4 paramètres dans le wizzard
# Set ref_brh_mode
# Set reference banks suffix
# Set trim_max_shift
# Set candiate banks suffix
# En fait, il y a plus d'explications dans le fichier yaml lui-même, mais puisqu'on ne l'a pas encore quand on lance le premier wizzard, il faudrait que les explications arrivent plus tôt, soit dans le manuel soit dans le wizzard
# 
# [DONE] 6. Si je prépare un fichier config localement pour ensuite l'uploader sur un cluster, si je veux indiquer le path vers le dossier qui contient les génomes de référence, ou vers les génomes à miner sur le cluster, je vais me retrouver a...

__END__

=pod

=head1 NAME

FortyTwo::Manual - User Guide for Forty-Two

=head1 VERSION

version 0.213470

=encoding UTF-8

=head1 Background

=head2 Aim and features

C<42> is a phylogenomic tool designed to add (and optionally align) sequences to
a preexisting multiple sequence alignment (MSA) while controlling for orthology
relationships and potentially contaminating sequences. Sequences to add are
either nucleotide transcripts resulting from transcriptome assembly or already
translated protein sequences. In theory, one can also use genomic nucleotide
sequences (because C<42> can splice introns), but this possibility has not been
extensively tested so far.

=for todo TODO: amend these paragraphs after publication...

The working hypothesis of C<42> is that its orthology-controlling heuristics can
enrich not only MSAs of single-copy genes but also more complicated MSAs
including terminally duplicated genes (in-paralogues) and/or corresponding to
multigenic families featuring different out-paralogues of different ages.
Preliminary tests on a broadly sampled eukaryotic data set suggest that the
orthology relationships enforced by C<42> are in good agreement with those
inferred with C<OrthoFinder> software [Emms and Kelly (2015) I<Genome Biol>
16:157]. To this end, it relies on complete proteomes of reference organisms.

C<42> is also able to enrich MSAs resulting from the split of complex multigenic
families after phylogenetic analysis. For this, it requires decoy files composed
of representative sequences of unwanted out-paralogues. Such C<PARA> files have
to be provided by the user.

Regarding contamination, C<42> implements a dual system of taxonomic filters
(based on I<NCBI Taxonomy>) allowing it to flag any new sequence for which the
taxonomic affiliation is doubtful. Two main approaches are available: C<42>
either checks that a new sequence is most similar to (an)other sequence(s) of
the expected taxon already present in the MSA (= positive filter) or that a new
sequence is more similar to a sequence in the MSA than to any sequence from a
set of complete proteomes that do not include the expected taxon (= TOL check
decoy). While the power of the first mechanism is dependent on the taxonomic
breadth of each MSA, the second approach is more widely applicable.

As species-rich ribosomal protein MSAs are available for both prokaryotic and
eukaryotic genomes, C<42> can also be used to probe the contamination level of
any genome or transcriptome of interest using the first approach. A special mode
is provided for this application, termed the C<metagenomic> mode.

=head2 Design principles

lib/Bio/MUST/Apps/FortyTwo/Manual.pod  view on Meta::CPAN

    [ +Amphibia, +Amniota ]             # any amphibian or amniote
    [ +Bacteria, -Cyanobacteria ]       # any non-cyanobacterial bacterium

In principle, C<42> can also use such filters, but the generator currently
supports only the plain C<tax_filter> syntax shown in the first example
(C<+Poaceae>). Yet, it can assist you in finding the adequate taxa to specify
based on the organisms you add.

=for todo clarify the part about choose_tax_filter

For this to work, you must define at which taxonomic level(s) you want to
set the filter. When specifying several levels (C<--levels> option), the script
will try to check for the next level in case one is missing. You can put as many
levels as you want separated by a comma (C<,>, no whitespace character) when
using the C<--wizard> option and by a whitespace character (C< >) as a command
line argument. Another possibility is to choose manually from the NCBI lineage
for those that fail (in this case use the C<--choose_tax_filter=1> argument). If
you want to select manually for each candidate organisme set the argument
C<--choose_tax_filter=2>.

Alternatively, you can define a custom taxonomic filter for each organism by
adding a third column to the C<bank_mapper.idm> file.

=head2 Running C<42>

=head3 Assisted configuration using the wizard

Now that you are done preparing files, let's run the wizard!

    $ yaml-generator-42.pl --wizard

Using the C<--wizard> option enables an interactive mode where you will be asked
to enter each parameter in the terminal.

B<Note:> Pressing the C<ENTER> key selects the default value encoded in C<42>.

Two C<run_mode> are available C<metagenomic> or C<phylogenomic>. The
C<phylogenomic> mode is designed to enrich MSAs with orthologues for subsequent
phylogenomic analysis. In contrast, the C<metagenomic> mode is designed to
estimate the contamination level of transcriptomic data using reference
ribosomal protein MSAs. The latter mode does not modify the MSAs but instead
produces one taxonomic report per MSA listing the lineage of each identified
orthologous sequence. When not specified, C<run_mode> internally defaults to
C<phylogenomic>.

=for todo TODO: explain how to debrief metagenomic mode for evaluating transcriptomes

B<Note:> the C<phylogenomic> mode also produces taxonomic reports but deprived
of taxonomic affiliations for the purpose of C<one-on-one.pl> (not currently
distributed on C<CPAN>).

The wizard does its best to assist you in building your C<config> file. In
particular, it scans the directories you specify for relevant files. Hence, to
identify the C<banks> and C<ref_banks> files, it looks for files ending with
a specific suffix. These suffices can be provided using the C<bank_suffix> and
C<ref_bank_suffix> options, respectively. If your banks are built from protein
sequences, use C<.psq>; otherwise, for nucleotide sequences, use C<.nsq>.

Because of this scanning behavior, it is better to prepare your files directly
on the computer on which you plan to run C<42>. If you try to prepare your
C<config> file locally (for subsequent upload on a remote computer), it is very
likely that the wizard complains about some directories not being found.

=head3 Command-line options

Since the configuration (C<config>) file specifies all the details, running
C<42> boils down to a simple command:

    $ forty-two.pl --config=config.yaml *.fasta

By default, C<42> is very terse. Yet it can be made quite verbose using the
corresponding C<--verbosity> option. If you need all the debugging information,
select level C<6>. In any case, it is useful to redirect the C<STDERR> stream to
a log file for post-run analysis.

    $ forty-two.pl --config=config.yaml --verbosity=3 *.fasta 2> 42.log

C<42> supports multithreading by allowing parallel enrichment of multiple MSAs.
This is controlled by the C<--threads> command line option. MSAs will be
arranged in an internal queue and processed in parallel using the specified
number of threads. As long as there remain more MSAs to enrich than that
number, C<42> will makes efficient use of the CPU cores. Obviously, there is
no speed gain in specifying more threads than MSAs to process.

    $ forty-two.pl --config=config.yaml --threads=20 *.fasta

Unfortunately, the current parallel implementation scheme leads to completely
scrambled log files. There is thus no point to ask for a high verbosity level.

=head3 Estimation of the contamination level in metagenomic mode

C<42> can be used as a stand-alone contamination detection tool to spot foreign
sequences and estimate the contamination level in transcriptomic or genomic data
as well as the taxonomic sources of contamination. To this end, it comes with
two sets of ribosomal protein MSAs: one set of 78 eukaryotic MSAs, manually
curated and continuously enriched with new species in H. Philippe's lab, and one
set of 90 prokaryotic MSAs, fetched from I<RiboDB>. Both sets are available at
L<https://bitbucket.org/phylogeno/42-ribo-msas/>.

For each transcriptome/genome, C<42> recovers the ribosomal protein orthologs
and then labels each one by computing the last common ancestor (LCA) of their
closest relatives (best BLAST hits) in the corresponding MSA (excluding
self-matches). The algorithm relies on the C<megan-like> mode described in
L<"Contamination detection and handling">. In this regard, since ribosomal
proteins are highly conserved, we suggest a more stringent parameterization of
the C<megan-like> algorithm, so as to avoid false positives during LCA
computation, with a C<--tax_score_mul> of C<0.99> instead of C<0.95> and a
C<--tax_min_ident> of C<50> instead of C<0>.

The follow up consists in running C<debrief-42.pl>, which parses the taxonomic
reports produced by C<42> in order to compare the taxonomic label (LCA) of each
ortholog computed by C<42> with the source organism lineage (according to I<NCBI
Taxonomy>) and classifies the sequences as contaminants if they differ at a
predefined taxonomic rank, based on a first user-defined list of taxa
(C<--seq_labeling>). After each ortholog has been classified, an estimated
contamination percentage is computed.

Additionally, contaminations are further classified to determine the main
sources of contaminants, based on a second user-defined list of taxa
(C<--contam_labeling>), which allows the user to fine control the output report.
In this regard, we distinguish two types of sequences, B<classified