diff --git a/README.md b/README.md
index 429731f1aacb01ace93a5c5ef28af2158f9abfb3..9580ab772b055e5b9aa0416bd9f5302395d1493d 100644
--- a/README.md
+++ b/README.md
@@ -71,6 +71,8 @@ This project is funded by the German Ministry for Science and Education (BMBF 03
The sections that follow outlines the key steps taken to explore and analyze the relationships described in the manuscript *From Habitat to Genotype: The Complex Interplay of Climate, Phenotypes, and Taxonomy in Teosinte*. Each step is detailed to ensure reproducibility and transparency in the analysis.
+*Caution: Please note that some of the directory paths in the scripts may need to be modified to match your local environment in order to ensure proper reproducibility.*
+
---
## Phenotype and Climate Data Analysis
@@ -150,8 +152,8 @@ The scripts and files for preprocessing data for GWAS are located in the directo
1. **Conversion of Genetic Data**
Raw genetic data in VCF format was converted to hapmap format using TASSEL software [doi:10.1093/bioinformatics/btm308]. Files were loaded into TASSEL and saved as diploid hapmap files. The source files can be found in the following directories:
- - `/initial_data/data/genetic/filtered/*.vcf`
- - `/initial_data/data/meta/ADN_pasap_3604.txt`
+ - `studies/initial_data/resources/data/genetic/filtered/*.vcf`
+ - `studies/initial_data/resources/data/meta/ADN_pasap_3604.txt`
2. **Hapmap File Cleaning**
Species-specific tags were removed from teosinte hapmap file IDs using `remove_spp_tags_in_hapmap_files.sh`, standardizing identifiers across files.
@@ -167,14 +169,14 @@ The scripts and files for preprocessing data for GWAS are located in the directo
Processed genotype and phenotype files are stored in:
- - `/mnt/data/joseph/TEOSINTE/analyses/GWAS/data/`
- - `studies/processed_genotype_phenotype_teosinte_data`
+ - `studies/processed_genotype_phenotype_teosinte_data/resources/gwas_data_3455_accessions/genotype*`
+ - `studies/processed_genotype_phenotype_teosinte_data/resources/gwas_data_3455_accessions/phenotype`
### GWAS Using GAPIT
Scripts and files for GWAS analysis are available in [workflows/gwas_gapit_pipeline](workflows/gwas_gapit_pipeline).
-The GWAS was performed using GAPIT, evaluating models such as GLM, MLM, FarmCPU, MLMM, CMLM, SUPER, and BLINK. FarmCPU and MLMM were identified as the best-performing models based on QQ plots and BIC criteria. Input files included genotype and phenotype data from `/mnt/data/joseph/TEOSINTE/analyses/GWAS/data/`.
+The GWAS was performed using GAPIT, evaluating models such as GLM, MLM, FarmCPU, MLMM, CMLM, SUPER, and BLINK. FarmCPU and MLMM were identified as the best-performing models based on QQ plots and BIC criteria. Input files included genotype and phenotype data from `studies/processed_genotype_phenotype_teosinte_data/resources/gwas_data_3455_accessions/genotype*` and `studies/processed_genotype_phenotype_teosinte_data/resources/gwas_data_3455_accessions/phenotype`.
Principal components (PCs) for population structure correction were selected based on eigenvalues calculated via PCA, using BIC for optimization. The main script for GWAS runs was `gwas_3455.R`, and various taxa-specific scripts include:
@@ -198,7 +200,7 @@ Principal components (PCs) for population structure correction were selected bas
**Objective:** Process and annotate GWAS result files from GAPIT.
The analysis scripts and files are available in [workflows/snp_gene_neighborhood_pipeline/scripts](workflows/snp_gene_neighborhood_pipeline/).
-Significant SNPs were annotated using the B73 reference genome. This process involved downloading the B73 reference sequence and its annotations, running Mercator and ProtScriber, and generating annotations for each SNP. The annotation pipeline utilizes `snp_physical_mapping_pl_v2.py` for mapping SNPs to nearby features within a +50kb window and `generate_reference_protein_function_annotations_v2.py` for annotating the SNPs with functional information from Mercator and InterProScan. The annotated results for each trait are stored in `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait/`. Additionally, heatmaps were generated to summarize the results, with scripts available in the directory `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/mercator_results_summarizer/`. These scripts provide both raw data and Z-transformed data.
+Significant SNPs were annotated using the B73 reference genome. This process involved downloading the B73 reference sequence and its annotations, running Mercator and ProtScriber, and generating annotations for each SNP. The annotation pipeline utilizes `snp_physical_mapping_pl_v2.py` for mapping SNPs to nearby features within a +50kb window and `generate_reference_protein_function_annotations_v2.py` for annotating the SNPs with functional information from Mercator and InterProScan. The annotated results for each trait are stored in `workflows/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait/`. Additionally, heatmaps were generated to summarize the results, with scripts available in the directory `workflows/snp_gene_neighborhood_pipeline/scripts/mercator_results_summarizer/`. These scripts provide both raw data and Z-transformed data.
#### Prerequisites
@@ -228,28 +230,28 @@ bash grep "^>" ../misc/b73/protein_seq_mercator_prot_swissprot_annot/b73_referen
Data Preparation: Ensure the GWAS results and reference files are ready for annotation.
-Annotation Script: Run `bash /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/annotate.sh` to annotate significant SNPs in your Teosinte GWAS results using the following components:
+Annotation Script: Run `bash workflows/snp_gene_neighborhood_pipeline/scripts/annotate.sh` to annotate significant SNPs in your Teosinte GWAS results using the following components:
`python3 snp_physical_mapping_pl_v2.py`: Maps SNPs to nearby features from the GAPIT.Filter_GWAS_result.csv.
`python3 generate_reference_protein_function_annotations_v2.py`: Annotates SNPs with Mercator, InterProScan, and ProtScriber information.
Fast Processing Option: Use `annotate_v2.sh` for faster annotation, which runs`process_trait.sh`. This script includes refactored versions of the above scripts.
-Results: Each trait’s annotated SNP features was stored in `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait`.
+Results: Each trait’s annotated SNP features was stored in `workflows/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait`.
-Final SNP Annotation Table: The final annotated SNP table, combining information from Mercator, InterProScan, and ProtScriber, was stored in the `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait` directory with filenames like `${trait}_annotated_snp_table_${model}.csv`.
+Final SNP Annotation Table: The final annotated SNP table, combining information from Mercator, InterProScan, and ProtScriber, was stored in the `workflows/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait` directory with filenames like `${trait}_annotated_snp_table_${model}.csv`.
-Combined Annotations: The combined annotations for each trait and model was stored in the directory `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait` directory with filenames like `${taxon}_final_annotation_${trait}_${model}.csv`.
+Combined Annotations: The combined annotations for each trait and model was stored in the directory `workflows/snp_gene_neighborhood_pipeline/results/taxon/all_features/trait` directory with filenames like `${taxon}_final_annotation_${trait}_${model}.csv`.
#### Step 5: Cleanup and Merging
Remove empty files generated from models with no significant SNPs:
Clean the repository by removing 0-byte files created from models with no significant SNPs:
-`find /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results -size 0c -delete`
+`find workflows/snp_gene_neighborhood_pipeline/results -size 0c -delete`
Merge all annotation files into a mega CSV file containing all annotations for the final analysis
-`/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/merge_all_annotatated_files.py`
+`workflows/snp_gene_neighborhood_pipeline/scripts/merge_all_annotatated_files.py`
---
@@ -257,9 +259,9 @@ Merge all annotation files into a mega CSV file containing all annotations for t
Scripts for heatmap generation and summary are available in [mercator_results_summarizer](workflows/snp_gene_neighborhood_pipeline/scripts/mercator_results_summarizer).
-- `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/mercator_summary.py`: Summarizes Mercator annotations into two count tables—one with raw data and another with Z-transformed data. These tables help analyze SNP functional associations with genes and biological pathways.
+- `workflows/snp_gene_neighborhood_pipeline/scripts/mercator_results_summarizer/mercator_summary.py`: Summarizes Mercator annotations into two count tables—one with raw data and another with Z-transformed data. These tables help analyze SNP functional associations with genes and biological pathways.
-- `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/plot_heatmap.R`: Generates heatmaps of Mercator annotations using the output of mercator_summary.py. Hardcoded path to phenotype data: `/mnt/data/joseph/TEOSINTE/analyses-on-pheotype_and_climatic_data/downstream-data-and-outputs/data/phenotype_and_env_data_vars_labeled.csv`
+- `workflows/snp_gene_neighborhood_pipeline/scripts/plot_heatmap.R`: Generates heatmaps of Mercator annotations using the output of `mercator_summary.py`. Hardcoded path to phenotype data: `workflows/phenotype_pca_and_hc/phenotype_and_env_data_vars_labeled.csv`
```bash
# parviglumis
@@ -288,9 +290,9 @@ python3 mercator_summary.py -p parameter_diplop.csv -d 2
Rscript plot_heatmap.R diplo_perennis_bin_depth_2_vs_traits_df.csv diplo_perennis_depth_2
```
-- `/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/unique_bins.R`: Preprocesses data to identify unique annotations for each taxon-trait combination and saves them in uniq_mercator_annotations.csv
+- `workflows/snp_gene_neighborhood_pipeline/scripts/unique_bins.R`: Preprocesses data to identify unique annotations for each taxon-trait combination and saves them in uniq_mercator_annotations.csv
--`/mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/src/unique_bins.R/plot_counts_snps_n_protein_coding_genes.py`: Summarizes the number of SNPs and protein-coding genes found in LD windows and generates horizontal bar plots.
+-`workflows/snp_gene_neighborhood_pipeline/scripts/unique_bins.R/plot_counts_snps_n_protein_coding_genes.py`: Summarizes the number of SNPs and protein-coding genes found in LD windows and generates horizontal bar plots.
---
@@ -357,7 +359,12 @@ Output tables stored in `../../results_mercator_heatmaps_enrichment/enrichment/i
##### main result output ora tables and plots
-Utilize R scripts `mercator_enrichment_v2.R` to analyze the generated CSV files and produce results for each clade.
+The script results are located in:
+
+- `workflows/snp_gene_neighborhood_pipeline/scripts/mercator_enrichment` (scripts)
+- `workflows/snp_gene_neighborhood_pipeline/results_mercator_heatmaps_enrichment` (results)
+
+Used R scripts `mercator_enrichment_v2.R` to analyze the generated CSV files and produce results for each clade.
```sh
Rscript mercator_enrichment_v2.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_1_parviglumis_clade_combined_enrich_input.csv parviglumis_d1 1
@@ -396,37 +403,11 @@ bash ./merge_tbls_plot_ora.sh
The script `grouped_enrichment_bar_plots.py` creates bar plots to visualize protein function counts based on the analysis output.
```sh
-
python3 grouped_enrichment_bar_plots.py mexicana ../../results_mercator_heatmaps_enrichment/enrichment/ora_mapman_result_tables/mexicana/mexicana_d2_ora_mapman_result_tbl/mexicana_merged_ORA_output_tbl.csv --output_dir=../../results_all_factors_snps_n_annotations/
python3 grouped_enrichment_bar_plots.py parviglumis ../../results_mercator_heatmaps_enrichment/enrichment/ora_mapman_result_tables/parviglumis/parviglumis_d2_ora_mapman_result_tbl/parviglumis_merged_ORA_output_tbl.csv --output_dir=../../results_all_factors_snps_n_annotations/
```
-##### Main result output Under Representation Analysus (URA) tables and plots
-
-```sh
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_1_parviglumis_clade_combined_enrich_input.csv parviglumis_d1 1
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_2_parviglumis_clade_combined_enrich_input.csv parviglumis_d2 2
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_1_mexicana_clade_combined_enrich_input.csv mexicana_d1 1
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_2_mexicana_clade_combined_enrich_input.csv mexicana_d2 2
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_1_mexicana_mesa_central_clade_combined_enrich_input.csv mexicana_mesa_central_d1 1
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_2_mexicana_mesa_central_clade_combined_enrich_input.csv mexicana_mesa_central_d2 2
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_1_mexicana_chalco_clade_combined_enrich_input.csv mexicana_chalco_d1 1
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_2_mexicana_chalco_clade_combined_enrich_input.csv mexicana_chalco_d2 2
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_1_diplo_perennis_clade_combined_enrich_input.csv diplo_perennis_d1 1
-
-Rscript mercator_enrichment_ura.R ../../results_mercator_heatmaps_enrichment/enrichment/input_tables/depth_2_diplo_perennis_clade_combined_enrich_input.csv diplo_perennis_d2 2
-
-```
-
#### Jaccard Similarity index
This was calculated and plotted following the steps outlined in the Jupyter notebook located at: `workflows/snp_gene_neighborhood_pipeline/scripts/snps_and_proteins_jaccard_index.ipynb`
@@ -474,7 +455,7 @@ The analysis scripts and files mentioned above are available in the directory [w
The results of the GWAS conducted on teosinte were systematically compared with established SNP datasets, including the GWAS SNPs for 21 traits from the Li et al. (2022) study and those from the GWAS Atlas database. Comparison tables were generated and stored alongside the annotated SNP results. The following commands were utilized to perform the comparisons:
```bash
-python3 literature_snps_comparisons.py -t /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results_all_factors_snps_n_annotations/mexicana -a /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_from_GWAS_Atlas_database_merged.gff3 -l /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_for_21_traits_Li_2022_Wang_Labs_merged.gff3
+python3 literature_snps_comparisons.py -t workflows/snp_gene_neighborhood_pipeline/results_all_factors_snps_n_annotations/mexicana -a workflows/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_from_GWAS_Atlas_database_merged.gff3 -l workflows/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_for_21_traits_Li_2022_Wang_Labs_merged.gff3
-python3 literature_snps_comparisons.py -t /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/results_all_factors_snps_n_annotations/parviglumis -a /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_from_GWAS_Atlas_database_merged.gff3 -l /mnt/data/joseph/TEOSINTE/analyses/GWAS/pipelines/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_for_21_traits_Li_2022_Wang_Labs_merged.gff3
+python3 literature_snps_comparisons.py -t workflows/snp_gene_neighborhood_pipeline/results_all_factors_snps_n_annotations/parviglumis -a workflows/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_from_GWAS_Atlas_database_merged.gff3 -l workflows/snp_gene_neighborhood_pipeline/gwas_snps_vs_published_snps/data/processed_data/GWAS_SNPs_for_21_traits_Li_2022_Wang_Labs_merged.gff3
```