This SOP describes standardized procedures for entering, organizing, validating, and managing research data in the EA Program. It covers the full data lifecycle from initial recording through archival storage, ensuring that all datasets are consistent, traceable, and reproducible (Broman & Woo, 2018; Wickham, 2014).
2 Scope
Applies to: All tabular data generated in the EA Program laboratory and field campaigns, including instrument outputs, field observations, and manually recorded measurements
Does not apply to: Spatial/GIS data (see SOP-L804), bioinformatics sequence data (see SOP-L705), or instrument-specific export procedures (see SOP-L807)
Location: EA Program laboratory (SGM 133) and RStudio Server
3 Responsibilities
Role
Responsibility
Lab PI
Approve data management plan; ensure compliance with funder requirements
Enter data following this SOP; run validation checks; document metadata
Data Reviewer
Verify data entries against lab notebooks before analysis
4 Definitions
Data dictionary: A companion metadata file that describes each variable in a dataset, including the column name, description, units, data type, expected range, and source SOP; required for every EA Program dataset.
ISO 8601: An international standard for representing dates and times as YYYY-MM-DD; the required date format for all EA Program data files because it sorts chronologically and is unambiguous across locales.
Missing value: A data cell for which no measurement was made or the measurement was invalid; represented as blank or NA in EA Program datasets; never coded as 0 (which is a valid measurement).
Primary key: A unique identifier for each row in a data table (e.g., sample_id); enables unambiguous linking between related tables and detection of duplicate entries.
Tidy data: A data structure in which each variable has its own column, each observation has its own row, and each type of observational unit forms its own table (Wickham, 2014); the standard format for EA Program analytical datasets.
Version control: A system that records changes to files over time so that specific versions can be recalled; GitHub/Git is the EA Program version control platform for all data, scripts, and SOPs.
5 Safety and Hazards
Data entry is a computational activity with no physical hazards. Ergonomic best practices (regular breaks, proper screen height) are encouraged for extended data entry sessions.
6 Equipment and Materials
6.1 Software
Item
Access
Notes
RStudio Server
rstudio.pomona.edu
Primary data processing tool
R (version 4.x)
Pre-installed on server
Required for validation scripts
Git / GitHub
github.com/marclos
Version control for all data files
Microsoft Excel / Google Sheets
Campus license
Acceptable for initial entry only
Text editor (VS Code, nano)
Various
For editing CSV files directly
6.2 File Storage
Location
Purpose
Backup Frequency
GitHub repository
Active project data
Every commit
College network drive
Archival storage
Semester end
External hard drive (lab)
Physical backup
Semester end
7 Procedure
7.1 File Naming Conventions
All data files must follow this naming pattern:
[project]_[datatype]_[YYYY-MM-DD].[ext]
For example: claremont_soilpH_2026-05-21.csv
Rules for file names:
Use lowercase letters only
Separate words with underscores (not spaces or hyphens)
Include the date of collection or generation in ISO 8601 format (YYYY-MM-DD)
Use .csv as the default file format for tabular data
TipTip
Never use spaces, special characters, or accented letters in file names. These cause problems across operating systems and with R’s file-reading functions.
7.2 Directory Structure
Organize project data using a consistent directory tree:
project_name/
├── data/
│ ├── raw/ # Unmodified instrument or field data
│ ├── processed/ # Cleaned, validated datasets
│ └── metadata/ # Data dictionaries, codebooks
├── analysis/
│ ├── scripts/ # R scripts and Quarto documents
│ └── output/ # Figures, tables, reports
└── README.md # Project overview and data description
Raw data files in data/raw/ must never be modified. All cleaning and transformation steps should produce new files in data/processed/ with scripts that document every change (Wilson et al., 2017).
7.3 Data Entry from Lab Notebooks
Open the lab notebook to the relevant page
Create a new CSV file (or open the project template) in data/raw/
Enter column headers in the first row following the variable naming conventions below
Enter each observation as a separate row
Record the notebook page number, date, and analyst initials in the metadata file
Save and commit to Git with a descriptive message
7.4 Variable Naming Conventions
Column headers (variable names) must be:
Lowercase, with underscores separating words (e.g., soil_ph, no3_mg_l)
Free of spaces, special characters, and units in the name itself
Documented in an accompanying data dictionary (see Section 6.7)
Use consistent naming across related datasets. For example, always use site_id (not Site, site, SiteID, or location) when referring to the same sampling locations.
7.5 Data Types and Formatting
Data Type
Convention
Example
Dates
ISO 8601 (YYYY-MM-DD)
2026-05-21
Times
24-hour (HH:MM)
14:30
Missing values
Leave cell empty or use NA
NA
Numeric
No commas; use period for decimal
1234.56
Categorical
Consistent spelling; lowercase
upstream
Below detection
Flag column; numeric = MDL/2
See QC SOP
WarningCommon Mistakes
Never use 0 to represent a missing value. A zero is a valid measurement. Use NA or leave the cell blank. Never merge cells in a spreadsheet, and never use color coding as the sole carrier of information (Broman & Woo, 2018).
7.6 Data Validation in R
After entry, run the validation script to check for common errors. A minimal validation template:
Code
library(tidyverse)# Read the raw datadat <-read_csv("data/raw/project_variable_2026-05-21.csv")# --- Structure checks ---glimpse(dat) # Column types correct?stopifnot(nrow(dat) >0) # File is not empty# --- Missing data summary ---dat %>%summarize(across(everything(), ~sum(is.na(.)))) %>%pivot_longer(everything(), names_to ="variable",values_to ="n_missing") %>%filter(n_missing >0)# --- Range checks ---dat %>%summarize(ph_min =min(soil_ph, na.rm =TRUE),ph_max =max(soil_ph, na.rm =TRUE) )# Are values within physically plausible ranges?# --- Duplicate check ---dat %>%group_by(site_id, date) %>%filter(n() >1)# Flag unexpected duplicates for review# --- Categorical consistency ---dat %>%count(site_id, sort =TRUE)# Look for typos (e.g., "upstrream" vs "upstream")
7.7 Data Dictionaries (Metadata)
Every dataset must have a companion data dictionary saved as data/metadata/[project]_datadictionary.csv with the following columns:
Column
Description
variable
Column name in the data file
description
Plain-language description
units
Units of measurement (or “categorical”, “text”)
type
Data type: numeric, integer, character, date, logical
range
Expected range or valid categories
source_sop
SOP number for the method that generated the data
7.8 Version Control with Git
All data files and scripts must be tracked in a Git repository hosted on GitHub. Follow the procedures in SOP-L801 for committing and pushing changes. Key rules:
Commit raw data files when they are first created (and never modify them after the initial commit)
Commit processed data files after each cleaning or transformation step
Write informative commit messages (e.g., “Add soil pH data from Apr 15 sampling” rather than “update”)
Use branches for experimental analyses; merge to main when finalized
7.9 Database Management for Multi-Year Projects
For projects spanning multiple semesters or involving multiple analysts, maintain a relational data structure:
Create a sites table (data/metadata/sites.csv) with location identifiers, coordinates, and site descriptions
Create a samples table (data/metadata/samples.csv) linking each sample to a site, date, and collector
Link analytical results to samples using unique sample_id values
This structure prevents duplication, supports merging across semesters, and enables relational joins in R (Wickham, 2014):
Code
sites <-read_csv("data/metadata/sites.csv")samples <-read_csv("data/metadata/samples.csv")results <-read_csv("data/processed/chemistry_results.csv")# Join all tablesfull_data <- results %>%left_join(samples, by ="sample_id") %>%left_join(sites, by ="site_id")
8 Quality Control and Acceptance Criteria
All data entries must be verified against the original lab notebook by a second person (or by the analyst after a 24-hour waiting period)
Validation scripts must pass with zero unexpected errors before data are used in analysis
The data dictionary must be complete for every variable in the dataset
Refer to SOP-L802 for analytical QC requirements that apply to the underlying measurements