Best Practices for Running a

Hyperfunctional Psychology
Laboratory

Greg J. Siegle, Ph.D.

University of Pittsburgh
School of Medicine

These slides available at

http://www.pitt.edu/~gsiegle/SiegleLaboratoryBestPracticesColloquium.pdf

Presented work supported by MH082998

 Why bother?
You and others can trust your data
Its easy to know when you step into a best-practices lab
Some researchers get a reputation as careful
Increase replicability
Decrease debacles
Example from my lab:
The chilling chiller incident
Example from the current fMRI world
 Stuff well discuss

Study setup
Data collection
Storing data
Analysis
Managing the lab
Study Setup
 Pre-emptive strike:
Clinical Operations Manual
INCLUDING template
documents
Common elements
Study Responsibility Log who
does what when
Study worksheet stuff which
has to happen and when, e.g.,
calibrations, audits
Assessment Schedule
Assessment Grid
Procedural Checklists
Regulatory Binder Template
From http://www.uth.tmc.edu/ctrc/studymanagement.html
 Regulatory Binders & Lab documents
Basic clinical trial model
2 folders per patient 1 for identifiable info, 1 for all study documents. + master list.
Excellent list of lab documents
http://www.uth.tmc.edu/ctrc/regulatory.html
Binders/Folders for
Protocol and amendments
Data
Subject Logs and Lists
Patient Data 1 per participant
Contact Logs and monitoring
Reporting
Corrospondence with outside organizations (e.g., FDA)
IRB Documents
Case Report Blank Forms
Adverse events
People
Investigator Information
Team Information
Lab information
Lab certifications, etc.
Equipment
Investigational product (e.g., drug) info
Meeting documents
Study meetings
Study reports
Publications
 Study management database
Stuff to include in addition to data
Subject information
Screening/Enrollment log
Visit Schedule Log
Tracking/Reporting information
Adverse Event Log
Protocol Deviation Log
Data Cleaning log
Accountability logs
Device calibrations and accountabilities

Note: SEPARATE database for Master subject log

 Quality management plan
what Data collection Data processing

will you
check,
how
will you
check
it?
 Quality assurance guidelines
What bad data looks like
Illustrations of procedures
 Create folders
Study folders: at least
data
pupil
heart
behav
.
analysis
matlab
spss
documents
publications
regulatory
software
Folder contents (from Dr. Nicole Prause)
Data
Raw, important processed stages, data processing scripts such as .m file
backup, compiled data, final data
The data folder should contain enough information to quickly reconstruct
important phases of data processing without storing too many large files on the
computer indefinitely.
Every data folder should include is a "notes.txt" file, where you note
abnormalities for particular subjects and files to enable quick reconstruction of
data sets. For example, if a person becomes ill and withdraws from the study, it
will be much easier to find this noted in a single file than to start searching to
understand why the last two test conditions are missing to make decisions
about data inclusion/exclusion.
Institutional Review Board Compliance
Submissions, revisions, letters of approval, up-to-date informed consent
Scripts
Electronic questionnaires, up-to-date DMDX scripts, backup of stimuli if size
reasonable
Publication
Poster presentations, papers being prepared, final drafts of accepted/published
papers
 Select protocols
carefully
Stay as close as possible to
industry standards when
possible (deviating as
necessary)
E.g., the Society for
Psychophysiological
Research has published
standards for EEG, ERP,
Startle, Heart rate, HRV,
EMG, disease transmission
http://www.sprweb.org/journa
l/index.cfm
Internet questionnaires:
Skitka
www.uvm.edu/~pdodds/files/
papers/others/2006/skitka200
6a.pdf
ASTM (standards body)
www.astm.org
Do an in-house ethics review
Data collection
 Procedural checklists & records
Every detail is golden: Have checklists
and how-to guides
Check the checklists
All records shall be prepared, dated, and
signed (full signature, hand written) by one
person and independently checked, dated, and
signed by a 2nd person (GMP (Good
Manufacturing practices) 211.186)
Electronic checklists?
Possible
Electronic records may be considered trustworthy
and reliable and be used in leiu of paper records
provided that the electronic records have proper
secuirty controls (21 CFR Part 11 Subpart A Sec
11.1)
Ensure authenticity & integrity of electronic
records such that the person responsible for the
electronic record cannot readily repudiate the
record as not genuine (21 CFR Part 11 Subpart B
Sec 11.10)
Ensure the system can discern invalid or altered
electronic records (21 CFR Part 11 Subpart B Sec
11.10 (a))
But I dont recommend it yet!
Trouble shooting guides

Guideline for ERP data by Cecile Ladouceur and Naho Ichikawa

 AFTER data collection
Data cleaning
Signing off
 Video more is better
Essential for clinical interviews to at
least get audio. Video is better.
Note: Need IRB Approval
 Task design
Validation
Check timing / event logging
w/ fMRI we test at the scanner 1x phantom + 1x pilot
before any protocol
Check single subjects
Write analysis scripts for single subjects
BEFORE your first real subject
Be a subject for your own protocols
Test everything completely BEFORE your
first pilot subject.
Test everything completely BEFORE your
first real subject.
 Psychophys lab setup
Dr. Nicole Prauses lab setup
Neat reproducable lab setups
Diagram in your Ops Manual
to show how to do stuff
exactly the same every time
As many procedural diagrams as
might be useful
http://www.span-lab.com/Assets/images/photos/EEGprep.JPG

Care about disease transmission

Bloodborne Pathogen control:
Gloves as much as possible
Dont abraid the skin more than you need to
Disposable electrodes when possible
Disinfect
CIDEX if you have ventillation
Control III + Suave shampoo if you dont
Wear a labcoat thats actually what theyre for
 Checking stuff works before data
collection
Protocols before your protocols
Check all communications between computers,
peripherals, and data collection devices
Make sure your stimuli show
Have this in your checklists
We have eprime routines to test
getting scanner trigger,
eye-tracker events
mouse/button pushes
Storing Data
 Data Security & Integrity
Whitebox standards:
Keep original data in unalterable form
Have 2nd copy for any necessary changes (e.g., remove a few
trials, concatenate runs)
Ensure the system can discern invalid or altered electronic
records (21 CFR Part 11 Subpart B Sec 11.10 (a))
Security
21 CFR part 11:
Double password protection
Standards
They exist for most things: http://www.astm.org/
IRB
E.g., consent forms separate from data
 Databases
Huge science - http://c2.com/cgi/wiki?DatabaseBestPractices
E.g.,
Have primary keys
Dont change schemas
Consistent long descriptive column names across tables
Try things first in a local database
Good rule of thumb: 20 columns per table more is weird design
Lab standards
Ids are in columns called id
All tables have id
21 CFR Part 11
Keep an audit history of date created and by who, and dates
changed/updated
 Backups
Ideally
Daily data backups
Weekly incremental computer backups
Monthly full backups
Keep a set of backups in a secure place outside
your lab
Analysis & Quality Control
 Document everything
Lab notebooks are essential
Documentation
Extreme: Open Lab Notebook
http://en.wikipedia.org/wiki/Open_
Notebook_Science
All work posted immediately to the
public eye
Good tool:
http://openwetware.org/wiki/Main_
Page
Commercial approaches
Big list at:
http://campusguides.lib.utah.edu/co
ntent.php?pid=126157&sid=21316
70
My approach: Powerpoints per
study
Gregs Journal template on the
PICAN server
\\oacres3\rcn\pican\docs\gjsjourna
l.pot
Sharepoint blog?
Database page for all changes
with name, date, change
description
Analyses should be reproducible
I like 1 matlab or SPSS file with
all commands that produce all
analyses for a given study.
 Reasons for using ELNs/
virtual workspaces
1. They are an efficient way of managing large projects, multiple
projects and multi-institution projects.
2. Provenance ensures that any accusation of fraud can easily be
addressed.
3. Addresses the problem of missing information due to turnover in
lab personnel (and students).
4. Can access research results from anywhere and therefore keep up
with the ongoing work in the lab while traveling.
5. These systems are already being used in industry, therefore are
studentsneed to be acquainted with them to be employable.
6. Meets requirements of granting agency mandates for data
managment plans.
7. Facilitates depositing data into data repositories for reuse and
repurposing.

http://campusguides.lib.utah.edu/content.php?pid=126157&sid=2131670
Example journal page
 Beyond Powerpoint
Lab Bench People layer

http://campusguides.lib.utah.edu/content.php?p
id=126157&sid=2131670
 Example commercial solution:
(Not endorsed just summarized)
From labarchives.com
Intuitive Electronic Lab
Notebook (ELN) organizes your
laboratory data
Preserve all your data securely,
including all versions of all files
Share information within your
laboratory
Keep abreast of developments in
your lab even when traveling
Collaborate with investigators by
sharing selected data from your
Electronic Laboratory Notebook
Publish selected data to specific
individuals or the public
Protect your intellectual property
Runs on all platforms, including
Windows, Mac, Linux, iPad and
Android devices Special classroom
version of our Electronic Lab
Notebook also available
Sample all-figures-in-paper script

%% associations of power change with change in other things within and between groups
ctrl=find((s.grp==1) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));
cct=find((s.grp==2) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));
tau=find((s.grp==3) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));
cct_tau=[cct; tau];
fprintf('----------------------------------\n');
fprintf('CCT r(power_on change, rumination change)\n');
st.r_powerOnChg_rsqchg_CCT=r(poweronchg(cct),s.rsqchg(cct),0,1,1.5,-999);
figure(7); clf;
regplot(rescaleoutliers(poweronchg(cct)),rescaleoutliers(s.rsqchg(cct)));
xlabel('Trial Frequency Power Post CCT - Pre CCT');
ylabel('Rumination (RSQ) Post CCT - Pre CCT');

figure(8); clf;
regplot(rescaleoutliers(poweronchg(tau)),rescaleoutliers(s.rsqchg(tau)));
xlabel('Trial Frequency Power Post TAU - Pre TAU');
ylabel('Rumination (RSQ) Post TAU - Pre TAU');

figure(9); clf;
focindfromcct_change=-9.94-151.94.*poweronchg+109.13.*poweroffchg;
regplot(rescaleoutliers(focindfromcct_change(cct)),rescaleoutliers(s.rsqchg(cct)));
xlabel('Unfocus Index Post CCT - Pre CCT');
ylabel('Rumination (RSQ) Post CCT - Pre CCT');
 Use best practices for
preprocessing data
Again with the Psychophysiology guidelines
http://www.sprweb.org/journal/index.cfm
Visual inspection of artifacts
When are artifacts ok to let in to data?
How much should we say were letting in?
Contingency planning
What if you change preprocessing midway through?
I think you should reprocess everything
What if you change preprocessing after-the-fact?
Depending on how serious, note it.
 Quality control
Diagnosis and Clinical
dispositions:
Case conferences
Reliability on
ANYTHING subjective
Double data entry
See your Research
Methods textbook
 Check your data early and often
Quality check psychophys data that day and
fMRI data within a week (while its on the
servers)
Single subject analyses
Group analyses with N=5
Maintaining a lab
 Calibrations
Regular monthly calibrations of all
instruments
Currently done for pupilometer
Other stuff?
MR center and BIRC have done calibrations,
e.g., stability checks regularly. We dont
request them. But we should for our own
documentation.
 Security
Double-locked file cabinets
Password protection for computers, files, etc.
Note: 21 CFR (Code of Federal Regulations) Part
11 - Food and Drug Administration (FDA)
guidelines on electronic records
has security standards for data
audits, system validations, audit trails, electronic
signatures, and documentation
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfC
FR/CFRSearch.cfm?CFRPart=11
 Audits

Every 6 months, all

data within that 6
months
Quality management
help at:
http://www.uthouston.edu/CT
RC/trial_conduct/quality-
management.htm
There are chart audit
tools
Regulatory file review
tools
Every year full audit
should be easy
 The human thing
Laboratory mentality is important. Attend to it.
Anecdotal evidence suggests happy inspired
labs are often more functional.
You will likely not be in touch with the
emotional health of the lab. Have someone
who is. Make their report to you on lab health
a regular thing.
 Hire for your weaknesses
Good labs often have people who are (not all
of these at once)
Detail oriented
Socially attuned
Tech savvy
Inspired
 Sources
21 CFR (Code of Federal Regulations) Part 11 - Food and
Drug Administration (FDA) guidelines on electronic records
has security standards for data
audits, system validations, audit trails, electronic signatures, and
documentation
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSe
arch.cfm?CFRPart=11
Esp. Subpart B electronic records
Good Manufacturing Practices (GMP)
ASTM (standards body)
www.astm.org
Robert L. Zimmerman Jr, 10 Best Practices for Good
Laboratories. Nov Dec, 2010, November/December,
Standardization News
Clinical Trials Resource Center
http://www.uthouston.edu/CTRC/trial_conduct/