You are on page 1of 26

HLT Timing Studies

HLT Tutorial

Edgar F. Carrera
B ecarrera@cern.ch

Universidad San Francisco de Quito

October 27, 2017

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 1 / 24


Timing Tutorial Outline

1 Introduction

2 Timing Measurement

3 Analysis of Results

4 Hands-on Exercise

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 2 / 24


Introduction

HLT Timing:

What is it? Why do we need to measure it?


Timing refers to the computing time it takes for each event to
be processed (on average) by the computer farm at P5
Large but limited resources: HLT uses ∼ 26k CPU cores
Avoid dead-time due to back-pressure in the DAQ chain
Available budget (constraint): ≈ 260 ms/event
All proponents of new (updated) triggers need to have the
timing under control (not only rates)
Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 3 / 24
Introduction

HLT Flow
Filters are cheap
Producers are expensive
Decide efficiently and swiftly

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 4 / 24


Introduction

HLT Recent Timing Performance

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 5 / 24


Introduction

Documentation and Instructions

Twiki Instructions
https://twiki.cern.ch/twiki/bin/view/CMS/TriggerStudiesTiming

Linked from the


TriggerStudies Twiki
Infrastructure
How to perform the
measurement
How to study the results
Some additional information
Comments and suggestions
welcome

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 6 / 24


Introduction

A Few Technical Jargon:

Processors information under file /proc/cpuinfo (in a Linux system) or with the lscpu
command.

Hyperthreading: usage of several logical processors per physical core if programs


are threaded. No-hyperthreading means using just one logical processor per physical
core
TurboBoost: dynamic overclocking. Raise processor’s operating frequency.
Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 7 / 24
Introduction

Available Machines for Timing Measurements:


Offline Machines
Two benchmarked machines available: vocms003 and vocms004
Setting the right architechture may be required:
ssh vocms003
export SCRAM_ARCH=slc7_amd64_gcc630 # (for bash) or
setenv SCRAM_ARCH slc7_amd64_gcc630 # (for tcsh)

Access restricted but can subscribe to cms-tsg-operations e-group

User space under /data/user/$USER


Your measurements MUST be made in one of these machines.
Make sure just one timing measurement runs at the same time
Hands-on exercise will be tested in a regular lxplus machine

The HiLTOn (online-like)


Online-like machines available: High Level Trigger On-line
(HiLTOn)infrastructure.
Special permissions necessary. Ask an HLT expert.
Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 8 / 24
Introduction

Software And Tools:

FastTimerService
CMSSW C++ class used to measure the time spent in each
path and module in a CMSSW job.
It is the heart of the timing test
https://twiki.cern.ch/twiki/bin/view/CMS/FastTimerService

FastTimerServiceClient
Harvests the results from the FastTimerService and computes
some additional information (associated with harvesting step).
Documentation also at:
https://twiki.cern.ch/twiki/bin/view/CMS/FastTimerService

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 9 / 24


Introduction

Software And Tools:

TimingScripts
The TimingScripts package is a set of tools written in C++
and/or Python (mostly independent of CMSSW) in order to help
with machine characterization and menu validation
Can be found in git: https://github.com/cms-steam/TimingScripts
HEAD version can be extracted to any area (even your laptop):
git clone git@github.com:cms-steam/TimingScripts.git

Documentation:
https://twiki.cern.ch/twiki/bin/viewauth/CMS/
TriggerStudiesTiming#How_to_Perform_a_Timing_Validati
https://indico.cern.ch/event/390512/contributions/1823750/
attachments/781457/1071405/TSG_Meeting_06.23.2015.pdf

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 10 / 24


Introduction

Two Main Parts for Timing Measurement

1 Timing:
running a CMSSW HLT configuration file (with cmsRun) that includes the
FastTimerService code
most timer information is gathered by this tool, which acts like a DQM source
The file(s) produced are generally named DQM.root or DQMIO.root; these
can then be read by a DQM client.
2 Harvesting:
cmsRun job with another (and simpler) CMSSW configuration file, which
includes the FastTimerServiceClient (a DQM client) tool, needs to be run
over the output of the first step
produces a file normally named
DQM V000Y R000XXXXXX HLT FastTimerService All.root, where
XXXXXX is the run number.
output file contains relevant plots under the root directory DQM Data/Run
XXXXXX/HLT/Run summary/TimerService. These can be explored and
analyzed directly by the user
several useful scripts have been developed in order to study them
(TimingScripts) section above

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 11 / 24


Introduction

To Keep in Mind Before the Performing the Measurement:

Measurement must be done in one of the timing machines


described above: vocms003 or vocms004
Run only one job at a time on a machine (to avoid issues with
Hyperthreading and/or TurboBoost)
Ensure that other are not using the machine at the same time.
Single job withouth multithreading should not see any
differences using local or remote input samples.
Can run with Hyperthreading active if you know what you are
doing (specific instructions below)

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 12 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

1. Choose the right CMSSW release


CMSSW release version that you pick and the HLT menu to be
studied need to be compatible
Find guidance at
https://twiki.cern.ch/twiki/bin/view/CMSPublic/
SWGuideGlobalHLT
on which release to use for what menu
Follow the official trigger announcements

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 13 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

2. Getting the HLT Menu Configuration (hltGetConfiguration script)


For MC studies:
hltGetConfiguration hlt_menu_confdb_address
--globaltag name_of_global_tag --input location_of_input_root_sample
--mc --process name_of_process
--full --offline --l1-emulator name_of_emulator
--l1 name_of_l1menu_configuration --prescale prescale_value
--max-events max_num_events --output none --timing > hlt.py

For Data studies:


hltGetConfiguration hlt_menu_confdb_address
--globaltag name_of_global_tag --input location_of_input_root_sample
--data --process name_of_process
--full --offline --l1 name_of_l1menu_configuration
--l1 name_of_l1menu_configuration --prescale prescale_value
--max-events max_num_events --output none --timing > hlt.py

The – –timing switch instruments the HLT configuration file with the FastTimerService →

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 14 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

2. Getting the HLT Menu Configuration (FastTimerService)


if ’ FastTimerService ’ in process . __dict__ :
del process . FastTimerService
process . load ( " HLTrigger . Timer . F a s t T i m e r S e r v i c e _ c f i " )
process . FastTimerService . pri ntE ven tSum mar y = False
process . FastTimerService . printRunSummary = False
process . FastTimerService . printJobSummary = True
process . FastTimerService . enableDQM = True
process . FastTimerService . enableDQMbyPath = True
process . FastTimerService . ena ble DQM byMo dul e = True
process . FastTimerService . e n a b l e D Q M b y L u m i S e c t i o n = True
process . FastTimerService . d q m L u m i S e c t i o n s R a n g e = 2500
process . FastTimerService . dqmTimeRange = 1000. # ms
process . FastTimerService . dqm Tim eRe solu tio n = 5. # ms
process . FastTimerService . dqmPathTimeRange = 100. # ms
process . FastTimerService . d q m P a t h T i m e R e s o l u t i o n = 0.5 # ms
process . FastTimerService . d qm Mo d ul e Ti me R an g e = 40. # ms
process . FastTimerService . d q m M o d u l e T i m e R e s o l u t i o n = 0.2 # ms
process . FastTimerService . dqmPath = " HLT / TimerService "
process . FastTimerService . e n a b l e D Q M b y P r o c e s s e s = False
process . dqmOutput = cms . OutputModule ( " D QM R o o t O u t p ut M o d u l e " ,
fileName = cms . untracked . string ( " DQM . root " ))
process . FastTimerOutput = cms . EndPath ( process . dqmOutput )

https://twiki.cern.ch/twiki/bin/view/CMS/FastTimerService

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 15 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

3. Choosing an Appropiate Input Sample


Access your favorite input samples from EOS, and from all over the grid via
xrootd.
Find some samples at:
https://twiki.cern.ch/twiki/bin/view/CMS/TimingStudiesRecipes2017#Data_Samples or ask for
one
Can make your own input with TimingScripts (pointed above):
https:
//twiki.cern.ch/twiki/bin/viewauth/CMS/TriggerStudiesTiming#Making_Your_Own_Input_Sample_Ski

The access time itself should not be an issue


Time spent to read the data accounted by the hltGetRaw module in the
HLTriggerFirstPath path
Make sure the parameter ’skipFirstPath’ is set to ’False’
Do not run jobs with Hyperthreading using remote samples
Run at least 30k events

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 16 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

4. Executing the CMSSW Timing Job


If running without multithreading enabled, execute just one cmsRun job as always:
cmsRun hlt.py >& full.log&

This is safe whether accessing local or remote files.


To better compare results to online use local files and:
# enable TrigReport, TimeReport and MultiThreading
process.options = cms.untracked.PSet(
wantSummary = cms.untracked.bool( True ),
numberOfThreads = cms.untracked.uint32( 4 ),
numberOfStreams = cms.untracked.uint32( 0 ),)

Then start the script and assign the threads to 4 cores. This will assure you run as
online (i.e., without hyperthreading: one logical core used per physical core).
#example assign to run on cores 0,1,2,3
nohup taskset -c 0-3 cmsRun hlt.py >& full.log&

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 17 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

5. Creating the Harvesting configuration file


DQM root files created in the last step need to be ”harvestedı̈n order to be analyzed
Done invoking the FastTimerServiceClient, which is a DQM client
Copy the python lines given at
https://twiki.cern.ch/twiki/bin/view/CMS/FastTimerService

import FWCore . ParameterSet . Config as cms


process = cms . Process ( ’ HARVESTING ’)
# read all the DQMIO files produced by the previous jobs
process . source = cms . Source ( " DQMRootSource " ,
fileNames = cms . untracked . vstring (
" file : DQMIO_00 . root " ,
" file : DQMIO_01 . root " ,
...
)
)
# DQMStore service
process . load ( ’ DQMServices . Core . DQMStore_cfi ’)
# F astTimerService client
process . load ( ’ HLTrigger . Timer . f a s t T i m e r S e r v i c e C l i e n t _ c f i ’)
process . f a s t T i m e r S e r v i c e C l i e n t . dqmPath = " HLT / TimerService "
# DQM file saver
process . load ( ’ DQMServices . Components . DQMFileSaver_cfi ’)
process . dqmSaver . workflow = " / HLT / FastTimerService / All "

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 18 / 24


Timing Measurement

Operational Steps for Measurement on Data or MC:

6. Executing the Harvesting Job


Harvesting is executed as usual:
cmsRun hlt.py >& full.log&

Produces files normally named like


DQM V000Y R000XXXXXX HLT FastTimerService All.root, where XXXXXX is
the run number
can be explored and analyzed directly by the user; or use the TimingScripts:
https://github.com/cms-steam/TimingScripts
https://twiki.cern.ch/twiki/bin/viewauth/CMS/
TriggerStudiesTiming#How_to_Perform_a_Timing_Validati

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 19 / 24


Analysis of Results

Exploring the DQM root file by hand:


Open the DQM V000Y R000XXXXXX HLT FastTimerService All.root file in
root

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 20 / 24


Analysis of Results

Exploring the DQM root file by hand:


Open the DQM V000Y R000XXXXXX HLT FastTimerService All.root file in
root

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 20 / 24


Analysis of Results

Exploring the DQM root file by hand:


Open the DQM V000Y R000XXXXXX HLT FastTimerService All.root file in
root

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 20 / 24


Analysis of Results

Smarter analysis...

One can choose to write its own simple python code in order to retrieve the
appropiate histograms from
DQM V000Y R000XXXXXX HLT FastTimerService All.root
Or use the available scripts at
https://github.com/cms-steam/TimingScripts
Documentation available at
https://twiki.cern.ch/twiki/bin/viewauth/CMS/TriggerStudiesTiming#
How_to_Perform_a_Timing_Validati

Example of Twiki Instructions

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 21 / 24


Analysis of Results

Example of Plots with TimingScripts


Comparing the average time of two menus

python validation_plot.py --inputfiles file1,file2


--runs run1,run2 --processes process1,process2

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 22 / 24


Analysis of Results

Example of Plots with TimingScripts


Comparing the average timing of a path (or paths) in different menus:

python plotPath_Timing.py --inputfiles DQM-1.root,DQM-2.root,...DQM-N.root


--runs run1,run2,...runN --processes process1,process2,...processN
--paths HLT_TkMu20_v2,HLT_TkMu27_v2

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 23 / 24


Hands-on Exercise

https://twiki.cern.ch/twiki/bin/view/CMS/TriggerStudiesTutorial2017#HLT_
timing_studies

Edgar Carrera (ecarrera@usfq.edu.ec) HLT Tutorial October 27, 2017 24 / 24

You might also like