CORE modules

Contents

• CORE modules
  • Challenge
  • Rocs
  • settings
  • Synapse utilities
  • Ziptools
  • Downloader
  • Layout
  • Concordance Index

Challenge

Common utility to all challenges

class LocalData

Used by Challenge

getpath_data(filename)

getpath_gs(filename)

getpath_lb(filename)

getpath_template(filename)

Return full path of the template location named filename

class Challenge(challenge_name, verbose=False, download=True, **kargs)

Common class to all challenges

If you have not setup a .synapseConfig in your HOME, you must provide a synapse client

from dreamtools import *
s = Challenge('D2C1')
client = Login(username=username, password=pwd).client
s.client = client

constructor

Parameters:challenge_name (str) – Must be formatted as DXCY where X and Y are numbers. Intermediate challenges from e.g. D9.5 should be encoded as D9dot5CY

debug = None

alias of the challenge as DXCY form with X, Y being 2 numbers

directory

Gets directory where data will be stored.

download_template(sub_challenge=None)

Must be provided

get_pathname(filename)

Return pathname of a file to be found on ./config/dreamtools if available

import_scoring_class()

Dynamic import of a challenge class

c = Challenge('D7C1')
inst_class = c.import_scoring_class()

loadmat(filename)

Load a MATLAB matrix

mainpath = None

directory where is stored the configuration file and data files.

mkdir()

Create local dreamtools directory

onweb()

score(filename, sub_challenge=None)

Must be provided

test()

unzip(filename)

Simple method to extract all files contained in an archive

Rocs

Provides tools related to Receiver Operating Characteristic (ROC).

Those codes were directly translated from Perl or matlab codes. We should be using scikit-learn in the future.

class ROC(scores=None, classes=None)

A class to compute ROC, AUC and AUPRs for a binary problem

>>> r = ROC() # could provide scores and labels as arguments
>>> r.scores = [0.9,0.8,0.7,.6,.6,.3]
>>> r.classes = [1,0,1,0,1,1]
>>> r.compute_auc()
0.4375

Constructor

Parameters:

• scores (list) – the scores
• classes (list) – binary class made of 1 or 0 numerical values. Also called labels in the literature.

classes

Read/Write the classes

get_roc()

See get_statistics()

get_statistics()

Compute the ROC curve X/Y vectors and some other metrics

Returns:a dictionary with different metrics such as FPR (false positive rate), PTR (true positive rate).

plot_roc(roc=None)

Plot ROC curves

from dreamtools.core.rocs import ROC
r = ROC()
r.scores = [.9,.5,.6,.7,.1,.2,.6,.4,.7,.9, .2]
r.classes = [1,0,1,0,0,1,1,0,0,1,1]
r.plot_roc()

(Source code)

scores

Read/Write the scores

class ROCDiscovery(discovery)

A variant of ROC statistics

Used in D5C2 challenge.

Note

Does not work if any NA are found.

constructor

Parameters:discovery – a list of 0/1 where 1 means positives

compute_aupr(roc=None)

Returns AUPR normalised by (1-1/P) P being number of positives

get_statistics()

Return dictionary with FPR, TPR and other metrics

class D3D4ROC

get_statistics(gold_data, test_data, gold_index)

plot()

MCC(TP, TN, FP, FN)

Matthews correlation coefficient

settings

Tools to handle a configuration file.

class DREAMToolsConfig(verbose=False)

Synapse utilities

A module dedicated to synapse

The class SynapseClient is a specialised class built upon synapseclient package, which source code is on GitHub:

git clone git://github.com/Sage-Bionetworks/synapsePythonClient.git
cd synapsePythonClient
python setup.py install

This class may be removed but for now it is used in D8C1 challenge.

>>> from dreamtools.core import sageutils
>>> s = sageutils.SynapseClient()

class SynapseClient(username=None, password=None)

This class inherits all methods from synapseClient.

Be aware that most of the functionalities are now available in synapseclient itself. So, most of the methods that were written are hidden (double underscore) and may be removed in the future.

The only remaining feature is the automatic login, and simple version of the downloadSubmission method. There is also a json() method used throughout the dream8hpn code.

Constructor

Parameters:

• username – your synapse usename
• password – your synapse password

You can create create a file called .synapseConfig (note the dot) in your home directory and add something like:

[authentication]
username: yourlogin
password: yourpassword

downloadSubmissionAndFilename(sub, downloadFile=True, **kargs)

Return filename of a submission downloaded from synapse.

Parameters:

• sub – A submission (as a dictionary).
• version – The specific version to get. Defaults to the most recent version.
• downloadFile – Whether associated files(s) should be downloaded. Defaults to True. If set to False, downloadLocation and ifcollision are ignored
• downloadLocation – Directory where to download the Synapse File Entity. Defaults to the local cache.
• ifcollision – Determines how to handle file collisions. May be “overwrite.local”, “keep.local”, or “keep.both”. Defaults to “keep.both”.

Warning

ifcollision does not seem to work (0.5.1)

getMyProfile()

Returns user profile

json(data)

Transform relevant object into json object

class Login(client=None, username=None, password=None)

A simple class to login to synapse

Parameters:client – Connection to synapse takes a couple of seconds. This may be too much if in a debugging mode or accessing to synapse from different places. The login can be instantiate with an existing instance of SynapseClient, if which case, the instance creation is fast. Otherwise, the default behaviour is to create a new connection.

>>> from dreamtools.core.sageutils import Login
>>> l = Login()
This is a SynapseClient built on top of Synapse class.
Trying to login automatically.
Welcome, *****************
You're logged in Synapse
Welcome, XXX

In [10]: l = sageutils.Login(l)

Ziptools

class ZIP

Simple utility to load a ZIP file

Note

could be moved to easydev package

extractall(path)

loadZIPFile(filename)

Loads a ZIP file

This method uses the zipfile module and stores the data into zip_data. The filenames contained within this archive can be found in zip_filenames. To read the data contained in the first filename, type:

self.zip_data.open(self.filenames[0].read()

Parameters:filename (str) – the ZIP filename to load

read(filename)

Downloader

Utility to download a synapse project in the dreamtools directory

class Downloader(challenge, client=None, username=None, password=None)

Factory to download gold standard files

Download a synpase file once for all in the dreamtools directory.

constructor

Parameters:challenge (str) – alias of a challenge (e.g., D5C1)

To automatically connect to synapse, create a file called .synapseConfig with this content:

[authentication]
username: email
password: password

download(synid)

Download a file into the dreamtools directory

Parameters:synid – a valid synapse id (e.g., syn123456)

You must have a login on synapse website.

Layout

Layout to create a new challenge from scratch

class Layout(name, verbose=True)

Class to create automatic layout for a given challenge

Usage:

dreamtools-layout --name D8C2

Warning

for developers only.

create_layout()

layout(args=None)

This function is used by the standalone application called dreamscoring

dreamtools-layout --help

Concordance Index

Concordance index computation (exact version)

Based on R code provided by Ben Sauerwine and Erhan Bilal double checked with concordance.index from survcomp R package.

class ConcordanceIndex(survtime=None, survevent=None)

See cindex() function for details

cindex(prediction)

Returns concordance index

cindex(prediction, survtime, survevent)

Function to compute the concordance index for a risk prediction, i.e., the probability that, for a pair of randomly chosen comparable samples, the sample with the higher risk prediction will experience an event before the other sample or belongs to a higher binary class.

Parameters:

• prediction – a vector of risk predictions.
• survtime – a vector of event times.
• survevent – a vector of event occurence indicators (True and False).

>>> from dreamtools.core.cindex import cindex
>>> print(cindex([0, 1, 3,4], [0, 4, 3, 1], [True]*4))
0.5
>>> print(cindex([0, 1, 3,4], [0, 1, 3, 4], [True]*4))
0.0

concordanceIndex(prediction, survtime, survevent)