4.4.2. Utility functions — hop.utilities

Random mix of convenience functions that don’t fit anywhere else.

For messages I should probably use python’s logger module but this is working so far (even though it’s pretty crappy).

class hop.utilities.CustomProgressMeter(numsteps, format=None, interval=10, offset=1, verbose=None, dynamic=True, format_handling='auto', quiet=None)

ProgressMeter that uses addition ‘%(other)s’ in format string.

See also

MDAnalysis.lib.log.ProgressMeter

echo(step, other)

Output status for step with additional information other.

class hop.utilities.DefaultDict(defaultdict, userdict=None, **kwargs)

Dictionary based on defaults and updated with keys/values from user.

class hop.utilities.Fifo
pop()

Remove and return the leftmost element.

class hop.utilities.IntrospectiveDict(*args, **kwargs)

A dictionary that contains its keys as attributes for easier introspection.

Keys that collide with dict methods or attributes are _not_ added as attributes.

The implementation is simple and certainly not optimized for larger dictionaries or ones which are often accessed. Only use it for ‘final results’ collections that you are likely to investigate interactively.

ARGH: This cannot be pickled safely.

hop.utilities.Pearson_r(x, y)

Pearson’s r (correlation coefficient)

r = Pearson(x,y)

x and y are arrays of same length

Historical note: Naive implementation of Pearson’s r:

Ex = scipy.stats.mean(x) Ey = scipy.stats.mean(y)

covxy = numpy.sum((x-Ex)*(y-Ey)) r = covxy/math.sqrt(numpy.sum((x-Ex)**2)*numpy.sum((y-Ey)**2)) return r

class hop.utilities.Ringbuffer(capacity, iterable=None)

Ring buffer of size capacity; ‘pushes’ data from left and discards on the right.

append(x)

Add an element to the right side of the deque.

class hop.utilities.Saveable(*args, **kwargs)

Baseclass that supports save()ing and load()ing.

Override the class variables

_saved_attributes = [] # list attributes to be pickled _merge_attributes = [] # list dicts to be UPDATED from the pickled file with load(merge=True) _excluded_attributes = [] # list attributes that should never be pickled

Note:

_saved_attributes = ‘all’ # pickles ALL attributes, equivalent to self.__dict__.keys()
# (use _excluded_attributes with ‘all’!)

Use _excluded_attributes to filter out some attributes such as type(‘method-wrapper’) objects that cannot be pickled (e.g. when using properties).

filename(filename=None, ext=None, set_default=False, use_my_ext=False)

Supply a file name for the object.

fn = filename() —> <default_filename> fn = filename(‘name.ext’) —> ‘name’ fn = filename(ext=’pickle’) —> <default_filename>’.pickle’ fn = filename(‘name.inp’,’pdf’) –> ‘name.pdf’ fn = filename(‘foo.pdf’,ext=’png’,use_my_ext=True) –> ‘foo.pdf’

The returned filename is stripped of the extension (use_my_ext=False) and if provided, another extension is appended. Chooses a default if no filename is given. Raises a ValueError exception if no default file name is known.

If set_default=True then the default filename is also set.

use_my_ext=True lets the suffix of a provided filename take priority over a default ext(tension).

load(filename=None, merge=False)

Reinstantiate class from a pickled file (produced with save()).

save(filename=None)

Save class to a pickled file.

hop.utilities.asiterable(obj)

Return an object that is an iterable: object itself or wrapepd in a list.

iterable <– asiterable(something)

Treats strings as NOT-iterable.

hop.utilities.autocorrelation_fft(series, include_mean=False, periodic=False, start=None, stop=None, **kwargs)

Calculate the auto correlation function.

acf = autocorrelation_fft(series,include_mean=False,**kwargs)

The time series is correlated with itself across its whole length. It is 0-padded and the ACF is corrected for the 0-padding (the values for larger lags are increased) unless mode=’valid’ (see below). Only the [0,len(series)[ interval is returned. The series is normalized to ots 0-th element.

Note that the series for mode=’same’|’full’ is inaccurate for long times and should probably be truncated at 1/2*len(series). Alternatively, only sample a subseries with the stop keyword.

Arguments:

series (time) series, a 1D numpy array include_mean False: subtract mean(series) from series periodic False: corrected for 0-padding

True: return as is
start,stop If set, calculate the ACF of series[start:stop] with series;
in this case mode=’valid’ is enforced
kwargs keyword arguments for scipy.signal.fftconvolve
mode = ‘full’ | ‘same’ | ‘valid’ (see there)
hop.utilities.averaged_autocorrelation(series, length=None, sliding_window=None, **kwargs)

Calculates the averaged ACF of a series.

mean(acf), std(acf) = averaged_autocorrelation(series,length=None,sliding_window=None):

Calculate the ACF of a series for only a fraction of the total length, <length> but repeat the calculation by setting the origin progressively every <sliding_window> steps and average over all the ACFs.

Arguments:

series time series (by default, mean will be removed) length length (in frames) of the ACF (default: 1/2*len(series)) sliding_window repeat ACF calculation every N frames (default: len(series)/100) kwargs additional arguments to autocorrelation_fft()

hop.utilities.close_log()

Close open logfile; must be done manually.

hop.utilities.easy_load(names, baseclass, keymethod)

Instantiate a class either from an existing instance or a pickled file.

instance_list = easy_load(names,baseclass,keymethod)

>>> x = easy_load(<filename>,Xclass,'my_method_name')
>>> [x1,x2,...] = easy_load([<filename1>, <fn2>,...], Xclass,'my_method_name')
>>> [x1,x2,...] = easy_load([x1, x2, ..], Xclass,'my_method_name')

If the argument does not implement the keymethod then try loading from a file.

API:

For this to work, the baseclass (eg Saveable) must be able to instantiate itself using

x = baseclass(filename=name)

If a single name is given, a singlet is returned, otherwise a list of instances.

(The docs are longer than the code…)

hop.utilities.fileextension(filename, default=None)

Return the file extension without the leading dot or the default.

hop.utilities.filename_function(self, filename=None, ext=None, set_default=False, use_my_ext=False)

Supply a file name for the object.

fn = filename() —> <default_filename> fn = filename(‘name.ext’) —> ‘name’ fn = filename(ext=’pickle’) —> <default_filename>’.pickle’ fn = filename(‘name.inp’,’pdf’) –> ‘name.pdf’ fn = filename(‘foo.pdf’,ext=’png’,use_my_ext=True) –> ‘foo.pdf’

The returned filename is stripped of the extension (use_my_ext=False) and if provided, another extension is appended. Chooses a default if no filename is given. Raises a ValueError exception if no default file name is known.

If set_default=True then the default filename is also set.

use_my_ext=True lets the suffix of a provided filename take priority over a default ext(tension).

hop.utilities.fixedwidth_bins(delta, xmin, xmax)

Return bins of width delta that cover xmin,xmax (or a larger range).

dict = fixedwidth_bins(delta,xmin,xmax)

The dict contains ‘Nbins’, ‘delta’, ‘min’, and ‘max’.

hop.utilities.flatiter(seq)

Returns an iterator that flattens a sequence of sequences of sequences… (c) 2005 Peter Otten, at http://www.thescripts.com/forum/thread23631.html

hop.utilities.flatten(sequence) → list

Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables).

Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, MyVector(8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10]

From http://kogs-www.informatik.uni-hamburg.de/~meine/python_tricks

hop.utilities.iterable(obj)

Returns True if obj can be iterated over and is NOT a string.

hop.utilities.linfit(x, y, dy=[])

Fit a straight line y = a + bx to the data in x and y; errors on y should be provided in dy in order to assess the goodness of the fit and derive errors on the parameters.

result_dict = linfit(x,y[,dy])

Fit y = a + bx to the data in x and y by analytically minimizing chi^2. dy holds the standard deviations of the individual y_i. If dy is not given, they are assumed to be constant (note that in this case Q is set to 1 and it is meaningless and chi2 is normalised to unit standard deviation on all points!).

Returns the parameters a and b, their uncertainties sigma_a and sigma_b, and their correlation coefficient r_ab; it also returns the chi-squared statistic and the goodness-of-fit probability Q (that the fit would have chi^2 this large or larger; Q < 10^-2 indicates that the model is bad — Q is the probability that a value of chi-square as _poor_ as the calculated statistic chi2 should occur by chance.)

result_dict =

intercept, sigma_intercept a +/- sigma_a slope, sigma_slope b +/- sigma_b parameter_correlation correlation coefficient r_ab

between a and b

chi_square chi^2 test statistic Q_fit goodness-of-fit probability

Based on ‘Numerical Recipes in C’, Ch 15.2.

hop.utilities.mkdir_p(path)

Create a directory path with subdirs but do not complain if it exists.

This is like GNU mkdir -p path.

hop.utilities.msg(level[, m])

1) Print message string if the level <= verbose. level describes the priority with lower = more important.

Terminate string with n if a newline is desired or r to overwrite the current line (eg for output progress indication)

Note that if the global verbosity level is < 0 then the message is also written to the logfile.

2) If called without a string then msg(level) returns True if it would print a message, and False otherwise.

hop.utilities.set_verbosity([level, ]logfilename=<filename>)

Set the verbosity level level < 0 : level <- abs(level) but output is also appended to logfile level == 0: minimum level == 3: verbose level > 3 : debugging

Unlink path but do not complain if file does not exist.