apis

The apis package contains a collection of tools for working with non-SHEPHARD format files and third-party tools. In particular, SHEPHARD uses FASTA files for sequence storage — a format SHEPHARD does not control — so interaction with FASTA files occurs via APIs.

The distinction between an interface and an api is one of coupling. Interfaces read/write the strict SHEPHARD tab-separated formats and maintain a deliberate loose coupling to the outside world. APIs, by contrast, bridge to formats or tools where some coupling is unavoidable or desirable (FASTA parsing, or first-party Holehouse-lab predictors such as metapredict and ALBATROSS). We anticipate the number of api modules to remain small.

fasta

The fasta module reads and writes generic (non-UniProt) FASTA files. fasta_to_proteome builds a Proteome from a FASTA file, with control over how unique IDs are derived (auto-incremented, taken from the header, or parsed via a user-supplied function), how attributes are extracted, and how invalid sequences are handled. proteome_to_fasta writes a Proteome out (optionally embedding attributes in the header), and shephard_fasta_to_proteome reads a FASTA file that was generated by proteome_to_fasta, restoring the unique ID, name, and attributes.

fasta_to_proteome(filename, proteome=None, build_unique_ID=None, build_attributes=None, use_header_as_unique_ID=False, force_overwrite=False, invalid_sequence_action='fail')[source]

Stand alone function that allows the user to build a Proteome from a standard FASTA file, or add sequences in a FASTA file to an existing Proteome.

This function can be used to read additional sequences into an existing Proteome object, or create a new Proteome object from a FASTA file. In addition, some control over how invalid sequences should be dealt with are defined by the invalid_sequence_action flag.

The input filename must be a FASTA file without duplicate headers. If the file has duplicate headers and these have to be further processed we suggest using the protfasta (https://protfasta.readthedocs.io/) package to parse through the FASTA file first creating a santizied input FASTA.

The FASTA file is parsed into a set of proteins, each of which has (1) a unique ID, (2) a name, (3) a sequence, and (4) optionally, a dictionary of attributes.

The protein name is defined as the full FASTA header, and the sequence based on the FASTA record sequence. Sequence validation is also provided at the point of file-parsing. The unique ID and attributes are discused below.

Each protein in a Proteome must have a unique_ID associated with it. There are three ways a FASTA file can be used to generate a unique ID:

By using the FASTA header as a unique ID, although this fails if there are non-unique FASTA headers.

By parsing the FASTA header, to extract out a unique ID. For example, FASTA files generated by other databases often include unique identifiers in a structured way, which could be extracted in a consistent manner for every FASTA record.

By incrementing an automatically unique ID, removing any dependence on the FASTA header itself.

These options can be selected using the flags provided in the function signature. Note that if both using the FASTA header directly and parsing the FASTA header are selected an exception will be raised as only one of these two can be requested simultaneously.

By default, a numerically unique value is used. Note that if your are reading FASTA files generated by UniProt, we recommend the api.uniprot functions instead of the more generic api.fasta functions.

To build protein attributes, this can in principle be parsed out of the FASTA header by providing a build_attributes() function. However, we would in general suggest that the better way to do this would be annotate a Proteome with attribibutes and the same those using the associated interfaces.si_protein_attributes functionality.

Parameters:

filename (string) – Name of the FASTA file we’re going to parse in.
proteome (Proteome (default = None)) – If a Proteome object is provided the FASTA file will be read and added to the existing proteome, whereas if set to None a new Proteome will be generated.
build_unique_ID (funct (default = None)) –
This parameter allows a user-defined function that is used to convert the FASTA header to a (hopefully) unique string. This can be useful if the FASTA header is well structured and includes a specific, useful unique ID that can be used as the unique_ID.

Specifically, the build_unique_ID function should take in the a str (a FASTA header) and return a string which will be used as a unique ID.
build_attributes (funct (default = None)) – This parameter allows a user-defined function that allows meta-information from the FASTA header to be converted into protein attributes. Specifically, build_attributes should be a function which takes in the FASTA header as a string and returns a dictionary where key:value pairs are assigned as protein attributes. This can be useful if the FASTA header is well-structured.
use_header_as_unique_ID (bool (default = False)) – If this flag is set to true, it means the unique_ID is set to the FASTA file header. If non-unique headers are found this will trigger an exception.
force_overwrite (bool (default = False)) – If this flag is set to true and we encounter a unique_ID that is already in the proteome the newer value overwrites the older one. This is mostly useful if you are adding in a file with known duplicate entries OR combining multiple FASTA files where you know there’s some duplications. Important - if we’re building unique IDs based on numerical record indices then EVERY FASTA entry will be given a unique_ID (meaning force_overwrite is irrelevant in this case).
invalid_sequence_action (str (default = 'fail')) –
Selector which defines the behaviour if a sequence with a non- standard amino acid is encountered. Valid options and their meaning are listed below:
- ignore - invalid sequences are completely ignored.
- fail - invalid sequence cause parsing to fail and throw an exception.
- remove - invalid sequences are removed.
- convert - invalid residues are converted to valid residues.
- convert-ignore - invalid sequences are converted to valid sequences and any remaining invalid residues are ignored.

Returns:

Returns an initialized Proteome object

Return type:

Proteome

shephard_fasta_to_proteome(filename, proteome=None, force_overwrite=False, invalid_sequence_action='fail')[source]

Stand alone function that allows the user to build a proteome from a FASTA file generated by SHEPHARD (using the proteome_to_fasta() function. When SHEPHARD generates a FASTA file it uses a general convention for encoding the unique ID, protein name, and attributes.

Specifically, the FASTA header has the form

>SHPRD|<UNIQUE_ID>|<PROTEIN_NAME|SHPRD_ATTRIBUTES=<ATTRIBUTE_NAME>=<ATTRIBUTE_VALUE>

Where an arbitrary number of name/value attribute paris can be encoded separated by a tab character.

WARNING: The support for protein attributes in FASTA files is included mainly for easy sharing of FASTA files that are usable outside of SHEPHARD. We recommend using interfaces.si_protein_attributes functions for dealing with Protein attributes.

Parameters:

filename (str) – Name of the FASTA file we’re going to parse in.
proteome (Proteome (default = None)) – If a Proteome object is provided the FASTA file will be read and added to the existing proteome, whereas if set to None a new Proteome will be generated.
force_overwrite (bool (default = False)) – If this flag is set to true and we encounter a unique_ID that is already in the proteome the newer value overwrites the older one. This is mostly useful if you are adding in a file with known duplicate entries OR combining multiple FASTA files where you know there’s some duplications. Important - if we’re building unique IDs based on numerical record indices then EVERY FASTA entry will be given a unique_ID (meaning force_overwrite is irrelevant in this case).
invalid_sequence_action (str (default = 'fail')) –
Selector which defines the behaviour if a sequence with a non- standard amino acid is encountered. Valid options and their meaning are listed below:
- ignore - invalid sequences are completely ignored.
- fail - invalid sequence cause parsing to fail and throw an exception.
- remove - invalid sequences are removed.
- convert - invalid residues are converted to valid residues.
- convert-ignore - invalid sequences are converted to valid sequences and any remaining invalid residues are ignored.

Returns:

Returns an initialized Proteome object

Return type:

Proteome Object

proteome_to_fasta(filename, proteome, include_attributes_in_header=False)[source]

Stand alone function that allows the user to write a SHEPHARD-specific FASTA file from a Proteome object.

WARNING: The support for protein attributes in FASTA files is included mainly for easy sharing of FASTA files that are usable outside of SHEPHARD. We recommend using interfaces.si_protein_attributes functions for dealing with Protein attributes.

Parameters:

filename (str) – Name of the FASTA file we’re going to write to. We will automatically overwrite a file if it’s there, so be careful! Note that no extension is added in part because FASTA files can be .f/.fa/.fasta. Recommended a .fasta file extension.
proteome (Proteome) – The proteome object that will be written to disk
include_attributes_in_header (bool (default = False)) – Flag which if set to true means each Protein’s attributes will be included in the FASTA header. We generally do not recommend this other than times when sharing annotated FASTA files outside of a SHEPHARD ecosystem would be useful.

Returns:

No return object but a new file will be written

Return type:

None

uniprot

The uniprot module provides tools for working with UniProt data. UniProt FASTA headers follow the convention >xx|ACCESSION|xxxx, and uniprot_fasta_to_proteome uses the accession as the unique ID automatically. uniprot_accession_from_line exposes the header-parsing helper directly, and uniprot_proteome_to_fasta writes a Proteome back out using the UniProt-style header.

uniprot_fasta_to_proteome(filename, proteome=None, force_overwrite=False, invalid_sequence_action='fail')[source]

Stand alone function that allows the user to build a proteome from a standard FASTA file downloaded from UniProt

This function assumes the uniprot-standard format for the header file has been maintained - i.e.

>>> >xx|ACCESSION|xxxx

Where ACCESSION is the uniprot accession and will be used as the unique_ID

Parameters:

filename (string) – Name of the FASTA file we’re going to parse in. Note the protein name will be defined as the full FASTA header for each entry.
proteome (Proteome) – If a Proteome object is provided the FASTA file will be read and added to the existing proteome, whereas if set to None a new Proteome will be generated.
force_overwrite (bool (default = False)) – If this flag is set to true and we encounter a unique_ID that is already in the proteome the newer value overwrites the older one. This is mostly useful if you are adding in a file with known duplicate entries OR combining multiple FASTA files where you know there’s some duplications. Important - if we’re building unique IDs based on numerical record indices then EVERY FASTA entry will be given a unique_ID (meaning force_overwrite is irrelevant in this case).
invalid_sequence_action (str (default = 'fail')) –
Selector which defines the behaviour if a sequence with a non- standard amino acid is encountered. Valid options and their meaning are listed below:
- ignore - invalid sequences are completely ignored
- fail - invalid sequence cause parsing to fail and throw an exception
- remove - invalid sequences are removed
- convert - invalid residues are converted to valid residues
- convert-ignore - invalid sequences are converted to valid sequences and any remaining invalid residues are ignored.

Returns:

Returns an initialized Proteome object

Return type:

Proteome

uniprot_proteome_to_fasta(filename, proteome)[source]

Stand alone function that allows the user to write a FASTA file from a Proteome under the assumption that the Proteome was built from a uniprot FASTA.

Practically, this just means that the Protein.name variable is used for the FASTA header, although the function will fail if duplicate headers are found.

Parameters:

filename (string) – Name of the FASTA file we’re going to write sequences to
proteome (Proteome) – The Proteome object from which FASTA file will be generated

Returns:

No return variable but wll write to file

Return type:

None

uniprot_accession_from_line(line)[source]

Function that converts a header from a uniprot fasta file to extract the uniprot ID. This an example of the type of function that can be passed to quickstart using the extract_unique_id argument.

This function assumes the uniprot-standard format for the header file has been maintained - i.e.

>>> >xx|ACCESSION|xxxx

where ACCESSION is the uniprot accession.

Parameters:: line (string) – String where we expect the uniprot ID to be contained within two ‘pipe’ characters (‘|’).
Returns:: Returns the uniprot ID, although this is not formally validated. However, assuming the string follows standard uniprot fasta header conventions this should be true!
Return type:: string

Annotating from the UniProt REST API

As well as reading UniProt FASTA files, the uniprot module can pull annotations directly from the UniProt REST API and convert them into SHEPHARD Domains, Sites, and Tracks. This adds no new dependencies — requests are made using the Python standard library — but it does, of course, require network access.

This means you can go from a bare FASTA file to a richly annotated Proteome in two lines, without downloading or parsing any UniProt flat files:

from shephard.apis import uniprot

P = uniprot.uniprot_fasta_to_proteome('human_proteome.fasta')
summary = uniprot.annotate_proteome_with_uniprot(P)

Two functions are provided:

annotate_protein_with_uniprot() annotates a single Protein using a single API call.
annotate_proteome_with_uniprot() annotates an entire Proteome, batching accessions so a proteome of several thousand proteins costs a few tens of API calls rather than one call per protein.

Both are stateless, and both use the same conversion engine internally, so annotating one protein and annotating a whole proteome always give identical results.

By default the unique_ID of each Protein is used as its UniProt accession — one of the reasons we recommend using UniProt accessions as unique IDs. If your unique IDs are something else, pass accession (single protein) or a function via accession_from_protein (proteome).

How UniProt features become SHEPHARD annotations

Features are mapped by their shape rather than by their name, which means a feature type that is sometimes a point and sometimes a range is always handled sensibly:

UniProt feature	SHEPHARD annotation
Spans a range of residues	A Domain, with `domain_type` derived from the feature type (e.g. `uniprot_domain`, `uniprot_region`, `uniprot_coiled_coil`)
Falls on a single residue	A Site, with `site_type` derived from the feature type (e.g. `uniprot_modified_residue`, `uniprot_glycosylation`)
Disulfide bond or cross-link	A pair of Sites, one on each bonded residue, each recording the position of its partner in the `partner_position` attribute
PDB cross-references (`experimental_structures=True`)	A Domain per structurally-resolved region, recording the PDB ID, method, resolution, and chains
Secondary structure (`secondary_structure_track=True`)	A symbols Track where each residue is `H` (helix), `E` (strand), `T` (turn), or `-`
Structural coverage (`structure_coverage_track=True`)	A values Track that is `1` for residues covered by an experimental structure and `0` elsewhere

The disulfide/cross-link case is worth calling out. UniProt encodes these using a start and an end position, but those two positions are bonded partners, not the boundaries of a contiguous region — so adding a Domain spanning them would assert a region that does not exist. SHEPHARD therefore adds a Site on each partner instead.

Annotation metadata

Every annotation created this way carries an attribute dictionary, so annotations remain interpretable and traceable long after they were added:

Attribute	Meaning
`source`	Always `uniprot`
`uniprot_accession`	The accession the annotation came from
`uniprot_feature_type`	The original UniProt feature type, e.g. `Modified residue`
`description`	UniProt’s description, e.g. `Phosphoserine`
`evidence`	Flattened evidence codes, e.g. `ECO:0000269\|PubMed:25732823`
`cross_references`	Feature cross-references, e.g. the ChEBI ID of a bound ligand
`uniprot_feature_id`	The UniProt feature identifier, where one exists
`partner_position`	For disulfide bonds and cross-links, where the bonded partner is

All values are strings, so annotations round-trip cleanly through the SHEPHARD file formats (see SHEPHARD files).

Choosing what to retrieve

Each class of annotation is controlled by its own boolean keyword. The complete list of group names is available programmatically:

from shephard.apis import uniprot
print(uniprot.uniprot_annotation_groups())

The groups, and their defaults, are:

Keyword	UniProt features	Default
`domains`	Domain	`True`
`regions`	Region	`True`
`motifs`	Motif	`True`
`repeats`	Repeat	`True`
`zinc_fingers`	Zinc finger	`True`
`coiled_coils`	Coiled coil	`True`
`compositional_bias`	Compositional bias	`True`
`dna_binding`	DNA binding	`True`
`transmembrane`	Transmembrane, Intramembrane, Topological domain	`True`
`binding_sites`	Binding site	`True`
`active_sites`	Active site	`True`
`other_sites`	Site	`True`
`modified_residues`	Modified residue	`True`
`glycosylation`	Glycosylation	`True`
`lipidation`	Lipidation	`True`
`disulfide_bonds`	Disulfide bond	`True`
`cross_links`	Cross-link	`True`
`molecule_processing`	Chain, Peptide, Propeptide, Signal, Transit peptide, Initiator methionine	`False`
`secondary_structure`	Helix, Beta strand, Turn (as Domains)	`False`
`mutagenesis`	Mutagenesis	`False`
`natural_variants`	Natural variant	`False`

The four groups that are off by default are off for good reasons. molecule_processing is dominated by the Chain feature, which usually spans the whole protein; mutagenesis and natural_variants describe experiments and population variation rather than the native protein, and for well-studied proteins the variant list can run to thousands of entries (human p53 currently carries over 1,300); and secondary_structure as Domains is usually less useful than the equivalent Track.

Only the data needed for the requested annotations is downloaded, so turning groups off makes queries meaningfully faster and smaller.

Sequence verification (important)

UniProt annotations are positional — they are statements about residue numbers. If the sequence in your Proteome differs from the sequence UniProt currently holds (a different isoform, a sequence edited locally, or simply a FASTA file downloaded before UniProt revised the entry), then applying those positions produces annotations that are silently wrong rather than absent. This is a far more dangerous failure mode than an error.

Both functions therefore compare the local sequence against the UniProt sequence before annotating anything, and refuse to annotate a protein whose sequence has drifted. This is not a hypothetical concern: of the nine proteins in SHEPHARD’s own bundled test set, one (Q9UJX3) has since been revised by UniProt from 599 to 565 residues, and is correctly refused.

The check can be turned off with verify_sequence=False, but we strongly recommend leaving it on.

Error handling and the summary dictionary

Both functions return a summary reporting what was actually done. For a single protein this is a dictionary of counts; for a proteome it is a dictionary of such dictionaries, keyed by unique_ID:

summary = uniprot.annotate_proteome_with_uniprot(P, safe=False, verbose=False)

print(summary['O00401'])
# {'domains': 20, 'sites': 6, 'tracks': 0, 'skipped': 0}

print(summary['Q9UJX3'])
# {'domains': 0, 'sites': 0, 'tracks': 0, 'skipped': 0, 'error': 'sequence mismatch'}

The safe keyword decides what happens when something goes wrong — an accession missing from UniProt, a sequence mismatch, an annotation that falls outside the protein, or a clash with an existing annotation. With safe=True (the default) any of these raises an APIException; with safe=False the affected annotation or protein is skipped, recorded in the summary under error or skipped, and the run continues. For proteome-scale work safe=False is usually what you want, so that a handful of stale accessions cannot abort a long job.

Network failures are handled separately and always raise. Rate-limited (HTTP 429) and transient server-side (5xx) responses are retried with exponential backoff, honouring UniProt’s Retry-After header where present; the number of retries is set with n_retries. Client errors such as a malformed accession are not retried, and UniProt’s own error message is included in the exception.

Batching and being a good citizen

annotate_proteome_with_uniprot() requests accessions in batches of chunk_size (default 100), so annotating 5,000 proteins is roughly 50 requests rather than 5,000. Duplicate accessions are collapsed, and proteins whose unique_ID is not a validly-formatted accession are rejected locally without ever reaching the API (a malformed accession would otherwise fail the entire batch).

If you are annotating a large proteome repeatedly, consider annotating once and then writing the results out with the Interfaces functions, rather than re-querying UniProt each time you run an analysis. This is faster, is reproducible (UniProt is a moving target), and is considerably kinder to a free public resource.

annotate_protein_with_uniprot(protein, domains=True, regions=True, motifs=True, repeats=True, zinc_fingers=True, coiled_coils=True, compositional_bias=True, dna_binding=True, transmembrane=True, molecule_processing=False, secondary_structure=False, binding_sites=True, active_sites=True, other_sites=True, modified_residues=True, glycosylation=True, lipidation=True, mutagenesis=False, natural_variants=False, disulfide_bonds=True, cross_links=True, experimental_structures=False, secondary_structure_track=False, structure_coverage_track=False, accession=None, verify_sequence=True, prefix='uniprot_', safe=True, autoname=True, verbose=True, base_url='https://rest.uniprot.org/uniprotkb', timeout=30, n_retries=3)[source]

Function that annotates a single Protein with annotations pulled live from the UniProt REST API.

The function is stateless - it makes one API call, converts the returned UniProt features into SHEPHARD annotations, and applies them to the passed Protein.

By default the Protein’s unique_ID is used as the UniProt accession, which is why we recommend using UniProt accessions as unique_IDs. If your unique_IDs are something else, pass the accession explicitly.

If you are annotating more than a handful of proteins, use annotate_proteome_with_uniprot() instead - it batches accessions into a small number of API calls rather than making one call per protein, and processes the results identically.

UniProt features are converted to SHEPHARD annotations as follows:

Features that span a region become Domains
Features on a single residue become Sites
Disulfide bonds and cross-links become a pair of Sites, one on each bonded residue, each recording its partner’s position
Optionally, secondary structure and experimental structure coverage become Tracks

Every annotation carries an attribute dictionary with the UniProt feature type, its description, its evidence codes, any cross-references (e.g. the ChEBI ID of a bound ligand), the UniProt feature ID where one exists, and the accession the annotation came from.

Parameters:

protein (shephard.protein.Protein) – Protein object to be annotated.
domains (bool (default = True)) – Annotate UniProt ‘Domain’ features.
regions (bool (default = True)) – Annotate ‘Region’ features.
motifs (bool (default = True)) – Annotate ‘Motif’ features.
repeats (bool (default = True)) – Annotate ‘Repeat’ features.
zinc_fingers (bool (default = True)) – Annotate ‘Zinc finger’ features.
coiled_coils (bool (default = True)) – Annotate ‘Coiled coil’ features.
compositional_bias (bool (default = True)) – Annotate ‘Compositional bias’ features.
dna_binding (bool (default = True)) – Annotate ‘DNA binding’ features.
transmembrane (bool (default = True)) – Annotate ‘Transmembrane’, ‘Intramembrane’ and ‘Topological domain’ features.
molecule_processing (bool (default = False)) – Annotate ‘Chain’, ‘Peptide’, ‘Propeptide’, ‘Signal’, ‘Transit peptide’ and ‘Initiator methionine’ features.
secondary_structure (bool (default = False)) – Annotate ‘Helix’, ‘Beta strand’ and ‘Turn’ features as Domains.
binding_sites (bool (default = True)) – Annotate ‘Binding site’ features.
active_sites (bool (default = True)) – Annotate ‘Active site’ features.
other_sites (bool (default = True)) – Annotate general ‘Site’ features.
modified_residues (bool (default = True)) – Annotate ‘Modified residue’ features.
glycosylation (bool (default = True)) – Annotate ‘Glycosylation’ features.
lipidation (bool (default = True)) – Annotate ‘Lipidation’ features.
mutagenesis (bool (default = False)) – Annotate ‘Mutagenesis’ features.
natural_variants (bool (default = False)) – Annotate ‘Natural variant’ features. Off by default because well-studied proteins can carry thousands of these.
disulfide_bonds (bool (default = True)) – Annotate ‘Disulfide bond’ features as paired Sites.
cross_links (bool (default = True)) – Annotate ‘Cross-link’ features as paired Sites.
experimental_structures (bool (default = False)) – Add a Domain for every region covered by an experimentally-determined structure, taken from the PDB cross-references, recording the PDB identifier, experimental method, resolution and chains.
secondary_structure_track (bool (default = False)) – Add a symbols Track of secondary structure (‘H’, ‘E’, ‘T’ or ‘-‘).
structure_coverage_track (bool (default = False)) – Add a values Track that is 1 for residues covered by an experimental structure and 0 elsewhere.
accession (str (default = None)) – UniProt accession to look this Protein up by. If None the Protein’s unique_ID is used.
verify_sequence (bool (default = True)) – If True, the local sequence is checked against the UniProt sequence and the protein is not annotated if they differ. Strongly recommended - UniProt annotations are positional.
prefix (str (default = ‘uniprot’)) – Prefix applied to every generated domain_type, site_type and Track name.
safe (bool (default = True)) – If True, any problem raises an exception. If False, problems are skipped and counted in the returned summary.
autoname (bool (default = True)) – Passed through to Protein.add_domain().
verbose (bool (default = True)) – Whether to print warnings about skipped annotations.
base_url (str (default = UNIPROT_REST_URL)) – Base URL of the UniProtKB REST API.
timeout (int or float (default = 30)) – Per-request timeout in seconds.
n_retries (int (default = 3)) – Number of times a transient failure is retried before giving up.

Returns:

Returns a summary dictionary with the counts of ‘domains’, ‘sites’ and ‘tracks’ added, and of ‘skipped’ annotations. If the protein could not be annotated the dictionary also carries an ‘error’ key.

Return type:

dict

Raises:

shephard.exceptions.APIException – Raised if the UniProt request fails, or - when safe is True - if the protein cannot be annotated.

annotate_proteome_with_uniprot(proteome, domains=True, regions=True, motifs=True, repeats=True, zinc_fingers=True, coiled_coils=True, compositional_bias=True, dna_binding=True, transmembrane=True, molecule_processing=False, secondary_structure=False, binding_sites=True, active_sites=True, other_sites=True, modified_residues=True, glycosylation=True, lipidation=True, mutagenesis=False, natural_variants=False, disulfide_bonds=True, cross_links=True, experimental_structures=False, secondary_structure_track=False, structure_coverage_track=False, accession_from_protein=None, verify_sequence=True, prefix='uniprot_', safe=True, autoname=True, verbose=True, base_url='https://rest.uniprot.org/uniprotkb', timeout=30, n_retries=3, chunk_size=100)[source]

Function that annotates every Protein in a Proteome with annotations pulled live from the UniProt REST API.

This is the batch equivalent of annotate_protein_with_uniprot(). Rather than making one API call per protein, accessions are requested in batches (of chunk_size), so annotating a proteome of a few thousand proteins is a few tens of API calls. The records that come back are then processed exactly as they are for a single protein, so the two functions always give the same answer.

By default the unique_ID of each Protein is used as its UniProt accession, which is why we recommend using UniProt accessions as unique_IDs. If your unique_IDs are something else, pass a function via accession_from_protein.

Which annotations are retrieved is controlled by the boolean flags below. Only the data needed for the requested annotations is downloaded, so turning flags off makes the query faster and the response smaller.

UniProt features are converted to SHEPHARD annotations as follows:

Features that span a region become Domains
Features on a single residue become Sites
Disulfide bonds and cross-links become a pair of Sites, one on each bonded residue, each recording its partner’s position
Optionally, secondary structure and experimental structure coverage become Tracks

Every annotation carries an attribute dictionary with the UniProt feature type, its description, its evidence codes, any cross-references (e.g. the ChEBI ID of a bound ligand), the UniProt feature ID where one exists, and the accession the annotation came from.

Parameters:

proteome (shephard.proteome.Proteome) – Proteome object to be annotated.
domains (bool (default = True)) – Annotate UniProt ‘Domain’ features.
regions (bool (default = True)) – Annotate ‘Region’ features - the free-text regions of interest that cover things like interaction interfaces.
motifs (bool (default = True)) – Annotate ‘Motif’ features.
repeats (bool (default = True)) – Annotate ‘Repeat’ features.
zinc_fingers (bool (default = True)) – Annotate ‘Zinc finger’ features.
coiled_coils (bool (default = True)) – Annotate ‘Coiled coil’ features.
compositional_bias (bool (default = True)) – Annotate ‘Compositional bias’ features.
dna_binding (bool (default = True)) – Annotate ‘DNA binding’ features.
transmembrane (bool (default = True)) – Annotate ‘Transmembrane’, ‘Intramembrane’ and ‘Topological domain’ features.
molecule_processing (bool (default = False)) – Annotate ‘Chain’, ‘Peptide’, ‘Propeptide’, ‘Signal’, ‘Transit peptide’ and ‘Initiator methionine’ features. Off by default because the ‘Chain’ feature usually spans the entire protein.
secondary_structure (bool (default = False)) – Annotate ‘Helix’, ‘Beta strand’ and ‘Turn’ features as Domains. Note that for most purposes secondary_structure_track is more useful.
binding_sites (bool (default = True)) – Annotate ‘Binding site’ features.
active_sites (bool (default = True)) – Annotate ‘Active site’ features.
other_sites (bool (default = True)) – Annotate general ‘Site’ features.
modified_residues (bool (default = True)) – Annotate ‘Modified residue’ features (phosphorylation, acetylation and so on).
glycosylation (bool (default = True)) – Annotate ‘Glycosylation’ features.
lipidation (bool (default = True)) – Annotate ‘Lipidation’ features.
mutagenesis (bool (default = False)) – Annotate ‘Mutagenesis’ features. Off by default as these describe experiments rather than the native protein.
natural_variants (bool (default = False)) – Annotate ‘Natural variant’ features. Off by default because well-studied proteins can carry thousands of these.
disulfide_bonds (bool (default = True)) – Annotate ‘Disulfide bond’ features as paired Sites.
cross_links (bool (default = True)) – Annotate ‘Cross-link’ features as paired Sites.
experimental_structures (bool (default = False)) – Add a Domain for every region of the protein covered by an experimentally-determined structure, taken from the PDB cross-references. Each Domain records the PDB identifier, the experimental method, the resolution and the chains.
secondary_structure_track (bool (default = False)) – Add a symbols Track of the UniProt secondary structure, where each residue is ‘H’ (helix), ‘E’ (beta strand), ‘T’ (turn) or ‘-‘.
structure_coverage_track (bool (default = False)) – Add a values Track where each residue is 1 if it is covered by an experimental structure and 0 if it is not.
accession_from_protein (function (default = None)) – Optional function that takes a Protein and returns the UniProt accession to look it up by. If None the Protein’s unique_ID is used.
verify_sequence (bool (default = True)) – If True, the local sequence is checked against the UniProt sequence and proteins whose sequences differ are not annotated. Strongly recommended - UniProt annotations are positional, so applying them to a different sequence gives wrong answers rather than no answer.
prefix (str (default = ‘uniprot’)) – Prefix applied to every generated domain_type, site_type and Track name.
safe (bool (default = True)) – If True, any problem (an accession that is missing from UniProt, a sequence mismatch, an annotation that falls outside the protein, a clash with an existing annotation) raises an exception. If False these are skipped and reported in the returned summary.
autoname (bool (default = True)) – Passed through to Protein.add_domain(). Recommended to leave True, since UniProt regularly reports several distinct features that share the same coordinates.
verbose (bool (default = True)) – Whether to print warnings about skipped annotations.
base_url (str (default = UNIPROT_REST_URL)) – Base URL of the UniProtKB REST API.
timeout (int or float (default = 30)) – Per-request timeout in seconds.
n_retries (int (default = 3)) – Number of times a transient failure (rate limiting, a 5xx response, a network blip) is retried before giving up.
chunk_size (int (default = 100)) – Number of accessions requested per API call.

Returns:

Returns a summary dictionary keyed by unique_ID, where each value is a dictionary with the counts of ‘domains’, ‘sites’ and ‘tracks’ added and of ‘skipped’ annotations. Proteins that could not be annotated at all appear with an ‘error’ key explaining why.

Return type:

dict

Raises:

shephard.exceptions.APIException – Raised if the UniProt request fails, or - when safe is True - if any protein cannot be annotated.

uniprot_annotation_groups()[source]

Function that returns the names of every UniProt annotation group that can be requested by annotate_protein_with_uniprot() and annotate_proteome_with_uniprot().

Returns:: Alphabetically sorted list of the valid annotation group names.
Return type:: list of str

is_valid_uniprot_accession(accession)[source]

Function that checks if a passed string looks like a valid UniProt accession. Note this checks the FORMAT only - it makes no statement as to whether the accession actually exists in UniProt.

Parameters:: accession (str) – String to be checked.
Returns:: Returns True if the passed string is a validly-formatted UniProt accession (with or without an isoform suffix), else False.
Return type:: bool

metapredict_api

The metapredict_api module provides tools for annotating proteome-scale information using metapredict. This depends on having metapredict available (it is not a hard requirement of SHEPHARD) but, if used, enables the annotation of entire proteomes with IDRs and per-residue disorder scores in a single function call.

annotate_proteome_with_disorder_track(proteome, name='disorder', device=None, version=3, show_progress_bar=True, safe=True)[source]

Function that annotates a proteome with disorder Tracks for every protein.

By default, disorder Tracks are named ‘disorder’, although this can be changed by setting the name parameter.

Disorder prediction uses the batch mode in metapredict, which leverages parallel predictions automatically on GPUs or CPUs. However, if a specific device is requested, this can be passed

Parameters:

proteome (shephard.proteome.Proteome) – Proteome object to be annotated.
name (str) – Name of the Track added to each Protein. Default = ‘disorder’
device (int or str) – Identifier for the device to be used for predictions. Possible inputs: ‘cpu’, ‘mps’, ‘cuda’, or an int that corresponds to the index of a specific cuda-enabled GPU. If ‘cuda’ is specified and cuda.is_available() returns False, instead of falling back to CPU, metapredict will raise an Exception so you know that you are not using CUDA as you were expecting. The default is None, which means we check if there is a cuda-enabled GPU and, if there is, try to use it. If an int is passed we use cuda:<int> as the device, where the GPU numbering is 0-indexed (so 0 corresponds to the first GPU). Only set this if you know which GPU you want to use. Note that MPS is only supported in PyTorch 2.1 or later, and is still fairly new, so use it at your own risk.
version (int) – Defines the metapredict version to use (must be one of 1, 2 or 3).
show_progress_bar (bool) – Flag which, if set to True, means a progress bar is printed as predictions are made, while if False no progress bar is printed. Default = True
safe (bool) – Flag which, if set to False, means the function overwrites existing tracks and domains if present. If True, overwriting will trigger an exception. Default = True.

Returns:

No return type, but the Protein objects in the Proteome will be annotated with per-residue disorder Tracks.

Return type:

None

annotate_proteome_with_disordered_domains(proteome, name='IDR', disorder_threshold=None, annotate_folded_domains=False, folded_domain_name='FD', device=None, version=3, show_progress_bar=True, safe=True)[source]

Function that annotates a proteome with disordered Domains (IDRs) for every protein.

By default, disordered Domains are named as ‘IDR’s, although this can be changed by setting the name parameter.

In addition, if requested, folded domains can also be annotated as those domains which are not IDRs. These folded domains are named ‘FD’s by default, although this can be changed by setting the folded_domain_name parameter.

Disorder prediction uses the batch mode in metapredict, which leverages parallel predictions automatically on GPUs or CPUs. However, if a specific device is requested this can be passed

Parameters:

proteome (shephard.proteome.Proteome) – Proteome object to be annotated.
name (str) – Name to give IDR domains.
disorder_threshold (float) – Threshold to be used to define IDRs by the metapredict domain decomposition algorithm. If set to None (default) metapredict uses the threshold appropriate for the version being used, which is what we strongly recommend.
annotate_folded_domains (bool) – Flag which, if included, means we ALSO annotate the regions that are not IDRs as ‘FD’ (folded domains), where the name can be changed using the folded_domain_name variable. Default = False
folded_domain_name (str) – String used to name Folded Domains. Only relevant if annotate_folded_domains is set to True. Default = ‘FD’
device (int or str) – Identifier for the device to be used for predictions. Possible inputs: ‘cpu’, ‘mps’, ‘cuda’, or an int that corresponds to the index of a specific cuda-enabled GPU. If ‘cuda’ is specified and cuda.is_available() returns False, instead of falling back to CPU, metapredict will raise an Exception so you know that you are not using CUDA as you were expecting. The default is None, which means we check if there is a cuda-enabled GPU and, if there is, try to use it. If an int is passed we use cuda:<int> as the device, where the GPU numbering is 0-indexed (so 0 corresponds to the first GPU). Only set this if you know which GPU you want to use. Note that MPS is only supported in PyTorch 2.1 or later, and is still fairly new, so use it at your own risk.
version (int) – Defines the metapredict version to use (must be one of 1, 2 or 3).
show_progress_bar (bool) – Flag which, if set to True, means a progress bar is printed as predictions are made, while if False no progress bar is printed. Default = True
safe (bool) – Flag which, if set to False, means the function overwrites existing tracks and domains if present. If True, overwriting will trigger an exception. Default = True.

Returns:

No return type, but the Protein objects in the Proteome will be annotated with disordered Domain annotations.

Return type:

None

annotate_proteome_with_disorder_tracks_and_disordered_domains(proteome, track_name='disorder', domain_name='IDR', disorder_threshold=None, annotate_folded_domains=False, folded_domain_name='FD', device=None, version=3, show_progress_bar=True, safe=True)[source]

Function that annotates a proteome with disorder Tracks and disorder Domains for every protein.

By default, disorder Tracks are named ‘disoder’, although this can be changed by setting the track_name parameter.

By default, disordered Domains are named as ‘IDR’s, although this can be changed by setting the name parameter.

In addition, if requested, folded domains can also be annotated as those domains which are not IDRs. These folded domains are named ‘FD’s by default, although this can be changed by setting the folded_domain_name parameter.

Disorder prediction uses the batch mode in metapredict, which leverages parallel predictions automatically on GPUs or CPUs. However, if a specific device is requested this can be passed

Parameters:

proteome (shephard.proteome.Proteome) – Proteome object to be annotated.
track_name (str) – Name of the Track added to each Protein. Default = ‘disorder’
domain_name (str) – Name of the Domain added to each Protein. Default = ‘IDR’
disorder_threshold (float) – Threshold to be used to define IDRs by the metapredict domain decomposition algorithm. If set to None (default) metapredict uses the threshold appropriate for the version being used, which is what we strongly recommend.
annotate_folded_domains (bool) – Flag which, if included, means we ALSO annotate the regions that are not IDRs as ‘FD’ (folded domains), where the name can be changed using the folded_domain_name variable. Default = False
folded_domain_name (str) – String used to name Folded Domains. Only relevant if annotate_folded_domains is set to True. Default = ‘FD’
device (int or str) – Identifier for the device to be used for predictions. Possible inputs: ‘cpu’, ‘mps’, ‘cuda’, or an int that corresponds to the index of a specific cuda-enabled GPU. If ‘cuda’ is specified and cuda.is_available() returns False, instead of falling back to CPU, metapredict will raise an Exception so you know that you are not using CUDA as you were expecting. The default is None, which means we check if there is a cuda-enabled GPU and, if there is, try to use it. If an int is passed we use cuda:<int> as the device, where the GPU numbering is 0-indexed (so 0 corresponds to the first GPU). Only set this if you know which GPU you want to use. Note that MPS is only supported in PyTorch 2.1 or later, and is still fairly new, so use it at your own risk.
version (int) – Defines the metapredict version to use (must be one of 1, 2 or 3).
show_progress_bar (bool) – Flag which, if set to True, means a progress bar is printed as predictions are made, while if False no progress bar is printed. Default = True
safe (bool) – Flag which, if set to False, means the function overwrites existing tracks and domains if present. If True, overwriting will trigger an exception. Default = True.

Returns:

No return type, but the Protein objects in the Proteome will be annotated with per-residue disorder Tracks and disordered Domain annotations.

Return type:

None

albatross_api

The ALBATROSS API interfaces with the radius of gyration (Rg) and end-to-end distance (Re) predictions provided by ALBATROSS (Lotthammer et al., Nat. Methods, 2024). This depends on sparrow being installed, but enables you to annotate sequences at either a protein level or a domain level with predicted Rg and Re values.

annotate_proteome_with_dimensions(proteome, rg_name='rg', re_name='re', gpuid=0, show_progress_bar=True, batch_mode=None, safe=True)[source]

Function that annotates a proteome with it’s predicted radius of gyration (rg) and end-to-end distance (re) for every protein.

By default, rg and re are added as attributes to each Protein, with the names ‘rg’ and ‘re’ respectively. However, this can be changed by setting the rg_name and re_name parameters.

Dimension prediction uses the batch mode in sparrow, which leverages parallel predictions automatically on GPUs or CPUs. However, if a specific device is requested, this can be passed via the gpuid parameter.

Parameters:

proteome (shephard.proteome.Proteome) – Proteome object to be annotated.
rg_name (str) – Name of the rg attribute added to each Protein.
re_name (str) – Name of the re attribute added to each Protein.
gpuid (int) – Identifier for the GPU being requested. Note that if this is left unset the code will use the first GPU available and if none is available will default back to CPU; in general, it is recommended not to try and set this unless there’s a specific reason why a specific GPU should be used. Default = 0.
show_progress_bar (bool) – Flag which, if set to True, means a progress bar is printed as predictions are made, while if False no progress bar is printed. Default = True
batch_mode (None) – Deprecated and ignored - sparrow always uses batch mode here. Retained so existing code that passes this keyword continues to work.
safe (bool) – Flag which, if set to False, means the function overwrites existing tracks and domains if present. If True, overwriting will trigger an exception. Default = True.

Returns:

No return type, but the Protein objects in the Proteome will be annotated with per-residue disorder Tracks.

Return type:

None

annotate_domains_with_dimensions(proteome, domain_type, rg_name='rg', re_name='re', gpuid=0, show_progress_bar=True, batch_mode=None, safe=True)[source]

Function that annotates every domain matching the domain_name in a proteome with it’s predicted radius of gyration (rg) and end-to-end distance (re).

By default, rg and re are added as attributes to each Domain, with the names ‘rg’ and ‘re’ respectively. However, this can be changed by setting the rg_name and re_name parameters.

Dimension prediction uses the batch mode in sparrow, which leverages parallel predictions automatically on GPUs or CPUs. However, if a specific device is requested, this can be passed via the gpuid parameter.

Parameters:

proteome (shephard.proteome.Proteome) – Proteome object to be annotated.
domain_type (str) – Type of the domain to be annotated.
rg_name (str) – Name of the rg attribute added to each Protein.
re_name (str) – Name of the re attribute added to each Protein.
gpuid (int) – Identifier for the GPU being requested. Note that if this is left unset the code will use the first GPU available and if none is available will default back to CPU; in general, it is recommended not to try and set this unless there’s a specific reason why a specific GPU should be used. Default = 0.
show_progress_bar (bool) – Flag which, if set to True, means a progress bar is printed as predictions are made, while if False no progress bar is printed. Default = True
batch_mode (None) – Deprecated and ignored - sparrow always uses batch mode here. Retained so existing code that passes this keyword continues to work.
safe (bool) – Flag which, if set to False, means the function overwrites existing tracks and domains if present. If True, overwriting will trigger an exception. Default = True.

Returns:

No return type, but the Protein objects in the Proteome will be annotated with per-residue disorder Tracks.

Return type:

None