Globals

Downloads the webpage or file at url to the location loc.

Parameters:

• url (str) – A url pointing to a file to be downloaded.

• loc (str) – A folder to download to; leave blank for the current working directory. If the folder doesn’t exist, it will be created automatically.

• name (str) – If specified, the file will be saved with this name instead of an automatically generated one.

• kwargs (**) – Added in 1.2: A collection of arguments to request with. Same format as requests.get(), but without the url parameter.

Downloads a list of files with their website folder structure to the location loc.

For those that are familiar with cURLsDownloader, this function acts in a similar way. Invalid characters for a file/folder name will be replaced with “_”, but “://” will be replaced with “.”.

Parameters:

• urls ([str|(str,dict),....]) – A list of urls pointing to files to be downloaded. Since 1.2, if a url is a tuple instead of a string, the first item in the tuple will be read as the url and the second will be read a dictionary of arguments to request with. The dictionary of arguments is in a similar format to the arguments requests.get() uses, but without the url parameter.

• loc (str) – A folder to download to; leave blank for the current working directory. If the folder doesn’t exist, it will be created automatically.

• preserve (bool) – If True, any files from “web.archive.org” will stay in the “web.archive.org” folder. By default, files are moved to their original domains with any port numbers removed.

• keep_vars (bool) – If True, files will be saved with all urlvars intact in their file name. This prevents files such as “http://www.example.com/game.php?id=123” and “http://www.example.com/game.php?id=321” from overwriting each other. Question marks are replaced with “_”.

• ignore_errs (bool) – If True, any errors will be ignored and returned as part of a tuple including the urls that failed alongside each error.

Returns:

None or a list of tuples including failed urls and errors.

Downloads the image from url to the location loc as a PNG file.

Parameters:

• url (str) – A url pointing to the image to be downloaded.

• loc (str) – A folder to download to; leave blank for the current working directory. If the folder doesn’t exist, it will be created automatically.

• name (str) – If specified, the file will be saved with this name instead of an automatically generated one.

• kwargs (**) – Added in 1.2: A collection of arguments to request with. Same format as requests.get(), but without the url parameter.

Formats url to a normalized format.

This involves making it use the http protocol, fixing escaped slashes (/), removing url vars, and stripping it. Accepts strings starting with any protocol, no protocol, “//”, or “/”.

Parameters:

• url (str) – A string url to format.

• preserve (bool) – If False and the url is from “web.archive.org”, the url will be formatted to it’s original domain without the “web.archive.org” prefix.

• keep_vars (bool) – If True, url vars at the end of the string will stay on the string.

• keep_prot (bool) – If True, the url protocol will not be made to use the http protocol unless there is no protocol given or the protocol is formatted incorrectly.

Returns:

url in a normalized format, or None if url is None.

Reads the webpage or file at url and returns its contents as a text string.

Parameters:

• url (str) – A string url of a webpage or file.

• ignore_errs (bool) – Added in 1.3: If False, instead of returning None on any errors, those errors will be raised.

• content (bool) – Added in 1.3 If True, this function will return the contents of the webpage or file as a byte string instead of reading it as text (response.content instead of response.text).

• kwargs (**) – Added in 1.2: A collection of arguments to request with. Same format as requests.get(), but without the url parameter.

Returns:

The contents of the webpage at url as a string or None if the url is blank.

Reads the webpage at url and creates a BeautifulSoup object with it.

Parameters:

• url (str) – A string url of a webpage.

• parser (str) – The BeautifulSoup parser to use.

• ignore_errs (bool) – Added in 1.3: If False, instead of returning None on any errors, those errors will be raised.

• kwargs (**) – Added in 1.2: A collection of arguments to request with. Same format as requests.get(), but without the url parameter.

Returns:

A BeautifulSoup object created from url or None if the url is blank or there was an error and ignore_errs is True.

Reads the Flashpoint online datahub at https://flashpointarchive.org/datahub/{page} and returns a list of possible values; you can use this to get the most up-to-date tags and platforms. It will automatically check the web archive if the given wiki page is down.

Parameters:

• page (str) – Can be “Platforms”, “Tags”, or any other page on the datahub with tables in it.

• ignore_errs (bool) – If False, instead of returning None on any errors, those errors will be raised.

Returns:

A list of all Platforms, Tags, etc. depending on each page, or an empty list on failure.

Since 1.3:

Reads the contents of the file file_name into a string.

The returned string is in utf-8.

Parameters:

file_name (str) – The location/name of a file in the local file system.

Returns:

The contents of the file file_name as a string.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

• InvalidFileError – If file_name is not a file.

• FileNotFoundError – If file_name cannot be found.

Reads the lines of the file file_name into a list of strings split by any new line character (rn, r, or n).

The returned strings are in utf-8.

Parameters:

• file_name (str) – The location/name of a file in the local file system.

• ignore_lines (bool) – By default this function disregards empty lines at the end of the file. Set this to False to disable that.

Returns:

A list of strings containing all of the lines in the file file_name.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

• InvalidFileError – If file_name is not a file.

• FileNotFoundError – If file_name cannot be found.

Reads the contents of the file file_name into a two dimensional list of strings with rows split by any new line character (rn, r, or n) and columns split by delimiter.

The returned strings are in utf-8.

Note:

This function disregards empty lines at the end of the file.

Parameters:

• file_name (str) – The location/name of a file in the local file system.

• delimiter (str) – A string to split columns in the table by.

• ignore_lines (bool) – By default this function disregards empty lines at the end of the file. Set this to False to disable that.

Returns:

A two-dimensional list of strings of the content in the file file_name.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

• InvalidFileError – If file_name is not a file.

• FileNotFoundError – If file_name cannot be found.

Makes a folder at folder, along will all parent folders.

It will not raise a FileExistsError if the folder already exists, but will instead return False.

Parameters:

• folder (str) – A string location to create a folder.

• change (bool) – If True, this function will set the new folder as the working directory.

• overwrite (bool) – If True, if there is a non-folder file in the same location as the folder being created, the folder will overwrite this file.

Raises:

• EmptyLocationError – If folder is an empty string.

• InvalidCharacterError – If folder contains invalid characters.

• FileExistsError – If a non-folder file exists in the location given by folder and overwrite is not set.

Returns:

True on successful folder creation, False on failure.

Note:

Even if folder creation fails, if change is True, the working directory will still be changed to that folder.

Deletes the file or folder file_name recursively.

It will not raise a FileNotFoundError if the file or folder doesn’t exist, but will instead return False.

Parameters:

file_name (str) – A string location of a file or folder to delete.

Returns:

True on successful deletion, False on failure.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

Writes contents to the file file_name in utf-8.

If the parent directory of this file doesn’t exist, it will be automatically created.

Parameters:

• file_name (str) – The location/name of a file in the local file system.

• contents (str|[str,....]) – Contents to write to the file file_name. If this is an Iterable (and not a string), each item in this will be written to the file file_name as a separate line split by line feed (n) characters.

• force (bool) – If True, if there is a non-writable file (like a directory) in the same location as the file being written to, the function will overwrite this file.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

• InvalidFileError – If a non-writable file exists in the location given by file_name and force is not set.

Append line to the file file_name.

If the parent directory of this file doesn’t exist, it will be automatically created.

A line feed (n) character is written after line.

Parameters:

• file_name (str) – The location/name of a file in the local file system.

• line (str) – A line to write to the file file_name.

• force (bool) – If True, if there is a non-writable file (like a directory) in the same location as the file being written to, the function will overwrite this file.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

• InvalidFileError – If a non-writable file exists in the location given by file_name and force is not set.

Writes the two-dimensional list table to the file file_name.

If the parent directory of this file doesn’t exist, it will be automatically created.

Rows are joined by a line feed (n) character, and columns are joined by delimiter.

Parameters:

• file_name (str) – The location/name of a file in the local file system.

• table ([[str,....],....]) – A two-dimensional list of strings to format and write to the file file_name.

• delimiter (str) – A string to join columns in the table by.

• force (bool) – If True, if there is a non-writable file (like a directory) in the same location as the file being written to, the function will overwrite this file.

Raises:

• EmptyLocationError – If file_name is an empty string.

• InvalidCharacterError – If file_name contains invalid characters.

• InvalidFileError – If a non-writable file exists in the location given by file_name and force is not set.

Scans the directory folder recursively and returns all files and sub-folders as two lists.

Parameters:

• folder (str) – A directory to scan. Defaults to the current working directory.

• regex (str|re) – If given, this function will only return files whose full paths match this regex. This has no effect on sub-folders.

• recursive (bool) – If False, this function will not scan sub-folders recursively. You might be better off using os.walk() if you set this to False.

Returns:

files and subfolders as two lists in that order. return files, subfolders

Raises:

• InvalidCharacterError – If folder contains invalid characters.

• FileExistsError – If folder is not a folder.

• FileNotFoundError – If folder does not exist.

Note:

Any slashes (/) in file paths will be replaced with backslashes (\) in the output, and this is what regexes match with.

Since 1.4:

Replaces all instances of old with new in files.

Parameters:

• files ([str,....]|str) – A string or list of strings pointing to files to have their contents changed.

• old (str|re) – A regex or string to replace. If this is a string, it will not be treated as a regex. Use re.compile() to pass in a regex object.

• new (str) – A string of text to replace with.

• regex (str|re) – If given, this function will only replace in files whose full paths match this regex. It must be a real regex (i.e., you can’t use “*” in place of “.*”)!

• ignore_errs (bool) – If False, this function will throw an error at the end if files had invalid files.

Raises:

A bulk ValueError if files has any invalid files and ignore_errs is False. Errors will be thrown at the end of the function after everything that could be replaced was replaced.

Since 1.4:

Serializes obj and then returns a sha256 HASH object of it.

Two objects that yield the same bytes representation will yield the same hash.

Parameters:

obj – An object to hash.

Returns:

A sha256 HASH object of obj.

Seealso:

Python’s hashlib library. The object returned is the same object created by hashlib.sha256().

Serializes obj and then gets a four-byte integer representation of those bytes with the first four bytes of it’s sha256.

This function is meant to replace the default python hash() function because it does not rely on the object’s id.

Parameters:

obj – An object to hash.

Returns:

A four-byte integer representation of obj.

Seealso:

hash256()

Tests the library to make sure it works.

This test will curate “Interactive Buddy” and download the swf file linked in the launch command into the proper place in a folder in the working directory, while also testing basic validation.

Raises:

AssertionError – If the test gives back any errors.

Initializes or updates variables that are generated from online. The only ones are PLATFORMS and TAGS.

This method is only automatically called by Curation.validate() if rigid is True and either PLATFORMS or TAGS are not set. You must call this function manually if you want to access PLATFORMS or TAGS before then.

Since 1.3:

Prints text depending on mode and DEBUG_LEVEL.

Parameters:

• text (str) – A string of text to print.

• mode (int) – text will only be printed in two cases: if this is 1 and DEBUG_LEVEL > 0 - or if this is 2, DEBUG_LEVEL is 2 or higher, and DEBUG_LEVEL - 1 > TABULATION.

• items (*) – Any number of items to format text with. Uses str.format().

• pre (str) – This is a prefix of text to print to the left of text. This must be a key argument.

Since 1.3:

Deletes the “c-info.tmp” file generated by curate() and curate_regex() in the current working directory or loc.

Parameters:

loc (str) – Added in 1.3: An optional location to clear the save file in. If not set, the directory will be the current working directory.

It will not raise a FileNotFoundError if it doesn’t exist. Returns True on success, and False on failure.

Curates form a list of urls given by items with a sub-class of Curation specified by curation_class.

Parameters:

• [str|(str,dict),....] (items) – normally a list of string urls of webpages to curate from, but if you put a tuple with 2 items in the place of any string, the first item in the tuple will be treated as the url, and the second item will be treated as a dictionary of arguments passed to an instance of curation_class along with the url. You can mix tuples and strings.

• curation_class (class) – A sub-class of Curation to create curations from. Note that this must be the class and not an object created from the class.

• use_title (bool) – If True, each curation folder will be generated with the title of the curation instead of its id.

• save (bool) – If True, progress will be constantly saved to “c-info.tmp” so that if the function errors and is ever called with the same items variable in the same working directory again, it will resume where it left off. Note that calling this function again with a different list of items will restart the process and the save file from any other processes.

• ignore_errs (bool) – If True, any error that a curation raises will be ignored and the curation will be skipped. Any failed items will be returned as a list of 3-length tuples at the end of the function with the item, the error, and a dictionary of additional arguments that were passed in.

• overwrite (bool) – If True, this method will mix and overwrite files in existing curation folders instead of making the folder “Curation (2)”, “Curation (3)”, etc.

• validate (int) – Added in 1.3: Mode to validate each curation to make sure it has proper metadata. 0 means do not validate, 1 (default) means flexibly validate, and 2 means rigidly validate. See Curation.get_errors(). Invalid curations will not raise any errors and will be skipped; however, if ignore_errs is True, the curation will be returned with an InvalidMetadataError in the list at the end of the function.

Returns:

None or a list of tuples including failed urls, errors, and data passed in. The format for this is [(str,Exception,dict),…]

Raises:

• ValueError – If items is empty.

• TypeError – Added in 1.3: If curation_class is not a subclass of Curation.

Curates from a list of urls given by items with a list of links.

See:

curate()

Parameters:

• items ([str|(str,dict),....]) – A list of urls of webpages with games/animations to curate.

• links ([(str|re,class),....]) – A list of tuples containing a regex and a sub-class of Curation. The function will search the url of each item for each of the regexes using re.search(), and the first match will decide which sub-class gets used. If there are no matches, the curation would be skipped. A good example of the links variable is something like [('newgrounds.com', NewgroundsCuration), ('kongregate.com', KongregateCuration)]. Regexes can be strings or a re object.

• use_title (bool) – Specifies whether or not to use the title or id of a curation for its folder.

• save (bool) – Specifies whether or not to save progress and continue where left off if the function is called with the same arguments.

• ignore_errs (bool) – Specifies whether or not to ignore errors and return them at the end of the function.

• overwrite (bool) – Whether or not to mix new curations with older folders with the same name.

• validate (int) – Added in 1.3: Mode to validate each curation to make sure it has proper metadata.

Returns:

None or a list of tuples including failed urls, errors, and data passed in. The format for this is [(str,Exception,dict),…]

Raises:

ValueError – If items or links is empty.

Loads the curation in the folder curation into Curation object using ruamel.yaml.

Parameters:

curation (str) – A string pointing to the location of a folder where a curation is stored.

Returns:

A Curation object created with the metadata of the curation in the folder curation.

Note:

The curation at curation curation must be using the yaml curation format.

Raises:

• InvalidCharacterError – If curation has invalid characters.

• FileNotFoundError – If the folder given is not a valid curation.