qq et l'API REST Qumulo

Qumulo fournit une nouvelle version de notre logiciel Qumulo Core environ toutes les deux semaines et, avec chaque version de Qumulo Core, nous publions une nouvelle version de l’API REST Qumulo, ainsi Wrapper Python comme commodité pour les développeurs de langage Python. Ceci est très utile pour les utilisateurs de langages de programmation de haut niveau tels que Python ou tout autre langage de programmation pouvant effectuer des appels REST directement.

Mais qu'en est-il des administrateurs système et des autres utilisateurs qui utilisent les scripts bash / shell pour la plupart de leurs tâches? Comment peuvent-ils tirer parti de la puissance de l’API REST Qumulo? Pour ces utilisateurs, l'outil de ligne de commande qq fourni par Qumulo est le moyen d'exploiter la puissance de l'API REST de Qumulo à partir de l'interface de ligne de commande et des scripts shell. L'intégralité de l'API REST de Qumulo est exposée via qq.

Cet article vous montrera comment exploiter les informations de votre cluster Qumulo via qq directement dans le shell ou dans les scripts bash, avec quelques exemples montrant comment intégrer la sortie de qq dans d'autres scripts de gestion.

Installer qq

Le moyen le plus simple d'installer qq consiste à installer l'API REST Qumulo à partir de PyPI (le référentiel public d'index de package Python):

https://pypi.python.org/pypi/qumulo_api/

Pour installer, ouvrez une invite de commandes, puis exécutez:

pip install qumulo-api

You will need python 2.7 in order to run qq. If you prefer, you can also run qq locally on your cluster (via ssh) where it is pre installed.

Getting Started: Logging in and viewing cluster content

To start, you need to log in to your qumulo cluster, like so:

qq --host [hostname or IP address] login -u [username] -p [password]

Where username and password are credentials on your Qumulo cluster. On successful login, an access token is created behind the scenes, so while that token is valid, you do not need to specify username and password for subsequent qq commands.

To see all of the options from the command line, try running qq --help.

Once you’re authenticated, you can view information about cluster content…. This command lists the contents starting at the root directory:

qq --host music fs_read_dir --path /

Which returns content like this (a few thousand lines of output elided here….):

{
    "child_count": 131,
    "files": [
    (...)
        {
            "blocks": "1",
            "change_time": "2017-01-04T22:59:52.909017431Z",
            "child_count": 30,
            "creation_time": "2016-03-15T20:12:07.722559693Z",
            "datablocks": "0",
            "file_number": "3",
            "group": "17179869184",
            "group_details": {
                "id_type": "NFS_GID",
                "id_value": "0"
            },
            "id": "3",
            "metablocks": "1",
            "mode": "0777",
            "modification_time": "2017-01-04T22:59:52.909017431Z",
            "name": "users",
            "num_links": 30,
            "owner": "12884901888",
            "owner_details": {
                "id_type": "NFS_UID",
                "id_value": "0"
            },
            "path": "/users/",
            "size": "15360",
            "symlink_target_type": "FS_FILE_TYPE_UNKNOWN",
            "type": "FS_FILE_TYPE_DIRECTORY"
        }
    ],
    "id": "2",
    "paging": {
        "next": "",
        "prev": "/v1/files/%2F/entries/?before=4190957568&limit=16"
    },
    "path": "/"
}

This output illustrates a few things about qq and the Qumulo REST API:

  1. The output from qq commands is JSON format
  2. For some qq commands, the output can be very verbose

The unix philosophy is to “write programs to handle text streams”, and that is also the foundation of shell scripting in unix. While JSON format is text, it is more common to pipe either comma- or tab-separated values to subsequent commands. So to tailor the output from qq commands to be more terse and output different formats, for this blog post we will use the popular jq command-line JSON processor to illustrate how you can make qq fit in more readily with other CLI tools (on a Mac, you can install jq using homebrew; for linux, Windows and other flavors see the tool website).

Using the above example, let’s grab some of the output from qq and use jq to make it easier to read and repurpose in other tools (there is a tutorial for jq that shows you some of what you can do):

qq --host music fs_read_dir --path /media | jq '[.path, .id, .files[0].blocks] | @csv'

This will return the path, id and blocks value from the first entry returned by fs_read_dir in csv format, like this:

"\"/media/\",\"1000003\",\"1\""

Which could then be piped to other tools that expect CSV format. Note that you can also create files and directories using qq and the REST API, but the more common way is to mount shares using NFS or SMB and create content on the cluster that way.

Information about your Cluster, and viewing and creating Users, Groups and NFS or SMB Shares

Let’s say you want to retrieve the capacity and state of all of the slots (disks) in your cluster, order the results by slot, and then store the results in a csv file as part of an IT status report. You could run:

qq --host music cluster_slots | jq '[ .[] | {"cap":.capacity, "slot":.slot_type, "state":.state }] | sort'

You can create groups:

qq --host music auth_add_group --name "My Test Group"

{
    "gid": "",
    "id": "1026",
    "name": "My Test Group",
    "sid": "S-1-5-21-1203876804-525990034-1406135336-1026"
}
Expanded identity information for group 1026: [
    {
        "id_type": "LOCAL_GROUP",
        "id_value": "My Test Group"
    }
]

And add users to the group:

qq --host music auth_add_user --name "My_Test_User" --primary-group "My Test Group" -p junk

{
    "id": "1027",
    "name": "My_Test_User",
    "primary_group": "1026",
    "sid": "S-1-5-21-1203876804-525990034-1406135336-1027",
    "uid": ""
}
Expanded identity information for user 1027: [
    {
        "id_type": "LOCAL_USER",
        "id_value": "My_Test_User"
    },
    {
        "id_type": "LOCAL_GROUP",
        "id_value": "My Test Group"
    }
]

And then create a directory and add a share for the new group:

qq --host music fs_create_dir --name "my_test" --path "/"

With:

qq --host music smb_add_share --name "my_test_smb_share" --fs-path "/my_test/"

Or if you’re using NFS:

qq --host music nfs_add_share --fs-path "/my_test/" --export-path "/my_nfs_export" --no-restrictions

Performance: Retrieving Analytics and Aggregate Data

One of the most powerful aspects of the Qumulo REST API are the real-time analytics and aggregate data about the filesystem exposed via our REST API. Using qq this data is exposed to users using bash scripts as well.

Let’s say you want to retrieve all current activity in the cluster and display it in tabular form similar to iostat and other tools. To get current activity you can just run:

qq --host music current_activity_get | jq '.entries | sort_by(.ip)'

Like all qq commands this will return JSON-based data -- in this case showing current cluster activity such as file and metadata throughput and IOPs -- grouped by client IP address:

This data can then be piped to other tools such as monitoring tools.

Further Reading and tools

There is a good tutorial on JSON and the jq tool here.

If you are currently a Qumulo customer, you can find more information on the Qumulo Community site (search for ‘qq’ in conversations).

Test drive Qumulo for free

Test drive Qumulo for free

Explore a fully functional Qumulo environment, right in your browser.

Try Demo

Share this post