qq et l'API REST Qumulo

Which returns content like this (a few thousand lines of output elided here….):

{
    "child_count": 131,
    "files": [
    (...)
        {
            "blocks": "1",
            "change_time": "2017-01-04T22:59:52.909017431Z",
            "child_count": 30,
            "creation_time": "2016-03-15T20:12:07.722559693Z",
            "datablocks": "0",
            "file_number": "3",
            "group": "17179869184",
            "group_details": {
                "id_type": "NFS_GID",
                "id_value": "0"
            },
            "id": "3",
            "metablocks": "1",
            "mode": "0777",
            "modification_time": "2017-01-04T22:59:52.909017431Z",
            "name": "users",
            "num_links": 30,
            "owner": "12884901888",
            "owner_details": {
                "id_type": "NFS_UID",
                "id_value": "0"
            },
            "path": "/users/",
            "size": "15360",
            "symlink_target_type": "FS_FILE_TYPE_UNKNOWN",
            "type": "FS_FILE_TYPE_DIRECTORY"
        }
    ],
    "id": "2",
    "paging": {
        "next": "",
        "prev": "/v1/files/%2F/entries/?before=4190957568&limit=16"
    },
    "path": "/"
}

This output illustrates a few things about qq and the Qumulo REST API:

1. The output from qq commands is JSON format
2. For some qq commands, the output can be very verbose

The unix philosophy is to “write programs to handle text streams”, and that is also the foundation of shell scripting in unix. While JSON format is text, it is more common to pipe either comma- or tab-separated values to subsequent commands. So to tailor the output from qq commands to be more terse and output different formats, for this blog post we will use the popular jq command-line JSON processor to illustrate how you can make qq fit in more readily with other CLI tools (on a Mac, you can install jq using homebrew; for linux, Windows and other flavors see the tool website).

Using the above example, let’s grab some of the output from qq and use jq to make it easier to read and repurpose in other tools (there is a tutorial for jq that shows you some of what you can do):

qq --host music fs_read_dir --path /media | jq '[.path, .id, .files[0].blocks] | @csv'

This will return the path, id and blocks value from the first entry returned by fs_read_dir in csv format, like this:

"\"/media/\",\"1000003\",\"1\""

Which could then be piped to other tools that expect CSV format. Note that you can also create files and directories using qq and the REST API, but the more common way is to mount shares using NFS or SMB and create content on the cluster that way.

Information about your Cluster, and viewing and creating Users, Groups and NFS or SMB Shares

Let’s say you want to retrieve the capacity and state of all of the slots (disks) in your cluster, order the results by slot, and then store the results in a csv file as part of an IT status report. You could run:

qq --host music cluster_slots | jq '[ .[] | {"cap":.capacity, "slot":.slot_type, "state":.state }] | sort'

You can create groups:

qq --host music auth_add_group --name "My Test Group"

{
    "gid": "",
    "id": "1026",
    "name": "My Test Group",
    "sid": "S-1-5-21-1203876804-525990034-1406135336-1026"
}
Expanded identity information for group 1026: [
    {
        "id_type": "LOCAL_GROUP",
        "id_value": "My Test Group"
    }
]

And add users to the group:

qq --host music auth_add_user --name "My_Test_User" --primary-group "My Test Group" -p junk

{
    "id": "1027",
    "name": "My_Test_User",
    "primary_group": "1026",
    "sid": "S-1-5-21-1203876804-525990034-1406135336-1027",
    "uid": ""
}
Expanded identity information for user 1027: [
    {
        "id_type": "LOCAL_USER",
        "id_value": "My_Test_User"
    },
    {
        "id_type": "LOCAL_GROUP",
        "id_value": "My Test Group"
    }
]

And then create a directory and add a share for the new group:

qq --host music fs_create_dir --name "my_test" --path "/"

With:

qq --host music smb_add_share --name "my_test_smb_share" --fs-path "/my_test/"

Or if you’re using NFS:

qq --host music nfs_add_share --fs-path "/my_test/" --export-path "/my_nfs_export" --no-restrictions

Performance: Retrieving Analytics and Aggregate Data

One of the most powerful aspects of the Qumulo REST API are the real-time analytics and aggregate data about the filesystem exposed via our REST API. Using qq this data is exposed to users using bash scripts as well.

Let’s say you want to retrieve all current activity in the cluster and display it in tabular form similar to iostat and other tools. To get current activity you can just run:

qq --host music current_activity_get | jq '.entries | sort_by(.ip)'

Like all qq commands this will return JSON-based data -- in this case showing current cluster activity such as file and metadata throughput and IOPs -- grouped by client IP address:

This data can then be piped to other tools such as monitoring tools.

Further Reading and tools

There is a good tutorial on JSON and the jq tool here.

If you are currently a Qumulo customer, you can find more information on the Qumulo Community site (search for ‘qq’ in conversations).

Test drive Qumulo for free

Explore a fully functional Qumulo environment, right in your browser.

Try Demo