Operational Data, RPCs and Notifications

Operational data 

Writing API-agnostic code for YANG-modeled operational data is challenging. Sysrepo, for instance, has completely different API to fetch operational data. So how can we write API-agnostic callbacks that can be used by both the Sysrepo plugin, and any other northbound client that might be written in the future?

As an additional requirement, the callbacks must be designed in a way that makes in-place XPath filtering possible. As an example, a management client might want to retrieve only a subset of a large YANG list (e.g. a BGP table), and for optimal performance it should be possible to filter out the unwanted elements locally in the managed devices instead of returning all elements and performing the filtering on the management application.

To meet all these requirements, the four callbacks below were introduced in the northbound architecture:

/*
 * Operational data callback.
 *
 * The callback function should return the value of a specific leaf or
 * inform if a typeless value (presence containers or leafs of type
 * empty) exists or not.
 *
 * xpath
 *    YANG data path of the data we want to get
 *
 * list_entry
 *    pointer to list entry
 *
 * Returns:
 *    pointer to newly created yang_data structure, or NULL to indicate
 *    the absence of data
 */
struct yang_data *(*get_elem)(const char *xpath, void *list_entry);

/*
 * Operational data callback for YANG lists.
 *
 * The callback function should return the next entry in the list. The
 * 'list_entry' parameter will be NULL on the first invocation.
 *
 * list_entry
 *    pointer to a list entry
 *
 * Returns:
 *    pointer to the next entry in the list, or NULL to signal that the
 *    end of the list was reached
 */
void *(*get_next)(void *list_entry);

/*
 * Operational data callback for YANG lists.
 *
 * The callback function should fill the 'keys' parameter based on the
 * given list_entry.
 *
 * list_entry
 *    pointer to a list entry
 *
 * keys
 *    structure to be filled based on the attributes of the provided
 *    list entry
 *
 * Returns:
 *    NB_OK on success, NB_ERR otherwise
 */
int (*get_keys)(void *list_entry, struct yang_list_keys *keys);

/*
 * Operational data callback for YANG lists.
 *
 * The callback function should return a list entry based on the list
 * keys given as a parameter.
 *
 * keys
 *    structure containing the keys of the list entry
 *
 * Returns:
 *    a pointer to the list entry if found, or NULL if not found
 */
void *(*lookup_entry)(struct yang_list_keys *keys);

These callbacks were designed to provide maximum flexibility. Each callback does one and only one task, they are indivisible primitives that can be combined in several different ways to iterate over operational data. The extra flexibility certainly has a performance cost, but it’s the price to pay if we want to expose FRR operational data using several different management interfaces (e.g. Sysrepo+Netopeer2). In the future it might be possible to introduce optional callbacks that do things like returning multiple objects at once. They would provide enhanced performance when iterating over large lists, but their use would be limited by the northbound plugins that can be integrated with them.

The [[Plugins - Writing Your Own]] page explains how the northbound plugins can fetch operational data using the aforementioned northbound callbacks, and how in-place XPath filtering can be implemented.

Example

Now let’s move to an example to show how these callbacks are implemented in practice. The following YANG container is part of the ietf-rip module and contains operational data about RIP neighbors:

container neighbors {
  description
    "Neighbor information.";
  list neighbor {
    key "address";
    description
      "A RIP neighbor.";
    leaf address {
      type inet:ipv4-address;
      description
        "IP address that a RIP neighbor is using as its
         source address.";
    }
    leaf last-update {
      type yang:date-and-time;
      description
        "The time when the most recent RIP update was
         received from this neighbor.";
    }
    leaf bad-packets-rcvd {
      type yang:counter32;
      description
        "The number of RIP invalid packets received from
         this neighbor which were subsequently discarded
         for any reason (e.g. a version 0 packet, or an
         unknown command type).";
    }
    leaf bad-routes-rcvd {
      type yang:counter32;
      description
        "The number of routes received from this neighbor,
         in valid RIP packets, which were ignored for any
         reason (e.g. unknown address family, or invalid
         metric).";
    }
  }
}

We know that this is operational data because the neighbors container is within the state container, which has the config false; property (which is applied recursively).

As expected, the gen_northbound_callbacks tool also generates skeleton callbacks for nodes that represent operational data:

{
        .xpath = "/frr-ripd:ripd/state/neighbors/neighbor",
        .cbs.get_next = ripd_state_neighbors_neighbor_get_next,
        .cbs.get_keys = ripd_state_neighbors_neighbor_get_keys,
        .cbs.lookup_entry = ripd_state_neighbors_neighbor_lookup_entry,
},
{
        .xpath = "/frr-ripd:ripd/state/neighbors/neighbor/address",
        .cbs.get_elem = ripd_state_neighbors_neighbor_address_get_elem,
},
{
        .xpath = "/frr-ripd:ripd/state/neighbors/neighbor/last-update",
        .cbs.get_elem = ripd_state_neighbors_neighbor_last_update_get_elem,
},
{
        .xpath = "/frr-ripd:ripd/state/neighbors/neighbor/bad-packets-rcvd",
        .cbs.get_elem = ripd_state_neighbors_neighbor_bad_packets_rcvd_get_elem,
},
{
        .xpath = "/frr-ripd:ripd/state/neighbors/neighbor/bad-routes-rcvd",
        .cbs.get_elem = ripd_state_neighbors_neighbor_bad_routes_rcvd_get_elem,
},

The /frr-ripd:ripd/state/neighbors/neighbor list within the neighbors container has three different callbacks that need to be implemented. Let’s start with the first one, the get_next callback:

static void *ripd_state_neighbors_neighbor_get_next(void *list_entry)
{
        struct listnode *node;

        if (list_entry == NULL)
                node = listhead(peer_list);
        else
                node = listnextnode((struct listnode *)list_entry);

        return node;
}

Given a list entry, the job of this callback is to find the next element from the list. When the list_entry parameter is NULL, then the first element of the list should be returned.

ripd uses the rip_peer structure to represent RIP neighbors, and the peer_list global variable (linked list) is used to store all RIP neighbors.

In order to be able to iterate over the list of RIP neighbors, the callback returns a listnode variable instead of a rip_peer variable. The listnextnode macro can then be used to find the next element from the linked list.

Now the second callback, get_keys:

static int ripd_state_neighbors_neighbor_get_keys(void *list_entry,
                                                  struct yang_list_keys *keys)
{
        struct listnode *node = list_entry;
        struct rip_peer *peer = listgetdata(node);

        keys->num = 1;
        (void)inet_ntop(AF_INET, &peer->addr, keys->key[0].value,
                        sizeof(keys->key[0].value));

        return NB_OK;
}

This one is easy. First, we obtain the RIP neighbor from the listnode structure. Then, we fill the keys parameter according to the attributes of the RIP neighbor. In this case, the neighbor YANG list has only one key: the neighbor IP address. We then use the inet_ntop() function to transform this binary IP address into a string (the lingua franca of the FRR northbound).

The last callback for the neighbor YANG list is the lookup_entry callback:

static void *
ripd_state_neighbors_neighbor_lookup_entry(struct yang_list_keys *keys)
{
        struct in_addr address;

        yang_str2ipv4(keys->key[0].value, &address);

        return rip_peer_lookup(&address);
}

This callback is the counterpart of the get_keys callback: given an array of list keys, the associated list entry should be returned. The yang_str2ipv4() function is used to convert the list key (an IP address) from a string to an in_addr structure. Then the rip_peer_lookup() function is used to find the list entry.

Finally, each YANG leaf inside the neighbor list has its associated get_elem callback:

/*
 * XPath: /frr-ripd:ripd/state/neighbors/neighbor/address
 */
static struct yang_data *
ripd_state_neighbors_neighbor_address_get_elem(const char *xpath,
                                               void *list_entry)
{
        struct rip_peer *peer = list_entry;

        return yang_data_new_ipv4(xpath, &peer->addr);
}

/*
 * XPath: /frr-ripd:ripd/state/neighbors/neighbor/last-update
 */
static struct yang_data *
ripd_state_neighbors_neighbor_last_update_get_elem(const char *xpath,
                                                   void *list_entry)
{
        /* TODO: yang:date-and-time is tricky */
        return NULL;
}

/*
 * XPath: /frr-ripd:ripd/state/neighbors/neighbor/bad-packets-rcvd
 */
static struct yang_data *
ripd_state_neighbors_neighbor_bad_packets_rcvd_get_elem(const char *xpath,
                                                        void *list_entry)
{
        struct rip_peer *peer = list_entry;

        return yang_data_new_uint32(xpath, peer->recv_badpackets);
}

/*
 * XPath: /frr-ripd:ripd/state/neighbors/neighbor/bad-routes-rcvd
 */
static struct yang_data *
ripd_state_neighbors_neighbor_bad_routes_rcvd_get_elem(const char *xpath,
                                                       void *list_entry)
{
        struct rip_peer *peer = list_entry;

        return yang_data_new_uint32(xpath, peer->recv_badroutes);
}

These callbacks receive the list entry as parameter and return the corresponding data using the yang_data_new_*() wrapper functions. Not much to explain here.

Iterating over operational data without blocking the main pthread

One of the problems we have in FRR is that some “show” commands in the CLI can take too long, potentially long enough to the point of triggering some protocol timeouts and bringing sessions down.

To avoid this kind of problem, northbound clients are encouraged to do one of the following:

Create a separate pthread for handling requests to fetch operational data.
Iterate over YANG lists and leaf-lists asynchronously, returning a maximum number of elements per time instead of returning all elements in one shot.

In order to handle both cases correctly, the get_next callbacks need to use locks to prevent the YANG lists from being modified while they are being iterated over. If that is not done, the list entry returned by this callback can become a dangling pointer when used in another callback.

Currently the Sysrepo plugin runs only in the main pthread. The plan in the short-term is to introduce a separate pthread only for handling operational data, and use the main pthread only for handling configuration changes, RPCs and notifications.

RPCs and Actions 

The FRR northbound supports YANG RPCs and Actions through the rpc() callback, which is documented as follows in the lib/northbound.h file:

/*
 * RPC and action callback.
 *
 * Both 'input' and 'output' are lists of 'yang_data' structures. The
 * callback should fetch all the input parameters from the 'input' list,
 * and add output parameters to the 'output' list if necessary.
 *
 * xpath
 *    xpath of the YANG RPC or action
 *
 * input
 *    read-only list of input parameters
 *
 * output
 *    list of output parameters to be populated by the callback
 *
 * Returns:
 *    NB_OK on success, NB_ERR otherwise
 */
int (*rpc)(const char *xpath, const struct list *input,
           struct list *output);

Note that the same callback is used for both RPCs and actions, which are essentially the same thing. In the case of YANG actions, the xpath parameter can be consulted to find the data node associated to the operation.

As part of the northbound retrofitting process, it’s suggested to model some EXEC-level commands using YANG so that their functionality is exposed to other management interfaces other than the CLI. As an example, if the clear bgp command is modeled using a YANG RPC, and a corresponding rpc callback is written, then it should be possible to clear BGP neighbors using NETCONF and RESTCONF with that RPC (the Sysrepo plugin has full support for YANG RPCs and actions).

Here’s an example of a very simple RPC modeled using YANG:

rpc clear-rip-route {
  description
    "Clears RIP routes from the IP routing table and routes
     redistributed into the RIP protocol.";
}

This RPC doesn’t have any input or output parameters. Below we can see the implementation of the corresponding rpc callback, whose skeleton was automatically generated by the gen_northbound_callbacks tool:

/*
 * XPath: /frr-ripd:clear-rip-route
 */
static int clear_rip_route_rpc(const char *xpath, const struct list *input,
                               struct list *output)
{
        struct route_node *rp;
        struct rip_info *rinfo;
        struct list *list;
        struct listnode *listnode;

        /* Clear received RIP routes */
        for (rp = route_top(rip->table); rp; rp = route_next(rp)) {
                list = rp->info;
                if (list == NULL)
                        continue;

                for (ALL_LIST_ELEMENTS_RO(list, listnode, rinfo)) {
                        if (!rip_route_rte(rinfo))
                                continue;

                        if (CHECK_FLAG(rinfo->flags, RIP_RTF_FIB))
                                rip_zebra_ipv4_delete(rp);
                        break;
                }

                if (rinfo) {
                        RIP_TIMER_OFF(rinfo->t_timeout);
                        RIP_TIMER_OFF(rinfo->t_garbage_collect);
                        listnode_delete(list, rinfo);
                        rip_info_free(rinfo);
                }

                if (list_isempty(list)) {
                        list_delete_and_null(&list);
                        rp->info = NULL;
                        route_unlock_node(rp);
                }
        }

        return NB_OK;
}

If the clear-rip-route RPC had any input parameters, they would be available in the input list given as a parameter to the callback. Similarly, the output list can be used to append output parameters generated by the RPC, if any are defined in the YANG model.

The northbound clients (CLI and northbound plugins) have the responsibility to create and delete the input and output lists. However, in the cases where the RPC or action doesn’t have any input or output parameters, the northbound client can pass NULL pointers to the rpc callback to avoid creating linked lists unnecessarily. We can see this happening in the example below:

/*
 * XPath: /frr-ripd:clear-rip-route
 */
DEFPY (clear_ip_rip,
       clear_ip_rip_cmd,
       "clear ip rip",
       CLEAR_STR
       IP_STR
       "Clear IP RIP database\n")
{
        return nb_cli_rpc("/frr-ripd:clear-rip-route", NULL, NULL);
}

nb_cli_rpc() is a helper function that merely finds the appropriate rpc callback based on the XPath provided in the first argument, and map the northbound error code from the rpc callback to a vty error code (e.g. CMD_SUCCESS, CMD_WARNING). The second and third arguments provided to the function refer to the input and output lists. In this case, both arguments are set to NULL since the YANG RPC in question doesn’t have any input/output parameters.

Notifications 

YANG notifations are sent using the nb_notification_send() function, documented in the lib/northbound.h file as follows:

/*
 * Send a YANG notification. This is a no-op unless the 'nb_notification_send'
 * hook was registered by a northbound plugin.
 *
 * xpath
 *    xpath of the YANG notification
 *
 * arguments
 *    linked list containing the arguments that should be sent. This list is
 *    deleted after being used.
 *
 * Returns:
 *    NB_OK on success, NB_ERR otherwise
 */
extern int nb_notification_send(const char *xpath, struct list *arguments);

The northbound doesn’t use callbacks for notifications because notifications are generated locally and sent to the northbound clients. This way, whenever a notification needs to be sent, it’s possible to call the appropriate function directly instead of finding a callback based on the XPath of the YANG notification.

As an example, the ietf-rip module contains the following notification:

notification authentication-failure {
  description
    "This notification is sent when the system
     receives a PDU with the wrong authentication
     information.";
  leaf interface-name {
    type string;
    description
      "Describes the name of the RIP interface.";
  }
}

The following convenience function was implemented in ripd to send authentication-failure YANG notifications:

/*
 * XPath: /frr-ripd:authentication-failure
 */
void ripd_notif_send_auth_failure(const char *ifname)
{
        const char *xpath = "/frr-ripd:authentication-failure";
        struct list *arguments;
        char xpath_arg[XPATH_MAXLEN];
        struct yang_data *data;

        arguments = yang_data_list_new();

        snprintf(xpath_arg, sizeof(xpath_arg), "%s/interface-name", xpath);
        data = yang_data_new_string(xpath_arg, ifname);
        listnode_add(arguments, data);

        nb_notification_send(xpath, arguments);
}

Now sending the authentication-failure YANG notification should be as simple as calling the above function and provide the appropriate interface name. The notification will be processed by all northbound plugins that subscribed a callback to the nb_notification_send hook. The Sysrepo plugin, for instance, uses this hook to relay the notifications to the sysrepod daemon, which can generate NETCONF notifications to subscribed clients. When no northbound plugin is loaded, nb_notification_send() doesn’t do anything and the notifications are ignored.

Operational Data, RPCs and Notifications