Isilon OneFS: WebUI doesn't list local users and shows "Error: Failed while enumerating objects: ERROR_INVALID_DATA" instead

Local users cannot be listed in WebUI and CLI might show the same error occasionally as well. Instead of user listing, error "ERROR_INVALID_DATA" is returned.
 

The following stack is logged in /var/log/isi_papi_d.log:
 

STACK Failed while enumerating objects: ERROR_INVALID_DATA
     from --- (---:0):
     isi_exception::isi_exception(int, char const*, __va_list_tag*) (OFFSET:113)
     api_exception::api_exception(api_error_code, char const*, ...) (OFFSET:144)
     auth_users_handler::http_get(request const&, response&) (OFFSET:2185)
     uri_handler::execute_http_method(request const&, response&) (OFFSET:1405)
     uri_manager::execute_request(request&, response&) (OFFSET:256)
     request_thread::process(fcgi_request*) (OFFSET:189)
     request_thread::on_run() (OFFSET:27)
     pthread_getprio (OFFSET:349)

This issue can be caused by local sam.db database being updated too frequently. If a request to get local users listing comes in the middle of updating the database, OneFS will respond with the error in question: ERROR_INVALID_DATA. Returning user listings instead of the error would cause the result to be inconsistent.

The reason for the database being updated too quickly could be a large number of logins of local users (automated FTP services, for example) as the Last Logon and Logon Count fields are updated in the sam.db every time a user logs in.

The easiest way of confirming that the database is being updated frequently is to confirm how many times per second it's updated. You can do so using the following command from any node of the cluster:

sqlite3 /ifs/.ifsvar/modules/auth/sam/sam.db "select LogonCount, CreatedTime, round(LogonCount / ((julianday('now') - julianday(CreatedTime)) * 24 * 60),7) AS 'Logon Count / sec', CommonName from samdbobjects order by logoncount DESC;"

The results might be as follows, showing a large number of logins for the user rcarn:

Cluster-2# sqlite3 /ifs/.ifsvar/modules/auth/sam/sam.db "select LogonCount, CreatedTime, round(LogonCount / ((julianday('now') - julianday(CreatedTime)) * 24 *60),7) AS 'Logon Count / sec', CommonName from samdbobjects order by logoncount DESC;"
58905762|2012-09-22 13:53:32|77.9596063|rcarn
|2014-07-24 14:24:54||CLUSTER
|2014-07-24 14:24:54||BUILTIN
|2014-07-24 14:24:54||Administrators
|2014-07-24 14:24:54||Users
|2014-07-24 14:24:54||Guests
|2014-07-24 14:24:54||Backup Operators
|2014-07-24 14:24:54||Guest
|2014-07-24 14:24:54||Isilon Users
|2014-07-24 14:24:54||

If that's the case and the sam.db is updated too often, the only resolution at the moment is to upgrade to OneFS 7.1.1.0 and newer. The 7.1.1.x branch of OneFS implemented a change to the way the sam.db is updated, resulting in far less updates of the database at the same workload.
As the database is not updated so often, collision of local user listing and updating the database is hugely reduced.